search

tryolabs / luminoth

Deep Learning toolkit for Computer Vision.
https://tryolabs.com
BSD 3-Clause "New" or "Revised" License
2.4k stars 399 forks source link

How to resume training with updated tfrecords #206

Closed mauricio-meli closed 6 years ago

mauricio-meli commented 6 years ago

I have a trained model with some custom classes but I need to fine tune it adding new tagged examples to the training set. I've already updated the tfrecords but I can't resume the training job after it is canceled and I also can´t start a new training job with the old job_id.

All I could do is starting a new training job from zero with the updated tfrecords but it trains from the resnet checkpoint instead of my pretrained checkpoint.

So, is there a way to resume a training job after modifying the train.tfrecords?

dekked commented 6 years ago

Just added this in #207 (now merged), try it out and let us know!