search

kristinbranson / APT

Animal Part Tracker
GNU General Public License v3.0
71 stars 16 forks source link

Repackaging training data #386

Open kristinbranson opened 2 years ago

kristinbranson commented 2 years ago

Is it possible to avoid repackaging all the training data every time one hits train?

allenleetc commented 2 years ago

Yes should be. I thought we had some re-use/caching in there but probably not as optimized as it could be. Will check on it (caught a bug and am out sick atm).

What are you changing/toggling between successive Trains, just hyperparameters? Or are you adding labels etc?

kristinbranson commented 2 years ago

I don't think I changed anything. Just had training crash the first time I tried it, and was trying it a second time. But, something could have gone very wrong in between those times. In more detail: From when Roian last trained the tracker, I added a few labels and changed the maximum number of animals to 5 (was 2). Then, I hit train. It did ask me a lot of questions that I didn't totally understand about wanting to align stuff and flipping stuff and I think I said to go ahead and do whatever parameter changes it wanted. It regenerated all images for training from scratch. It crashed during training (please see info on Slack). I tried to train again after that. There should be a history of what trackers were stored in the printInfo I put on Slack.