search

stickeritis / sticker

Succeeded by SyntaxDot: https://github.com/tensordot/syntaxdot
Other
25 stars 2 forks source link

QoL improvement: Set best epoch in config after train #180

Open sebpuetz opened 4 years ago

sebpuetz commented 4 years ago

Maybe for sticker2, if it's too invasive for sticker?

danieldk commented 4 years ago

I think changing people's configuration files is a bit invasive and could lead to race conditions (e.g. if the user has left open an editor). What we could do is adding an option that copies the configuration file on each epoch with an improvement, with the model file for that epoch. So, you'd have epoch-15.conf, epoch-23.conf, etc.

sebpuetz commented 4 years ago

Imo, one for the ultimately best epoch should be enough. But on the other hand, it'd be straight forward to write a script to do predictions for all epochs which showed improvements if there's a config for each of those. So not completely sure what I'd prefer.

It's just a minor thing but I felt like setting the epoch was the only hurdle to automating prediction (& evaluation).

twuebi commented 4 years ago

Tensorflow has this convenience function:

https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/train/latest_checkpoint

picking the epoch with the highest number should correspond to the best epoch when used in conjunction with only saving upon improvements. Maybe there could be a flag use latest?