lmjohns3
commented
9 years ago Thanks for the poke!
Model saving is enabled by providing both
- the "save_progress" argument (a string: name of the filename to save in) as well as
- the "save_every" argument (a number: if positive, saves every N iterations; if negative, saves every N minutes).
If model saving is enabled and there is a file with the correct name at the start of training, that model will be loaded before training. This could be an issue if people make multiple calls to .train() but I'm going to leave it alone for now.
I've only tried this out on the mnist examples, so if you notice anything wrong, please reopen this issue.
I'm seeing the argument for periodic saving on the command line. However, I'm not seeing its implementation in the code. I suggest a periodic saving based on some elapsed time (as opposed to training cycles for example).
Given the long training times, this should be prioritized.