Closed flukeskywalker closed 8 years ago
I am currently using a version of SaveBestNetwork
which gets called at every update and keeps around the best value of the tracked log so far. It saves a network and prints a message only when there is an improvement.
This works fine, but we lose the ability to print something like 'no improvement, last best was
I'm slightly concerned with the performance implications of checking all the logs every update for improvement. Other than that I think this should be fine. If you'd like to keep those messages, you could detect changes in the epoch number and print it only then. It might mess with the progressbar a bit though. :-)
The hook tracks only one specified log, not all of them.
I thought of tracking the epoch number, but it'll just be janky when you're logging something after every N updates but the message appears after every epoch ('Quantity x did not improve during last epoch. Best value so far is
Another possibility is to fold the functionality of SaveBestNetwork
into MonitorScores
. Is that desirable? Then we have the timescale etc. information right there and this problem gets solved.
If we don't want to merge functionality as suggested above, a8523d1e9aa5b1e4833f32a2ebf3430f930cb542 provides the updated SavedBestNetwork
.
It is currently dangerous to use SaveBestNetwork with interval/timescale that is different from those used for the hook which logs the accuracy etc, since it simply checks if the best value is the last value in the log. Should this situation be improved somehow? EDIT: Perhaps due to this issue, currently SaveBestNetwork is always works on the 'epoch' timescale with an interval of 1. It seems that switching the fixed timescale to 'update' should take care of all cases. Thoughts?