search

IDSIA / brainstorm

Fast, flexible and fun neural networks.
Other
1.3k stars 152 forks source link

Relation between SaveBestNetwork and other monitors #56

Closed flukeskywalker closed 8 years ago

flukeskywalker commented 8 years ago

It is currently dangerous to use SaveBestNetwork with interval/timescale that is different from those used for the hook which logs the accuracy etc, since it simply checks if the best value is the last value in the log. Should this situation be improved somehow? EDIT: Perhaps due to this issue, currently SaveBestNetwork is always works on the 'epoch' timescale with an interval of 1. It seems that switching the fixed timescale to 'update' should take care of all cases. Thoughts?

flukeskywalker commented 8 years ago

I am currently using a version of SaveBestNetwork which gets called at every update and keeps around the best value of the tracked log so far. It saves a network and prints a message only when there is an improvement.

This works fine, but we lose the ability to print something like 'no improvement, last best was at <update/epoch>', since we obviously don't want to produce a message at every update.

Qwlouse commented 8 years ago

I'm slightly concerned with the performance implications of checking all the logs every update for improvement. Other than that I think this should be fine. If you'd like to keep those messages, you could detect changes in the epoch number and print it only then. It might mess with the progressbar a bit though. :-)

flukeskywalker commented 8 years ago

The hook tracks only one specified log, not all of them.

I thought of tracking the epoch number, but it'll just be janky when you're logging something after every N updates but the message appears after every epoch ('Quantity x did not improve during last epoch. Best value so far is ').

Another possibility is to fold the functionality of SaveBestNetwork into MonitorScores. Is that desirable? Then we have the timescale etc. information right there and this problem gets solved.

flukeskywalker commented 8 years ago

If we don't want to merge functionality as suggested above, a8523d1e9aa5b1e4833f32a2ebf3430f930cb542 provides the updated SavedBestNetwork.