Plan for Monitors

[flukeskywalker]

Issue A

Proposals:

1. Rename MonitorAccuracy to MonitorClassificationAccuracy. Additionally, later implement MonitorMeanSquaredError etc. We already have a separate MonitorHammingScore.
2. Implement MonitorScore/MonitorError (a better name) which allows the options to monitor various types of scores/errors such as classification error, hamming score, mean squared error etc.

2 will prevent code duplication. 1 might be more intuitive to use?

Issue B

Additionally, me and @Qwlouse talked some time ago about perhaps renaming Monitors (perhaps call them 'Hooks'?) This is because Monitors can do more than just monitoring stuff.

[Qwlouse]

For Issue A, I'd like to integrate some/most of the sklearn score functions. That should be rather straight-forward, and would give us lots of nice monitors. I'll have to check whether we can just write a general MonitorScore that takes any sklearn score function. If that works I would strongly suggest we do that (i.e. option 2).

Regarding Issue B: The monitors themselves I wouldn't rename, only the mechanism. So you would maybe do: network.add_hook(MonitorScore(...)). If we later have "Monitors" that do more than just monitoring we can call them "...Hook".

[flukeskywalker]

I support utilizing sklearn score functions. This would add sklearn as a dependency, but it makes sense. Perhaps we can bring in more goodies from sklearn such as text preprocessing etc.

On B, it might get a little confusing: We'd say that you can add both Monitors and Hooks, but Monitors are actually hooks, so you add them both using add_hook() :-S

[untom]

Regarding B: I also think "Monitor" is slightly confusing as a term. Who would guess that trainer.add_monitor(bs.MaxEpochsSeen(500)) is actually the correct way to set an upper limit on the number of epochs? If 'hook' does not fit the bill, then maybe add_post_epoch_callback? (though I like hook more than that... but I'm sure there are even better-fitting names).

[Qwlouse]

MaxEpochsSeen is a good example of a Monitor that is not a monitor. We really should rename them. I do like add_hook better, although add_callback is also a possibility (I would skip the post_epoch part, also because they can be called update-wise).

Now the next question becomes: Do we enforce a naming convention for hooks/callbacks? So would we use for example MaxEpochsSeenHook or MonitorScoreHook or do we leave them without the Hook suffix?

[untom]

I'd just not import them into the current namespace: hooks.MaxEpochsSeen

[flukeskywalker]

Keeping the suffix would be consistent with layers, trainers, steps etc. Things like Iterators, weight modifiers etc. currently do not follow this convention.

On 17 August 2015 at 22:38, Klaus Greff notifications@github.com wrote:

MaxEpochsSeen is a good example of a Monitor that is not a monitor. We really should rename them. I do like add_hook better, although add_callback is also a possibility (I would skip the post_epoch part, also because they can be called update-wise).

Now the next question becomes: Do we enforce a naming convention for hooks/callbacks? So would we use for example MaxEpochsSeenHook or MonitorScoreHook or do we leave them without the Hook suffix?

— Reply to this email directly or view it on GitHub https://github.com/Qwlouse/brainstorm/issues/26#issuecomment-131955146.

[flukeskywalker]

Perhaps we should remove the suffix convention altogether from the library? Perhaps it is not of much value and will make scripts look better and less verbose.

[Qwlouse]

After reading @untom's suggestion I was thinking the same. Maybe we should have bs.layers.Rnn instead of bs.RnnLayer and so on. I think I would like that.

[Qwlouse]

I took care of the renaming. See if you like it:

• I removed all the "Layer" suffixes
• I moved all layers to the bs.layers namespace
• I did not rename the implementations for the layers, i.e. they still end in LayerImpl
• I did not rename the implementation files for the layers, i.e. they still end in _layer.py
• I renamed all the monitors-related calls and members to hooks in the Trainer
• I moved all the Monitors to the bs.hooks namespace
• I did not remove the Monitor prefix from MonitorLoss and so on

[flukeskywalker]

LGTM