Stateful monitoring channels

[goodfeli]

A lot of monitoring channels involve modifying some sort of state, either when you define the channel, or when you compute the channel, or both. Mostly this happens for channels involving a random number generator, such as the objective function for an MLP trained with dropout. Currently, the same MRG_RandomStreams object is used to generate the dropout masks for the TrainingAlgorithm and for computing the value of the objective on all monitoring datasets.

This is bad because it means the actual parameters a job runs depends on how many datasets are monitored. If you keep track of the training error as well as the validation set error, you will actually learn something different from if you just keep track of the validation set error.

We should figure out some way to cleanly use different state for each monitoring channel, so that the choice of what to monitor doesn't affect the outcome of the job.

[dwf]

Where does the Monitor's RNG object get set?

[goodfeli]

It's not the Monitor's RNG object. It's generally RNGs owned by the model or the cost or just created by theano and lost in space. A lot of the time to Monitor is using the exact same theano expression for the cost as the training algorithm is. But if the cost is stochastic, that means when the monitor evaluates this expression it runs the default_update of a shared variable created by MRG_RandomStreams that pylearn2 never directly touches.

[lamblin]

The solution we discussed IRL at the time was to make a copy of the value for each shared variable of the graph that has a "default update" rule, and that is used both in the training function and the monitoring one. Then, we can restore the value of these shared variables at the end of the monitoring step.