nicholas-leonard
commented
10 years ago We need a Batch class, or at least an object. Maybe it could be implemented as a batchLog(), the batch equivalent of the epochLog. But then observers would have to use this batchLog, which is a table of values and tables. It has no class-like interface. Suppose we have a Batch class. We would have to extend it for each observer, or subclass a new optimizer, etc. It would probably be easiest to just use a table that is updated by all concerned parties. Or we just create a Batch class that holds inputs, targets, predictions and other batch-like data. Pylearn2 used to have observers (Extensions) view data through the Monitor (the batch/epoch log), which was also used for logging. It seems to be the easiest solution, indeed.
We could call it a State instead of a log. The logger would take an epoch state as input. BatchObservers could take a batch as input. The logger could just be an observer. We could have the logger at the level of the experiment only, or have loggers at the level of propagators as well. If each logger is assigned its own table, column or file, it would allow users to more easily distribute the data.
Discussion
The EarlyStopper needs the validation error. More specifically, it needs to early stop on some value that should be minimized.
It is initialized with a function that is given the experiment as a parameter in the hope that this will prove flexible enough that users can early stop on any metric.
We could create an object similar to pylearn2 costs, which takes outputs (predictions) and targets as params and compares them to generate a value. The ConfusionMatrix kind of does this through batchAdd(outputs, targets).
So these Cost objects could be BatchObservers if the batch outputs and targets would be made available to them via the experiment.
The experiment acts as a kind of host for all states of the experiment. By passing experiment to observers, these can acess all data made available in the experiment's interface.
The main issue is that observers could easily be implemented that call validator:doEpoch(), optimizer:doEpoch(), etc. Therefore, breaking the normal flow of learning. But then again, lua is not about protecting anything. And we want observers to be able to access and even call anything. Users can then build very powerful observers.
Some Propagators or Experiments can become default initialized with a set of commonly used observers to help new beginners.
So what? Propagators, Experimentors, i.e. objects made accessible to the observers should provide interfaces for all the state information that they store.
Okay, but what about the information that observers store within themselves, should we allow observers to access each other's states? The problem with this is that the order in which observers are updated is not garanteed, such that some observers reading the state of another observer may be acting on data not yet up-to-date.
We could create an Epoch and Batch object that is passed to all EpochObservers and BatchObservers respectively? Maybe later...