Marian forgets best ce/bleu/perplexity after restarting from a checkpoint

[afaji]

this impact early stopping criteria.

[emjotde]

We are already working on that (issue #64 is getting some attention now). Including saving Adam statistics. This should also save the state of the scheduler and current best validation results, number of stalls.

Later this should also try to resume in roughly the same area in the training corpus. Probably only for the SQLite container.

[snukky]

I store the following scheduler parameters:

• the best score from the first validation metric,
• the number of stalled validation steps,
• the number of epochs and batches,
• the actual learning rate and multiplication factor.

Do we need something else?

[emjotde]

Just as a reminder, the validation scores a not correctly recorded which messes up early stopping. Otherwise it seems to run fine.

[snukky]

Should be fixed now.

[emjotde]

How about the direction of validation, i.e. increasing/decreasing scores are better?