adding checkpoint related rules to the training rules

[christ1ne]

I'd like to add a few points in terms of the following considering we are using checkpoints in more than 1 BM.

• Must have portable format (ONNX?)
• Must not restrict choose of hyper-parameter (batch size, learning rate)
• Need to be friendly to port to other framework (i.e. Op are common among popular frameworks)
• Serve pre-defined purpose. (i.e. whether its accepted checkpoint is a method to reduce TTT; whether its accepted to use checkpoint to reduce variance of convergence).
• The step to get a checkpoint needs to be documented along with the reference implementation. The checkpoint needs to be reproducible, following steps in 5, in X% of the run, the result checkpoint should have same or better quality than the checkpoint provided with training reference implementation. (equal or less iterations-to-train from checkpoint).

[bitfort]

Check points show up in benchmarks such as Bert and Minigo. Working with checkpoints can be difficult, especially when using them with different frameworks.

Having a consistent policy around check points would be hugely helpful to submitters; this would include documentation, how they are made available, and in what formats they come in, etc.

Next steps: a proposal for what this would look like, followed by an example using an existing check point.