mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes
https://mlcommons.org/en/groups/training
Apache License 2.0
93 stars 66 forks source link

change the offline evaluation starting point without affecting TTT #335

Closed christ1ne closed 4 years ago

christ1ne commented 4 years ago

In some benchmarks like minigo, we'd like to start offline evaluation at a later point as implemented in the reference. Here is an example: the current minigo starts evaluation at 20th iteration and we'd like to start evaluation at the 50th iteration. The reported TTT won't change as the evaluation is offline. However, the software running time can be reduced to lower the cost of submission. I recommend an explicit rule update to clarify this point.

bitfort commented 4 years ago

SWG:

We will add to the rules that you don't need to do unnecessary offline evaluations.