In some benchmarks like minigo, we'd like to start offline evaluation at a later point as implemented in the reference. Here is an example: the current minigo starts evaluation at 20th iteration and we'd like to start evaluation at the 50th iteration.
The reported TTT won't change as the evaluation is offline. However, the software running time can be reduced to lower the cost of submission.
I recommend an explicit rule update to clarify this point.
In some benchmarks like minigo, we'd like to start offline evaluation at a later point as implemented in the reference. Here is an example: the current minigo starts evaluation at 20th iteration and we'd like to start evaluation at the 50th iteration. The reported TTT won't change as the evaluation is offline. However, the software running time can be reduced to lower the cost of submission. I recommend an explicit rule update to clarify this point.