search

mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes
https://mlcommons.org/en/groups/training
Apache License 2.0
92 stars 66 forks source link

Clarification of MLPerf Score Calculation #341

Open bitfort opened 4 years ago

bitfort commented 4 years ago

To Be Discussed:

I want to clarify for everyone that the official rules for calculating MLPerf scores is timestamp of run_stop minus timestamp of run_start. This has caused some confusion for the "off the clock" evaluation we are allowing this round. Ensure your logs follow the time calculation methodology in cases of off line evaluation by ensuring the timestamp of run_stop is correct; specifically, you may need to analytically calculate and set the timestamp for run_stop using the timestamp of checkpoints.

bitfort commented 4 years ago

SWG:

Make sure your run_stop logging tag has the appropriate timestamp in cases of "off the clock" evaluation. For example, run stop would be the eval timestamp. 

if check_point.meets_target():
    log("run_stop", timestmap=check_point.time_stamp())
bitfort commented 4 years ago

This is a code change for reference DLRM and minigo.