search

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks
https://mlcommons.org/en/groups/inference
Apache License 2.0
1.2k stars 521 forks source link

Clarification regarding how the accuracy.txt file is generated #1861

Open arjunsuresh opened 2 hours ago

arjunsuresh commented 2 hours ago

The submission generation rules for inference says that the accuracy.txt file should be generated from the accuracy scripts. My interpretation of this is that one should run the reference accuracy scripts stand alone using the logs from the accuracy run and obtain this accuracy.txt file and not dump the accuracy.txt file with in the implementation code. Is this the correct interpretation? 

accuracy.txt # stdout of reference accuracy scripts
arjunsuresh commented 2 hours ago

@psyhtest @ashwin @attafosu Can you please confirm?