search

mlcommons / inference_policies

Issues related to MLPerf™ Inference policies, including rules and suggested changes
https://mlcommons.org/en/groups/inference/
Apache License 2.0
55 stars 52 forks source link

Results Table Power Precision #267

Closed bitfort closed 1 year ago

bitfort commented 1 year ago

Look at the results table;

"Closed Power" Tab in https://mlcommons.org/en/inference-edge-21/

For Resnet-50 "System energy (J) per stream*", there are only 2 digits of precision and there are many submissions at the lowest end of precision, e.g. "0.01" and "0.02" and "0.04".

It's hard to differentiate submissions with this lack of precision. To understand these submissions I had to go back to the raw logs and parse out the power values myself.

A few fixes might be, for example,

  1. Use mJ instead of J (i.e. multiply all system energy numbers numbers by 1000)
  2. Add a 3rd (& 4th?) decimal place
  3. Use a specific number of significant digits, as opposed to fixed point

What are our options here?

psyhtest commented 1 year ago

The Power WG agrees to use mJ for SingleStream and MultiStream results (2023/01/31).