mlcommons / inference_policies

Issues related to MLPerf™ Inference policies, including rules and suggested changes
https://mlcommons.org/en/groups/inference/
Apache License 2.0
55 stars 52 forks source link

Update non-random audit rules #283

Closed mrmhodak closed 10 months ago

mrmhodak commented 11 months ago

Nomination system for audit

github-actions[bot] commented 11 months ago

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

itayhubara commented 11 months ago

I've carefully reviewed the rules, yet I'm still unclear on what exactly undergoes an audit and what constitutes a satisfactory audit. If the audit simply entails running the scripts and verifying that the auditor achieves similar performance, it seems reasonable that having "multiple submissions from different submitters that achieve similar performance" would exempt your system from an audit, even if it's new. However, if the audit has the authority to examine any aspect it desires, then it makes sense to conduct a comprehensive evaluation of the new system that everyone is adopting. Moreover, I am not sure that OEMs that utilize the same software and hardware (and thus achieve comparable performance) should possess multiple votes. They have no reason to vote against their own system and code. The current proposal raises concerns of unfairness, and I recommend implementing measures to discourage the group from repeatedly choosing the same vendor (potentially by providing justification to the board for reauditing the same system). Additionally, it would be beneficial to clarify the criteria for a fair audit, specifically outlining what should be audited initially and defining the scope of requests the auditor can make to the vendor whose system is undergoing scrutiny.

mrmhodak commented 11 months ago

WG Feedback:

  1. Add link to audit rules
  2. Remove " unless multiple submissions from different submitters that achieve similar performance (per accelerator) exist."
mrmhodak commented 10 months ago

Added link to MLPerf_Audit_Guidelines and removed part exempting submissions with multiple submitters