Closed mrmhodak closed 10 months ago
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅
I've carefully reviewed the rules, yet I'm still unclear on what exactly undergoes an audit and what constitutes a satisfactory audit. If the audit simply entails running the scripts and verifying that the auditor achieves similar performance, it seems reasonable that having "multiple submissions from different submitters that achieve similar performance" would exempt your system from an audit, even if it's new. However, if the audit has the authority to examine any aspect it desires, then it makes sense to conduct a comprehensive evaluation of the new system that everyone is adopting. Moreover, I am not sure that OEMs that utilize the same software and hardware (and thus achieve comparable performance) should possess multiple votes. They have no reason to vote against their own system and code. The current proposal raises concerns of unfairness, and I recommend implementing measures to discourage the group from repeatedly choosing the same vendor (potentially by providing justification to the board for reauditing the same system). Additionally, it would be beneficial to clarify the criteria for a fair audit, specifically outlining what should be audited initially and defining the scope of requests the auditor can make to the vendor whose system is undergoing scrutiny.
WG Feedback:
Added link to MLPerf_Audit_Guidelines and removed part exempting submissions with multiple submitters
Nomination system for audit