search

mlcommons / inference_policies

Issues related to MLPerf™ Inference policies, including rules and suggested changes
https://mlcommons.org/en/groups/inference/
Apache License 2.0
55 stars 52 forks source link

updating audit rules to ensure fair voting #284

Closed itayhubara closed 7 months ago

github-actions[bot] commented 10 months ago

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

mrmhodak commented 10 months ago

Needs more discussion. All submitters should have same voting privileges.

mrmhodak commented 10 months ago

WG Feedback: Add option for appeal to the board to the rules as a standalone PR.

itayhubara commented 9 months ago

I edit the PR per the WG decision - can you please merge

psyhtest commented 9 months ago

If a submitter undergoes consecutive audits spanning two or more rounds and finds it unfair, they can appeal to the MLCommons board to ensure fairness

Should we say "is selected for" instead of "undergoes"? A submitter may find it unfair it happens for the second time and successfully appeal - then they may only undergo it only once.

If a submitter undergoes consecutive audits spanning two or more rounds

Is it important for the audits to be consecutive? I'd think that being selected again even after a break could be construed as unfair.

psyhtest commented 9 months ago

If a submitter chosen for an audit finds it unfair, they can appeal to the MLCommons Executive Director to ensure fairness.