search

mlcommons / inference_policies

Issues related to MLPerf™ Inference policies, including rules and suggested changes
https://mlcommons.org/en/groups/inference/
Apache License 2.0
55 stars 52 forks source link

Peer Review Rules #290

Open mrmhodak opened 7 months ago

github-actions[bot] commented 7 months ago

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

arjunsuresh commented 7 months ago

Thank you @mrmhodak for formulating this. I would like to add the below point too.

Any submitter can request for access to a submission system to reproduce the submission. If the submitter grants the access and in case the reviewer is satisfied with his reproducibility work, this process can be used to waive (need not be binding) the system from the official audit.

mrmhodak commented 7 months ago

Arjun, that is an interesting proposition. I will bring it up for discussion in the WG leadership first.

bitfort commented 6 months ago

Any submitter can ask for access to a submission system to reproduce the submission.

Very interesting idea. One clarifying question here: "can ask" implies that the submitter asked can say "no.". I think clarifying right of refusal here is helpful in evaluating the language. Is it true that submitters can say "no" when asked?

arjunsuresh commented 6 months ago

Any submitter can ask for access to a submission system to reproduce the submission.

Very interesting idea. One clarifying question here: "can ask" implies that the submitter asked can say "no.". I think clarifying right of refusal here is helpful in evaluating the language. Is it true that submitters can say "no" when asked?

Thank you @bitfort Yes, the submitter can say no. I just edited the comment to make this clear.

mrmhodak commented 6 months ago

I have simplified the text per @bitfort.

Also, added @arjunsuresh suggestion on granting access and avoiding audit

mrmhodak commented 6 months ago

@bitfort : I would like to merge this before Review Meetings start. Excluding the last clause about audit, does the rest look good to you?

mrmhodak commented 6 months ago

Pushed an update to remove the waiving audit clause. Too close to submission deadline to get an agreement.

To make it easy to find, the final version of the waiving audit clause was:

During the Peer Review Process, a reviewer can ask for access to a submission system to reproduce the submission. If the request is granted and the reviewer is satisfied with the results, this can be used to waive the reviewed system from the audit, subject to the approval of the review group

.