search

mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes
https://mlcommons.org/en/groups/training
Apache License 2.0
92 stars 66 forks source link

Submission Events and Timeline #356

Open bitfort opened 4 years ago

bitfort commented 4 years ago

We need to enumerate when things will happen for the submission and review.

bitfort commented 4 years ago

Week0 - June 22nd Monday June 22 - Receive (over email) information for github uploads, reach out if you do not. Thursday June 25 - Notify SWG Chairs if you will not submit. Friday June 26 - 1PM PDT must have completed the submission. Friday June 26 - 1:30PM PDT Submitters receive email from SWG with our understand of results.

Week 0.5 - June 29th Holiday Week

Week 1 - July 6 Monday - Begin drafting neutral press release Tuesday - (4:30PM-5:30PM PDT) Review committee meeting, discuss objections Thursday - (8:30AM-9:30AM PDT) Review committee meeting Friday - 5PM PDT Objections due in github

Week 2 - July 13 Monday - 5PM PDT response to objections due: see document Tuesday - (4:30PM-5:30PM PDT) Review committee meeting, fix easy to resolve issues and discuss more difficult Wednesday - 5PM PDT written statements on disputed issues not resolved Tuesday Thursday - (8:30AM-9:30AM PDT) Review committee meeting, voting to resolve issues if necessary.

Week 3 - July 20 Tuesday - (4:30PM-5:30PM PDT) Review committee meeting Wednesday - 5PM PDT Final code due Thursday - (8:30AM-9:30AM PDT) Review committee meeting, final voting decisions taking into account code fixes, approval of press release. Friday - 1PM PDT Final logs due, 1:30PM final results distributed by SWG

Week 4 - July 27 Monday - Press and analyst pre-briefings allowed under embargo, all briefings to include neutral press release; 1PM PDT draft of results page available for comment Tuesday - 1PM PDT corrects to result page due, 5PM mock of results available Wednesday - 10AM PDT Results and PR public, press embargo ends

bitfort commented 4 years ago

NOTICE TO ALL SUBMITTERS:

DO NOT DISCUSS RESULTS WITH PUBLIC UNTIL JULY 29th 10AM PDT. All results are private until July 29th 10AM PDT.

christ1ne commented 4 years ago

@bitfort @petermattson the line about results withdraw is missing. should be the Friday of week 3.

jonathan-cohen-nvidia commented 4 years ago

I think there is a principle: once the submission deadline is passed, no one should be surprised by anyone's resubmissions.

Resubmissions can happen for 2 reasons:

  1. To address a compliance issue, apply a fix, then rerun
  2. To steal hyper parameters

All such resubmissions should be declared in advance so everyone knows what to expect.

So there should be a deadline for declaring intent to resubmit a result with stolen HPs. I would propose Week 3 Monday (July 20).

petermattson commented 4 years ago

SWG: Approve week 3 Monday declaration of borrowing w/ specific hps. Make withdraw deadline more obvious.

jonathan-cohen-nvidia commented 4 years ago

Just want to add that today we identified a valid 3rd reason for resubmit:

  1. In case the rules were slightly wrong/ambiguous, there are special cases where we will allow resubmission. I think these are (exhaustive list):
    • RN50 label smoothing - if submitted with 0, allowed to resubmit with 0.1
    • RN50 can switch from norm to trunc norm or vice versa
    • Transformer eval using the wrong dataset, can resubmit with the corrected dataset (which should improve score)