Concerns about deadline

mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes

https://mlcommons.org/en/groups/training

Apache License 2.0

92 stars 66 forks source link

Concerns about deadline #349

Open frank-wei opened 4 years ago

frank-wei commented 4 years ago

We(Alibaba) have some concerns about the deadline? Our cluster is still setting up the network. It may not catch up with the current deadline. I am not sure if there is any buffer time on the deadline?

bitfort commented 4 years ago

Proposal:

(1) 2 weeks delay to the deadline. (2) Submit code and HPs at deadline, submit run logs up to 2 weeks later

bitfort commented 4 years ago

SWG:

A possible extension policy:

The day of the deadline, the following will be submitted: the code which documents best known existing HPs, and an indication of expected performance (either on a different hardware or 1 on the correct hardware)
Submitters submit to a private repo and do not have access to the group repo if they take this exception.
Submitters taking advantage of this extension do not engage in review or HP borrowing, and do not join the review process until they submit all their logs.
Submit logs up to 2 weeks later, these may use the existing HPs which are "slightly" retuned.
Limit to 1 benchmark limited to the submitters who request (so far Alibaba and Intel have requested)

We will give 24 hours to solicit feedback.

christ1ne commented 4 years ago

I suggest we re-write the following text in emails as it may contain ambiguity about whether HP borrowing are allowed or not after submitting logs, for those who are not in the meeting today.

Submitters taking advantage of this extension do not engage in review or HP borrowing, and do not join the review process until they submit all their logs.

frank-wei commented 4 years ago

My understanding is that "After we merge the results after 2 weeks, we can join the review process and do the HP borrowing for the remaining review time period." ?

jonathan-cohen-nvidia commented 4 years ago

What is the deadline for anyone who isn't Ali or Intel to request to make use of this exception?

Seems like if we could agree on the rules for an exception, we would then want to allow all submitters a chance to decide whether they want to request to use it or not. So if if we allow exceptions of this type, then I think the request deadline should be something like 24 hours after we agree.

Akihiko-Kasagi commented 4 years ago

We(Fujitsu) allow this exception to extend deadline. And then, we need to re-write the rules for using exception clearly. We agree with christine and jonathan.

bitfort commented 4 years ago

SWG:

Unfortunately this proposal was not able to reach approval by all submitters. There are no extensions available for this round.