Ensure that all submitters have a review

mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes

https://mlcommons.org/en/groups/training

Apache License 2.0

92 stars 66 forks source link

Open jonathan-cohen-nvidia opened 4 years ago

jonathan-cohen-nvidia commented 4 years ago

We discussed this in the past (https://github.com/mlperf/training_policies/issues/250) but I'm not clear we ever actually came up with a process for assigning reviewers. We should discuss what this will be.

bitfort commented 4 years ago

SWG:

There are going to be a lot of submissions this time. Every submission deserves a review. There should be a process to do this.

For this round, we will follow this algorithm:

Stack rank by number of submissions
Assign reviewers in pairs walking down the stack rank
If an odd number of reviewers, the bottom 3 in the stack rank will review each other.

This is to be done on Friday!

bitfort commented 4 years ago

SWG:

Algorithm subject to review committee modification to ensure resources are well suited for review.