search

openreview / openreview-py

Official Python client library for the OpenReview API
https://openreview-py.readthedocs.io/en/latest/
MIT License
148 stars 22 forks source link

ARR: validate that no more than N assignments are being done per paper after deployment #2220

Closed melisabok closed 2 months ago

melisabok commented 3 months ago

Pasting the conversation from the slack channel:

"Melisa: After the reviewer proposed assignments are deployed, they become visible to the reviewers and the AEs can add/remove reviewers and invite external reviewers too. At this stage, the validation to restrict 3 reviewers becomes more complicated, here a few questions/scenarios: How many reviewers can a AE invite? If the AE needs a new reviewer and invites 3 and the 3 of them accepts the invitation. How do we tell the other 2 reviewers that they are not needed anymore? If an AE invites a reviewer, the reviewer accepts that invitation but doesn't have a profile. The assignment is not being confirmed until the user signs up. In the meantime the AE could have assigned another reviewer so when the reviewer complete the registration they won't get the paper assigned. What should we do in this case? The AE has 3 reviewers assigned to a paper, only 2 reviews are submitted, the late reviewer says that they can enter the review soon but they don't do it. So the AE needs to unassign the late reviewer in order to find another reviewer. What if the late reviewer finishes the review? they are not assigned anymore...."

"Vincent: the idea that was discussed with Harold at that time was that we would implement this as a parameter - before reviewer assignment is deployed, we will set it to 3. At a later stage (i.e., after reviewer deployment), when an AC needs additional reviewers we will increase its value (probably to 5 or 6). This restriction is most useful before reviewer deployment because we have seen all sorts of crazy things in previous cycles (there were ACs who assigned as many as 15 reviewers to a submission) even though we clearly asked them not to assign more than 3."

We still need to discuss what is the best way to address this. We don't want AEs to be assignments too many reviewers.

melisabok commented 3 months ago

I would follow @celestemartinez 's suggestion and define a quota that check the sum of assignments and invitation assignments.

  1. Save the quota number in the domain content
  2. Add a preprocess in the Assignment invitation where the sum of assignment + invite_assignment edges can not be higher than the quota
  3. Add a preprocess in the Invite Assignment invitation when the sum of assignment + invite_assignment edges can not be higher than the quota

I would implement this in the ARR class for now if another venue is asking for this we can move it to the Venue class.