search

autonomys-ambassadors / ambassador-os-peer-review

Automation for Ambassador OS Peer Review Process
Apache License 2.0
1 stars 1 forks source link

Reported error in evaluator assignments #9

Closed jrwashburn closed 3 weeks ago

jrwashburn commented 3 weeks ago

Sponsor reports one evaluator did not receive any submissions to review in the last cycle.

Review evaluator assignments and code to confirm whether there is an issue.

jrwashburn commented 3 weeks ago

I reviewed the selections. There were 70 submissions but 83 evaluators. Confirmed that this is possible based on the way the assignments work currently, and is not a bug.

We create a list of potential evaluators, which is just a triplicate of the registry. Before assigning evaluators, we filter out any ambassadors already selected for this submission, or that have been selected 3 times in total.

Then we simply get a random index into the remaining list and pick that ambassador.

So it will happen from time to time that an ambassador is not assigned any to evaluate.

It may be a good practice to add an email letting them know that they were not selected for any evaluations this period so they are not wondering, and will have that as evidence if there is a problem later. I'll suggest this be added with #8