Add an evaluation stage for actors (reviewers, ACs, SACs) in conference

[adinriv]

Summary: When working as PCs it's useful to gather indicators of the perceived performance of the different actors (reviewers, ACs and SACs). This request is to add a stage that allows the PCs to trigger a set of tasks to each of the actors (reviewers, ACs, and SACs) to evaluate each other, gather these indicators, aggregate and show them to the PCs.

Details

It will be useful to automatically have a task that prompts the actors (reviewers, ACs, and SACs) to evaluate others actors in their interactions above and below in the hierarchy. For example,

• A reviewer is asked to evaluate the ACs that handle related papers that they reviewed. Optionally, reviewers could evaluate other reviewers in each of the papers.
• An AC is asked to evaluate the reviewers and the SACs in their group of papers.
• A SAC is asked to evaluate their ACs.

A simple solution will be to have a numerical scale and a field for comments. However, a flexible and configurable form will be the best. For example, having a ranking evaluation within the group of papers or per paper (e.g., in the case of evaluating other reviewers) instead of a simple numerical one. Moreover, the form that asks the evaluation should easily show the details of the actor (e.g., associated paper, or review, etc. in a pop-up or to the side) to ease the evaluation.

The results should be aggregated and shown to the PCs (or others that are given visibility) afterwards. Feedback about the performance could be distributed to the actors as well.

[melisabok]

This is already supported using the CustomStage where you can configure any form. The PC console can show the stats for any custom stage.

Please contact the support team to request this to be enabled for your venue.

[melisabok]

Some references here:

• How to setup a CustomStage for a venue: https://github.com/openreview/openreview-py/blob/master/tests/test_cvpr_conference_v2.py#L581

• How to show the stats of the CustomStage in the PC console: https://github.com/openreview/openreview-py/blob/master/openreview/arr/webfield/programChairsWebfield.js#L119