Closed adinriv closed 6 days ago
This is already supported using the CustomStage where you can configure any form. The PC console can show the stats for any custom stage.
Please contact the support team to request this to be enabled for your venue.
Some references here:
How to setup a CustomStage for a venue: https://github.com/openreview/openreview-py/blob/master/tests/test_cvpr_conference_v2.py#L581
How to show the stats of the CustomStage in the PC console: https://github.com/openreview/openreview-py/blob/master/openreview/arr/webfield/programChairsWebfield.js#L119
Summary: When working as PCs it's useful to gather indicators of the perceived performance of the different actors (reviewers, ACs and SACs). This request is to add a stage that allows the PCs to trigger a set of tasks to each of the actors (reviewers, ACs, and SACs) to evaluate each other, gather these indicators, aggregate and show them to the PCs.
Details
It will be useful to automatically have a task that prompts the actors (reviewers, ACs, and SACs) to evaluate others actors in their interactions above and below in the hierarchy. For example,
A simple solution will be to have a numerical scale and a field for comments. However, a flexible and configurable form will be the best. For example, having a ranking evaluation within the group of papers or per paper (e.g., in the case of evaluating other reviewers) instead of a simple numerical one. Moreover, the form that asks the evaluation should easily show the details of the actor (e.g., associated paper, or review, etc. in a pop-up or to the side) to ease the evaluation.
The results should be aggregated and shown to the PCs (or others that are given visibility) afterwards. Feedback about the performance could be distributed to the actors as well.