request: per-round review forms & multiple per-reviewer, per-submission reviews

[dants]

The following feature request is probably a big ask. But I believe that the effort may be warranted, as the requested feature reflects how the review process has evolved and works in some conferences.

It is sometimes the case that a single reviewer reviews different versions of the same submission across multiple review rounds. For example, first, as an extended abstract, second, as a full submission, and third, as full submission that has undergone a major revision. The review form for each round can vary and include different fields only relevant for that round.

Presently, hotcrp does not support this process, leading program chairs to overwrite the previous form and reviewers to overwrite their earlier review. This approach has drawbacks, notably the loss of previous review content and the inability to track reviewers' work. For instance, program chairs aiming to distribute the workload evenly among reviewers must maintain an external record of reviewers' work across rounds, as hotcrp does not provide this information (e.g., one reviewer may be assigned to re-review a major revision while another might not, and this information is seemingly not available from hotcrp data).

It would therefore be beneficial if hotcrp could support per-round reviews and per-round review forms, such that a reviewer (R) could be assigned to the same submission across multiple rounds, using a different form for each round, and hotcrp would then maintain and display all per-round reviews by R on the submission page.

[kohler]

Hotcrp already supports per-round review forms: fields can be present or absent depending on review round.

But it currently supports one review per user per paper. For that reason the workflow you mention may be better implemented by (1) different submission classes, so a single “submission” has multiple submission numbers (#1 extended abstract becomes, say, #139 full submission and #172 major revision), all with the same reviewer set, or (2) a different submission system. Who knows maybe Not-Actually-Open Review.

In hotcrp many subsystem semantics, such as formulas, review visibility, etc. depend on the implicit constraint of at most one review per user. A way to make multiple reviews per user per paper happen sooner would be to contribute specific ideas about how to resolve these semantic constraints. Or even a prototype implementation. HotCRP's code is right here in this repository.

[dants]

Thanks for the suggestions and for pointing out the ability to make fields absent and present per round!

Unfortunately, for me, the drawbacks of your suggestions -- spreading the state of a single submission (1) across multiple submissions within the same hotcrp instance, or (2) across multiple submissions in different hotcrp instances -- outweigh the benefits, creating IMO a bigger problem than the lack of the requested functionality. So I prefer to get by with a single submission in a single instance and pay the cost.

I of course understand that the effort required to implement the proposed feature is significant (understatement), as it is incompatible with the current design, seemingly requiring a full overhaul. Understandably, the associated overheads might make this task unfeasible for the hotcrp maintainer. For this reason, however, I doubt that anyone else can realistically pull it off.

Thanks again, --Dan