kohler / hotcrp

HotCRP conference review software
http://read.seas.harvard.edu/~kohler/hotcrp
Other
331 stars 113 forks source link

reviews vs submission visibility for reviewer comments #357

Closed JuliaLawall closed 2 months ago

JuliaLawall commented 2 months ago

We ask the reviewers to make future author visible metareviews. Often the reviewers forget to change the visibility and just make an ordinary comment. Thus the chairs have to go through and make many comments future author visible as the deadline approaches. In making a whole bunch of comments future author visible, it is easy not to pay too much attention to the messages from hotcrp.

At the same time, there is another menu on the right side about reviews vs submissions. I have observed that one reviewer systematically changed the right menu from reviews to submissions, rather than changing the left menu to future author visibility. Then the chair comes through and if not paying enough attention makes the comment directly author visible. The text from hotcrp says that the authors are notified.

This seems like a very undesirable single point of failure. It's not clear to me why the submission option even exists, or if it needs to exist so that the chairs can eg inform authors about formatting problems, it's not clear why reviewers should be able to select it. And there should be some protection mechanism such that one has to check a box if any message is being sent to the authors without the reviews being visible. It seems very easy to make mistakes.

kohler commented 2 months ago

I agree that the ‘topic’ UI has not been great and that PC members often use it accidentally. (It is used extensively in more interactive contexts such as artifact evaluation. It is also important to show info to reviewers before they review.) Sorry you had a bad experience with it. I have made some changes to improve the UI but your issue is reporting a bad experience not a bug so I am closing it. The UI was telling you, most likely in orange, that authors would be notified. “Check this box to confirm” is usually an irritating UI choice and would be here.