search

pretalx / pretalx

Conference planning tool: CfP, scheduling, speaker management
https://pretalx.com
Apache License 2.0
740 stars 207 forks source link

Reviewers can see current talk score in anonymous mode #611

Closed glasnt closed 5 years ago

glasnt commented 5 years ago

Current Behavior

When navigating to event/<eventname>/reviews, if a submission has at least one review, the current score is revealed.

Expected Behavior

A reviewer should not be able to see comments or review scores for submissions that have not been reviewed.

Steps to Reproduce

  1. Submit a talk into the system
  2. Review a talk at least once
  3. Create a new reviewer account
  4. Access the event/<eventname>/reviews page
  5. The current score is visible

Context

Ideally, a new reviewer coming into review submissions should not see the reviewers of any other reviewer. This is currently available given that individual review comments and rankings cannot be seen on any one submission before review, but the aggregate score is still visible on the submission list on the review screen.

Your Environment

Testing project on pretalx.com, v0.9.0

rixx commented 5 years ago

That's a very good point! We'd also need to take away sorting by review score for reviewers, since that would show ranking information implicitly.

vmx commented 5 years ago

We'd also need to take away sorting by review score for reviewers

I'd like to keep that feature, but instead of showing the average score, show the score you have given.

That's useful if you want to adjust your score later on. When you start reviewing, you don't have much context of the other talks. So you find a talk really good and then you discover that there are many talks that are even better. If you now can sort by score you can get an overview on how you distributed your scores and if you might want to adjust that.

vmx commented 5 years ago

@glasnt To clarify that. Do you want to see the score of other reviewers once you've reviewed it yourself?

I am asking as it seems that I like a similar feature, but I certainly don't want to see the scores/comments from other reviewers at any point.

My workflow would be:

glasnt commented 5 years ago

It would be ideal to have the score be your personal score.

However, the current way pretalx works is:

This has merit, but it also allows for folks to see other's scores and potentially change their own (again, this can be useful)

If the data that can be used to calculate the average score can be seen after the initial review, then I think the aggregate score for that talk should be visible.

The mechanics of a "reveal all the things" might be too much, rather than just adjusting the displayed score in that aggregate view.

Another option would be to have the aggregate score be toggled by the team permissions, then an administrator could choose to change the visibility of this metric after their reviewers had reviewed talks (allowing for a second round of reviews, etc)

vmx commented 5 years ago

I wasn't clear enough. With "My workflow would be" I meant "me as an organiser". So I thought of this as a conference wide setting/permission in the system.

glasnt commented 5 years ago

Ah yes, sorry, I was replying in general to both comments

My workflow would be:

  • Disable "show scores/comments from other reviewers"
  • Let the reviewers do their job
  • Once the review phase is done, enable "show scores/comments from other reviewers" again. Now reviewers can discuss the outcome

That would make most sense, where "scores/comments" included the aggregate count and individual comments/reviews. This mechanic as a team permission should also be most easily implementable.

Remembering to keep the "never show me scores/comments for my own submission" which is a A+ feature for pretalx (more systems should have it)

vmx commented 5 years ago

I looked into implementing exactly this feature last weekend (it was as easy as I expected it to be). Nonetheless I'd like to again put down what I think it should do, to make sure what I'm doing serves both our needs.

glasnt commented 5 years ago

This sounds like exactly what I'm looking for 🎉

Thank you so much for taking the time to clarify my requirements, it's really a joy to see such care with this project.

rixx commented 5 years ago

First off: I appreciate the time and precision both of you have put in your descriptions of the features and workflows you'd like to see – I couldn't wish for more helpful feedback, and working with this level of explanations and suggestions is great.

Your combined suggestion sounds good – it would certainly give users the possibility to handle conferences in the way you have described … but: It feels like a hack, to me. Having to toggle settings halfway through an event to make my software behave like I want it to feels brittle and weird, and unintuitive. As a beginning pretalx user, this is not something you'd think about, or that would feel intuitive (I think).

Allow me to make an alternative suggestion, and tell me if this would help you, too:

I'd like to introduce the concept of review phases. A review phase would be associated with a timeframe (probably only a nullable end date[time] for now), and a couple of settings. Every pretalx event would start out with a sensible default of two phases:

Organisers could add/change review phases. This introduces the concept of review rounds to pretalx without having to change settings on the fly all the time. This would be good especially for larger events.

A real implementation of this would couple reviews with review phases, allowing people to submit multiple reviews per submission and event, but I think even if we leave this functionality for later, this would be a valuable addition to pretalx – because the thing I'm aiming at most with pretalx is helping people manage their events and reason about workflows.

glasnt commented 5 years ago

The concept of having review periods is super useful, and would mesh well with my conference's established workflow (previously having a "deanonymisation time" where a button in PaperCall would be clicked and reveal information)

vmx commented 5 years ago

Thanks @rixx for coming up with your ideas on how to implement this as you know best in which direction pretalx should go. I really like the idea of having review phases. I think all three of us are in agreement here.

This sounds like a major change. Is there already a good example in the code on how this would be implemented? I fear a bit that I won't be able implement this the right way. So pointers from you @rixx would be appreciated.

Now to a minor thing. I understand if you would like to keep the defaults like they are currently, though the conferences I'm know of, try to keep the bias as small as possible, hence try to anonymise things as much as possible.

So I would change the "Review Phase" to (and keep the selection phase as it was):

Though that's not a deal breaker as long as we have all those options :)

knarrff commented 5 years ago

I think it might be good to distinguish between looking at the reviews as "reviewer" and looking at them as "orga"/"editor". Since a person can wear both hats, I would suggest to have two pages: one that exposes all information (to orga members only), and one that is specifically tailored to reviewers (and where also orga members that are also reviewers only see what they should see).

The problem I see with phases is that in case orga-members are actually not reviewing, but have to manage the reviewers, they need to have all the information all the time, e.g., about which contributions still need reviews and which are problematic (vastly different reviews) and so on.

rixx commented 5 years ago

I would suggest to have two pages: one that exposes all information (to orga members only), and one that is specifically tailored to reviewers (and where also orga members that are also reviewers only see what they should see).

This is something that I'm very unsure about. Take the case of anonymous reviews – not seeing speaker names on the review dashboard would suggest to organisers that they are part of the anonymous review process, which is then negated if they look at the submission list instead.

The problem I see with phases is that in case orga-members are actually not reviewing, but have to manage the reviewers, they need to have all the information all the time, e.g., about which contributions still need reviews and which are problematic (vastly different reviews) and so on.

Review phases would always just concern/restrict reviewers, i.e. people with only review access.

rixx commented 5 years ago

As a clarification for @glasnt and @vmx: I'd expect that organisers can always change the state of a submission (including accepting/rejecting it), but that the additional setting of "[ ] Can change submission state" would allow reviewers to do the same, if this flag was set on a submission phase. (I'd clarify the wording, of course.)

I suggest this mode because I know conferences with fairly independent review teams, and because I would feel very weird in constraining non-reviewer organisers from changing the state of a submission. Does that seem ok to you?

vmx commented 5 years ago

@rixx This is how I understood that setting.

glasnt commented 5 years ago

I've used a number of CFP systems, and often the chair or head organiser would set this flag in the system, for a number of reasons:

I believe the second item isn't a problem for pretalx as there is a queuing system for emails so no action apart from "SEND MAIL" actually sends mail.

Having the option to allow this feature per team role would be useful, as it would expand functionality as an option (pretalx is shaping to be very good as all these features are options that can be assigned dynamically for how a group wants to run their reviews team, as opposed to other systems where you have to do it the way they have it setup, and no other)

rixx commented 5 years ago

Aaaand review phases are a thing now! Thank both of you for your input! Screenshot_2019-03-10 aldskjfdaljsf pretalx

vmx commented 5 years ago

WOW! Thank you so much! This will be deployed pretty soon for the FOSS4G, so that we can rid of my own hacks.

rixx commented 5 years ago

Great to hear that – please let me know about any issues you encounter! I've tested the feature, of course, but testing is never the same as using, and I'd like to catch any glaring issues before the 1.0 release.