Reduce confusion when authors are shown review round status

[forgive38]

Hi all,

During PKP sprint in group about UI/UX, we spoken about giving less information about status of round review for the author. As a first solution, I propose this plugin tested with ojs-stable-3.1.1.x which remove some information for the current round.

https://github.com/forgive38/plugin_RemoveInfoForAuthor

Simon

[asmecher]

This is a great example of a feature where it's tough to get the balance of information right for everyone. @stranack, I'm wondering whether you've heard of other user reports where we're giving too much or too little information to authors about submission progress -- we could either suggest supporting alterations with plugins (as @forgive38 has done), or offering something more built-in like a set of configuration options. I'm not sure a set of options is justified yet vs. the code and interface bloat that would come along with it. (It may also be possible to extend a plugin like this to add the options, rather than having those built into OJS.)

[eddoff]

I have got user reports from our journal editors that the authors get information about if "a review is overdue". This information should not be shown to the authors!

[eddoff]

Hi When you implement this, Is it possible to also remove the "Assigned reviews completed" information in "my queue" for authors?

[asmecher]

@eddoff, we're not going to be able to find a balance of information that'll work for every journal, but in general I think authors find journals too opaque. In my opinion it's possible to let them know what's happening during the review process without e.g. compromising the integrity of the review. You may find this is something you need to customize locally.

[amandastevens]

Two PKP PS hosted journals have recently asked to have review information hidden from authors, including how many review requests have been sent out and whether reviews are overdue. One publishers said it's unprofessional to have this information available to authors. Another said they have to spend a lot of time responding to queries from authors about the status of the reviews.

[asmecher]

Related discussion: https://forum.pkp.sfu.ca/t/authors-can-see-the-count-of-reviewer-assigned-to-submission/53069/4 So far I lean towards Nate's formulation: keeping authors in the loop is an intentional design decision. I wonder if we couldn't tease out the motivations for removing this information in more detail.

[mfelczak]

I wanted to add a +1 for a possible review and rethinking of the current defaults here. I've also received feedback from editors unhappy with the level of detail presented, both because it creates an open window on what they feel should remain an internal process and also because it creates additional confusion for authors which leads to additional follow-up and work for editors.

Maybe one way forward would be to provide well-defined status messages re: workflow state, but without all of the details and notifications. I'm including below some detailed feedback from an editor in case it's helpful.

This is something that I find very problematic. Whilst the idea of transparency is inherently a positive one, the question I raise is: what the benefits for the authors and the publishers if authors can see this side of the peer-review process?

I am not convinced that in the context of a "standard" double blind peer-review authors need to see all the "back-office" work. So, in order to ensure that a submission has 2-3 reviews, I may have to assign up to 6 reviewers. Authors see this, and this is something which I think they should not, especially as they will then receive only 2-3 as reviewers do not accept the task or do not complete the task.

The implications are:

1. When reviewers are late accepting a task, authors know this and get in touch with me asking me what this means/what the problem is. The same when a reviewer completes the review late - the author asks me what is happening, why it is happening, what the implications for their submission are.

2. In addition, I cannot remove a reviewer who has accepted a task but does not complete it. This means that the authors sees that something is not right, but not sure what. In addition, the author sees me adding reviewers without knowing why. So the author contacts me, asking me what is happening, why there are so man reviewers, and why more reviewers are being added.

3. The author knows when reviews are in, and demands to know if/when the decision is taken. Even though for other reasons, we may want to authors know a little later rather than immediately (e.g. in bulk communication, or only after an editorial meeting).

4. The author receives a notification/decision based on 2 reviews, but if the author doesn't like these, demands for the "other" reviews (which have not been made) made by other reviewers which show up on the author's side of the system.

Whilst authors are informed as to when they can expect a decisions in their first email if the paper goes into the review-process, the additional information about the number of reviews, the lateness of acceptance and submission of reviews, worries them unnecessarily. These worries are transferred to us, and we have to spend a lot of time answering emails and queries which are not necessary and which takes up a lot of our time. And we have to answer every single time a reviewer is late in doing anything or nothing at all.

Whilst this transparency may be suitable for another journal, in our case it makes things very frustrating for both sides. It may be suitable for an open peer-review system, or "ongoing publication", but this is not what the OJS system offers. In my opinion, and for the journal I am responsible for it is that type of "transparency" that leads to confusion, not clarity.

[amandastevens]

Another PKP PS hosted client has requested this feature.

[willinsky]

These very detailed comments are indeed helpful in terms of community feedback from publishers and editors, if not from authors. I side with Alec's point is, that we are working against the extent of the opacity, in finding a balance. But as well, we are moving to a new level of peer review clarity to reassure the public and readers that this is what distinguishes scholarly publication (compared to preprints, for example). So as we move, as part of this journal integrity initiative, to posting the date at which the peer review is initiated (when the first reviewer agrees to review) and the number of reviewers who ultimately submitted reviews on the article landing page, we could extend the same courtesy to the author. That is, I'm in favor of authors being able to see the date the review process is initiated (as part of that integrity factor) and the number of reviewers. It's good when we can base these sorts of decisions on a larger strategy that follows from the responsibilities for making research publically available.

[NateWr]

Thanks to @aguen for connecting the discussion here with https://github.com/pkp/pkp-lib/issues/6235. These two issues overlap closely and it is probably best to merge them into one. I'm going to copy the comments from that issue into this one, since there is a good discussion here.

Describe the problem you would like to solve The author is shown a message when reviews are overdue, but this message does not tell them what to expect. Authors should be given some additional contextual information so that they know editors have been made aware and will work to get the necessary reviews completed.

Similar information should be provided when reviews are completed, so that authors know that editors may need to seek further reviews, discuss the recommendations, and make a collective decision.

Describe the solution you'd like The following is a mockup I did a long time ago. It provides a rough outline of what this could look like in a slighly different author view for their submission.

Who is asking for this feature? This arose out of feedback about the confusing status of the overdue reviews message that authors see. Initial work to remedy that is at https://github.com/pkp/pkp-lib/issues/6234. This issue provides a more involved solution.

[NateWr]

A comment on the forum:

To be honest, I do not think that it makes much sense (for us) to provide authors with more automatically generated status information (as is given in your mockup). I fear that this will result in even more emails to editors, asking for clarification etc. and add even more work on them. Review processes are very diverse, e.g., for some subjects it is much more difficult to find reviewers as for others, there are unforeseen delays etc. I would prefer to give authors a general description of the review process (which can be done already on the public journal website) and same broad time ranges or mean turn-around times. That’s it.

[NateWr]

In addition, based on the editor's email above, when showing authors an assigned reviewer count, we need to hide or otherwise indicate reviewers who failed to respond with a review, so that authors do not expect to receive reviews from them.

[NateWr]

Closing this as it still needs consensus. Please consider making a proposal in the feature request category of our community forum where it can be discussed further.

[asmecher]

Re-opening; this is on the SciELO OxS improvements list.

[NateWr]

@alexxxmendonca can you make a specific proposal? I don't see a consensus in the comments above for what messages should be shown to authors and when these messages should be shown.

[alexxxmendonca]

Hi @NateWr,

This is indeed very difficult to get the right balance. The concerns I've read on this thread are all very valid. It's a challenge to provide enough transparency without adding complexity or giving extra work to the editorial staff and editors.

I've pondered about this and I'm going to describe how it is in ScholarOne. It seems to be a successful case given that the journals who use it don't seem to have problems with it (we don't receive any complaints from journals or authors).

Task | What the authors see Select Reviewers | Awaiting Reviewer Selection Invite Reviewers | Awaiting Reviewer Invitation Assign Reviewers | Awaiting Reviewer Assignment Awaiting Reviewer Scores | Awaiting Reviewer Scores

The "Awaiting Reviewer Scores" status is when a submission is under review. Authors do not know specifics, they don't know for how many reviewers the submission was sent, their deadlines and if they are overdue.

Regardless of all reviews beung on time or overdue (even if only one of them are), authors still only see "Awaiting Reviewer Scores" on their view.

The status only changes to the following one when the journal formally moves to the next step of the workflow.

Again, this is how it is in ScholarOne and it's been like that for a very long time. It's not an area that is going through changes anytime soon, which seems to indicate that it works well for all parties.

[NateWr]

Thanks @alexxxmendonca. The one thing I'm missing from your proposal is an estimate of when the author might expect to know the results of the review. Will it be 5 days, 5 weeks or 5 months? In speaking with my friends, this uncertainty regarding timelines is the biggest source of stress and frustration for submitting authors, and erodes their power within the scholarly publishing ecosystem. In my view, we should design the editorial workflow to reduce this uncertainty wherever possible.

[alexxxmendonca]

Hi @NateWr

Not sure I understood the question, but I don't think any journal can predict how long a peer review will take, and not sure they would like to inform that to the author on their submission, with risk of making the authors anxious after the deadline expires. A journal can always mention their average review time on their "Instructions to Authors" or "About the Journal" pages, but not getting into specifics for each paper.

[willinsky]

I support Alex's endorsement of ScholarOne's handling of this. Editors may not be able to estimate the time frame, especially when "awaiting reviewer scores," but to Nate's point authors can be reassured that their manuscript is moving through the four-step review process. That author awareness may also act as a slight nudge on the editors to keep the steps they control moving, while also enabling authors to ask if a step seems inordinately long (given the journal's average review time, say, over last two years, which OJS makes available to editors and could be posted, perhaps in the Publication Facts Label) That said, our steps would be more accurately set out as follows:

Task | What the authors see

1. Assign editor to submission | Editor selection underway. {Stays in place until editor selected]
2. Select and Invite reviewers | Reviewer selection underway. [Stays in place until two reviewer accept]
3. Monitor incoming reviews | Awaiting reviewer evaluations [Stays in place until editor decision is made]
4. Make editorial decision | Editorial decision sent to author [If rejected] or Awaiting revisions [if called for] or Sent to copyediting [if accepted]

[NateWr]

I don't think any journal can predict how long a peer review will take

We do have information about this, in the form of due dates for review assignments. Posting averages may be nice transparency for the journal as a whole, but authors are interested in the specific timeline of their submission.

[alexxxmendonca]

We do have information about this, in the form of due dates for review assignments.

These are purely deadlines. It doesn't mean they will be followed strictly at all.