Customizable Reviewer Recommendations

[stranack]

Describe the problem you would like to solve Reviewers are presented with a list of recommendations they can choose from, such as accept, decline, request revisions, etc. This list is not appropriate for all journals. Some would like to change the list to reflect their review practices.

Describe the solution you'd like Somewhere in the settings where review recommendations can be added, edited or removed.

Who is asking for this feature? Requests on the forum, PKP's publishing services, and institutional partners.

Additional Information The original description for this issue was modified:

Investigate the possibility of making the main editorial decisions and reviewer recommendations (accept, decline, revisions, resubmit) more flexible. One simple option would be to allow renaming of the decisions in the Settings.

A more complicated option would be to allow for the creation of new decisions (and the deletion of unwanted decisions), but we would need someway to associate these new decisions with new actions.

The editorial decisions request has been split to https://github.com/pkp/pkp-lib/issues/6074.

[NateWr]

This has been requested https://forum.pkp.sfu.ca/t/question-about-review-response-options/56499/2 and according to @ajnyga it is a common request, so I'll put the community priority label on it.

Unfortunately it's not so simple as just letting people add or rename them. The new editorial statistics calculate things like acceptance and rejection rates, and average time from submission to acceptance or rejection. For these statistics, we need to know precisely what a decision means.

For review recommendations we don't yet need this precision. But I can see such a desire for this in the future: for example, wanting to know the average accept/reject recommendations for an accepted submission, the average number of revisions requested, etc.

[ajnyga]

Thanks for the explanation, I was unaware of the editorial stats (but they sound great)

A quick idea here is of course to have "types" of decisions coupled with freely editable label.

Meaning that you can for example create a decision with the type "Request revisions" and the label "Please try again".

The types then would be of course hard coded, but you could have several of the same type in the list of available decisions.

Edit: just checked and did find at least three requests on this from the forum. In our case, out of the 80 journals we have, at least four have asked about this. They all wanted to have "Minor revisions" / "Major revisions" labels instead of the OJS default ones.

[alexxxmendonca]

A quick idea here is of course to have "types" of decisions coupled with freely editable label.

This is how ScholarOne operates indeed.

[asmecher]

Just to highlight something Nate said:

It would be relatively easy to allow customization of the reviewer recommendations. Nothing programmatic depends on them. Adding, removing, modifying, etc. would all be possible without major implications.

Editor decisions are not so simple. When you record an editor decision, it has major impacts on how the workflow proceeds. Re-labeling the decisions would be possible without major impacts, but adding and removing options would need some serious consideration.

[ajnyga]

Usually the request concerns just the reviewer recommendations.

Are there more cases than these when the workflow is affected:

1. Request revisions / resubmit for review
   • I have a feeling that both affect the workflow the same way. Meaning that if I do "Resubmit for review", the workflow does not force me to start a new review round and vice versa if I do a "Request revisions".
2. Decline
3. Accept / send to copyediting

[NateWr]

Are there more cases than these when the workflow is affected

In terms of stats, the acceptance/rejection from the submission stage is calculated as desk accept / desk reject. So that would need to be preserved as a distinct type.

As far as I know, the only stats we calculate related to review involve the number of rounds. So the exact difference between "resubmit" and "request revisions" is not calculated.

[ajnyga]

I would get to decide, I would replace the "resubmit/revisions required" lingo with "major/minor" revisions at least in the code. Meaning that there would be two "revisions needed" types that could be counted in the stats if needed. The labels by default could still look like "resubmit for review / revisions required" if we want to preserve that.

Having the types "major/minor revisions" needed in the code better reflects the way things are working. Meaning that "resubmit for review" decision does not necessarely lead to a new review round -> the editor can still just approve the changes and go forward.

---

But, do we need to have the same list of options both for the review recommendations and the editorial decisions if the two are not communicating in the code? AFAIK the recommendation is just a visible label for the editor. Why not allow the journals to edit whatever values they want to have there and deal with the actual editorial decisions separately?

[NateWr]

deal with the actual editorial decisions separately?

They are separate already.

[ajnyga]

Sure I realize that. I just meant that why not start with the issue just by allowing journals to edit the review recommendation options. If we got more far reaching requests later, then we can consider a more complicated solution. With just the recommendations it should not be a big change to the code. But it could be that allowing journals to edit those labels will automatically create a request to be able to edit the editorial decisions labels as well.

[unkej]

Hi @ all, I just want to add a "like" to this. If journal editors (or admins without getting too deep into the code) are able to edit the reviewer recommendations, it would be very nice.

[amandastevens]

Another +1 for editable reviewer recommendations from a Publishing Services client.

[NateWr]

@amandastevens is that a +1 for editable reviewer recommendations or editorial decisions or both?

[amandastevens]

Oops, didn't realize these were 2 separate things. It was reviewer recommendations - I edited my above comment.

[NateWr]

It seems like the more widespread community priority is reviewer recommendations. I have repurposed this issue to focus on reviewer recommendations and split the request for customizable editorial decisions out into its own issue: https://github.com/pkp/pkp-lib/issues/6074.

Before we move forward, I'd like to see a few more details provided to better understand the use-cases. What are some examples of different review recommendations that your journals would like to use?

I'd like to retain some information that can be used for editorial statistics. We will need a way to be able to say things like "how many reviews request revisions" or "how many reviewers accept a piece in round 1 vs round 2". To do this, we'll need to settle on some basic recommendation categories from which one or more recommendations stem.

[unkej]

One of our journals asked for these changes, if this is what your are looking for: Accept Submission -> Accept (I think this needs no change) Revisions Required -> Acceptable after minor revisions Resubmit for Review -> Not acceptable without major revisions Resubmit Elsewhere -> Reject Decline Submission -> Reject

[NateWr]

That's great, thanks @unkej!

[amandastevens]

Sometimes editors want new options and sometimes they want to customize the wording on the existing options. The recent request was for this:

• Accept
• Require minor revisions
• Require major revisions and resubmitting
• Reject
• See notes

[NateWr]

Do all the desired options we know of fit into three categories: accept, revise or reject? I'm thinking "see notes" would count as a revise recommendation. Or perhaps we need an "other" category as a catch-all for recommendations that don't fit.

This would allow us to provide editorial statistics about these three categories, while letting journals build whatever recommendation options they want on top. This way we could provide accurate long-term stats about number of reviews that result in accept, revise or reject. But not necessarily provide accurate long-term stats about the number of reviews that go for minor revisions vs major revisions. They'd get lumped together in the stats.

(That's because if the options are editable, they can change over time and therefore can't be used as accurate statistical indicators.)

[asmecher]

@NateWr, it might be possible to apply a similar approach to reviewer recommendations as we do to review forms: once a form (or in this case, reviewer recommendation) is used, it can never be modified. I don't think we need to know programmatically whether a piece of manager-configured text represents an accept or a reject, as long as we can group all cases where reviewers responded the same for the sake of statistics, correct?

[NateWr]

Yes, I guess that makes sense. We can add an active flag to allow people to deactivate recommendations without deleting them.

[amandastevens]

Would it be possible to have a few review recommendation options that are used for statistics, but to allow the text of what they say to be customizeable (at any time)? The exact wording seems to be important to editors, even if the recommendations are usually the same as the current options. Sometimes a new editorial team starts and wants to change the wording.

See Comments is used when a reviewer accepted a review but never submitted it, and other situations different from Revise. We should maintain an open "other" category like this.

[NateWr]

The exact wording seems to be important to editors, even if the recommendations are usually the same as the current options

In the near term, I think the customLocale plugin could take care of this? Or maybe I'm missing a requirement.

I've gone ahead and added this issue to our roadmap. I think it's a sufficient community priority and not too much work to undertake.

[amandastevens]

The wording can be edited with the Custom Locale plugin, but the plugin is difficult for an average user to use. I think it would be preferable if options could be edited within the configuration settings instead of by using the plugin.

[NateWr]

Comment from John Willinsky:

I can say in support of Kevin’s suggestion, which is in line with what I and Krista were bringing up, is that a journal’s distribution of reviewer recommendations is not a stat that has any currency in the field in terms of my experience (compared to acceptance/rejection rate, which is paramount, while days to publication for accepted work is also important). Only to say that a flexibility in reviewer recommendation is likely much more valuable to our community that the potential loss of the ability to have stats on reviewer recommendations.

[mjb24v]

Another +1 for this (and the editor decision equivalent)

[pmangahis]

+1 for editable reviewer recommendations from a Publishing Services client.

[felixhelix]

As a workaround I hid the unused options via the backend CSS:

#recommendation option[value="3"], #recommendation option[value="4"], #recommendation option[value="6"] { 
    display: none;
}

[twakeford]

+1 from Ubiquity on having editable reviewer recommendations.

Apart from adding extra flexibility for the journal, we've also had confusion from editors saying that the options given to the reviewer do not match the editorial decision options (OJS 3.3)