Editor Anonymity (Triply Anonymous Refereeing)

[jdutant]

In triply anonymous refereeing the editor(s) do not know an author's identity. The American Philosophical Association recommends this as a way of reducing bias. Some data suggests that this actually works. I'm helping flipping OA one of the journals that apply it (dialectica). We'll use OJS as our publishing platform. But we can't use it for the refereeing workflow as it doesn't support editor anyonymity.

Could OJS introduce editor anonymity? Or create an anonymous editor role?

The role holder would automatically receive papers submission (of a certain category) but without identifying information, and would be able to assign reviewers (including themselves).

This feature has some support in the forum. It is a standard peer review model. Given its recommendation from associations like the APA, I assume a fair number of journals (and perhaps an increasing number of journals) are using it.

[asmecher]

@jdutant, do you use two levels of editors?

• Managing Editor, who can see e.g. author metadata and access all submissions, and
• Section editor, who is assigned to the submission specifically (maybe automatically), and can only see submissions to which they are assigned, potentially without author metadata?

Just trying to imagine how this would fit in with the existing OJS role structure.

[jdutant]

Hi @asmecher, thanks! In our journal, yes. We have a `fishpond' model where:

• the Managing Editor is the only one to see author metadata.
• the Editorial committee members see all submissions anonymously, and self-assign the ones they want to act as editor for (Area/Section Editor, if you wish). Once self-assigned they make a first evaluation and if the paper is worth refereeing they (or the Managing editor) assign referees.

Another philosophy journal (Ergo) works with a slightly different system:

• the Managing Editor is the only one to see author metadata.
• the Managing Editor assigns each paper to some "Area editor".
• the Area editors take care of the evaluation / refereeing process after that. They only see the papers assigned to them.

[NateWr]

I think that this would be very difficult to weave into our existing authorization system, which is built around roles rather than capabilities. It may be easier to think about a way to add onto the reviewer role rather than to try to strip away from the subeditor role. None of this is an easy, one-day task. But here is one idea:

1. Create a special user group, "Managing Reviewer", that is based on the reviewer role.
2. Create a new page that lists all unassigned submissions. Only Managers and Managing Reviewers would have access to this page.
3. On this page, a Managing Reviewer can "claim" a pending submission. When they do this, they will be assigned as a reviewer (optional) and given permission to assign other reviewers.

This approach leaves all of the existing OJS workflow intact, which means there will be less scope for disruptive changes to the core code to unintentionally break or leak access for this configuration. I think it is a safer bet. The downside is that it will be a fair amount of code to maintain -- though you can reuse some of the pieces for reviewer selection.

This still leaves the question of editorial decisions, which are not available to reviewers. This is something the pending submissions page will need to allow managing reviewers to do, unless it is up to the Managing Editor to make a final decision.

Creating a custom page and custom actions through a plugin is not as hard as it might seem. If your institution is willing to build and maintain such a plugin, I'd be happy to provide some guidance along the way.

[jdutant]

Thanks @NateWr for the idea and the offer of help. We could look at developing a plugin, if I can find the resources to allocate to that.

I thought of another kind of workaround. I'm not familiar with the codebase, perhaps you can tell me if that'd be easier to implement? The idea is just to replace an author's identifying metadata with dummies. Roughly:

1. Create additional "hidden" metadata fields for identifying info (given name, last name, login, email address).
2. When a submission is made, identifying info is stored the "hidden" fields, and either (single/doubly anonymous) it is copied in the standard fields, or (triply anonymous) standard fields are filled with dummies like "Jane 123 Doe". In the latter case, all internal pages of OJS display "Jane 123 Doe" where you would have had the author's name.
3. When the paper is published, the "hidden" fields are copied to the standard ones. Or better (see issue 2 below): on the publication side, only the "hidden" fields are used.

I can foresee a few issues:

1. Even the journal manager won't see the identity. (In our journal this wouldn't be a problem.) A solution would be to have a manager-only page displaying the hidden metadata fields.
2. If an author has already published in the journal, and makes a new submission, will their name temporarily appear as "Jane 123 Doe" in the past articles? Possible solutions: publishing should work with the _hidden fields, not the basic ones.
3. In the author's page, authors will see their own login and profile replaced with the dummy addresses, which will be confusing. Solution: ? an explanatory warning?
4. Email loophole: an author's identity would be revealed by email. More precisely: when sending emails from OJS, editors would only see the dummy name, but if the author replies, their message is likely to contain identifying information. Solution (a): section/area editors simply don't email authors! The incentive to violate the policy is low: authors would know, who sends emails is visible to all editors. Solution (b) force authors to reply via the platform, e.g. by notifying them with a do-not-reply message. Solution (c) make all emails being sent by the manager's address, to ensure the manager gets the replies.

On the last issue I like solution (b) as it ensures the platform keeps track of the conversations with authors. There's no setting to enforce it atm though, is there?

[NateWr]

That's a good idea. You can store the anonymised dummy data during submission and then restore the correct data when the submission is accepted in the review stage. That way the data becomes available during Copyediting and Production, so it can be corrected before publication (for example, ORCIDs may need to be authenticated).

Our author records are distinct from user records, so you won't run into the problem where details from past articles are getting modified. However, there are other sources of identifying information and I think that you'll find it hard to anonymise all of them. For example, in addition to modifying the author records, you'll need to account for the following:

1. The editorial workflow shows a list of users assigned to the submission, including the submitting author. This record is tied to the user account, so you'll need to modify the participant list to disguise any author participants. (The user details will not be anonymised with the author details, and some parts of the user account, like the username, can not be changed.)
2. OJS offers a discussion tool for sending messages within the system. Submitting authors and editors can contact each other and see each other's names.
3. Editors have access to all files of their assigned submission. If an author forgets to anonymise their submission files, all bets are off.
4. As you surmised, email addresses would be an additional challenge to figure out. It may be possible to enforce a single reply-to address.

In addition to these challenges, you will also need to account for changes to the system going forward. For example, our plan is to increase transparency for the author during the editorial workflow. This includes showing authors on their own dashboard which editors are assigned, as well as providing authors with an activity log so that they can see the work that is done on their submission (for example, "Daniel Barnes assigned a reviewer.").

With each new version -- in a year, or two, or three or four -- you're going to run up against changes to our UI and REST API. Each new release is going to entail a lot of work to test if the anonymised details are exposed in a new feature or an update to an old feature. Our UI is moving pretty fast these days, as we complete a refactor towards an API-driven interface and conduct accessibility audits.

That's why I think it will be easier to build onto the reviewer role. Any time you work against a core assumption in the platform (that editors are not anonymous), you're going to be fighting an uphill battle. Although building your own UI for these Managing Reviewers may seem like a big challenge, I think it will be easier because you will be working on a clean, ring-fenced patch that you control, rather than trying to locate and maintain pinpoint interventions in a large codebase.

[pmangahis]

+1 from PKP|PS client