Support registered reports

[amyjko]

From Stefik:

We discussed this issue, but didn't get around to fixing it, as evidence standards was our narrow mission. The key debate is that currently TOCE BANS the practice at the journal. As in, you can't use trial registration in a study, because the practice requires 2-phase peer review, which we don't offer. There are multiple experiments on this going on in the literature. MSR uses it. Neil Brown is running a special issue on the topic in another journal, and many other fields don't allow publication without it (e.g., many psychology journals, many medical journals by law for certain kinds of experiments). It also fixes a mountain of issues in peer review. But, it requires concrete changes to the peer review process and not all studies need it (e.g., N of 1 studies? Not sure). So, it takes some thinking and it never hit our radar.

[amyjko]

More context from Chris:

Our evidence standards committee (of which Monica was a part) had discussions about that, but we never implemented a formal policy. The latest policy on “reporting” standards (no longer “evidence” standards) is in fact posted on the web site. Expanding and refining that policy could be something to take up during your tenure.

[amyjko]

Question from Monica:

I'm looking for guidelines. I just found the AERA webinar on pre-registration, which is helpful. But is there a process we should follow when we submit to TOCE? And where does it say that if we get that accepted that it will be "auto-accepted" later?

[amyjko]

More context from Monica:

We are currently using OSF's prereg protocol. We've looked at several. I'll let you know how it goes--we're in the process of completing one and starting another (both are qualitative studies). As I understood our discussions with the Evidence Standards team, the goal was for TOCE to vet preregistrations and once accepted, the article would then be accepted as well (<-- the latter not being well-defined as of where we ended).

[mmmcgill]

I believe this accurately captures the main points of our preliminary discussions. Looking forward to seeing progress with respect to this.

[amyjko]

• Neil impulsively committed to help.
• This is a place to start: https://osf.io/
• Neil published this article about the CSE experience with pre-registration: https://dl.acm.org/doi/10.1145/3501385.3543971

[amyjko]

Some questions to ponder:

1. Is this traditional pre-registration where you send it in for peer review before running a study to get it reviewed then, or is this just where people put up their packet when they submit? Both plausibly have value, and the first is subsumed by the second, but all else being equal I strongly prefer the first option is allowed. So many papers get rejected after a study has been run and I suspect that peer review is too late in the process for that to be helpful to authors. I bet it mostly just means the community misses out on data. AKA, you can't fix methodological (perceived or otherwise) issues after you've run it, but you absolutely can in 2-phase review. This also gets rid of other issues (e.g., this result is obvious/not), because reviewers might be less willing to make such claims because they don't have the answer in front of them.

2. If a paper "does" go through 2-phase peer review, is a paper "accepted" after the authors put in the study design, meaning that if you run the study reasonably how you said you would, regardless of the result, the paper is guaranteed to be published? Obviously, normal peer review about reporting still applies.

3. For pre-reg, will people be required to put up a replication packet in pre-review or will it be opt-in? If so, will TOCE pick a standardized place to do this to make it easier for authors to know "how" to do this?

[amyjko]

Stefik and Neil Brown are interested in joining this. Write them with some proposals.

[amyjko]

Monica might also be interested in this.

[amyjko]

Draft proposal:

https://docs.google.com/document/d/1z7FeH_wfRvNPvZ4Ykk-EkO1oUIJ3EpOKmqNax-RSjPA/edit#

One more week for board to give feeback.

[amyjko]

Summary of board discussion so far:

To help facilitate discussion, here’s a quick summary of what I’ve seen in the thread:

1. Do require this.
2. Don’t require this.
3. Let’s at least try it.
4. This might increase demands on reviewing labor
5. Be careful about how we position it; there are risks that it will be seen as privileging particular epistemologies, given the journal (and broader community’s) history.

So you all know where I stand, my primary goal is to broaden the epistemologies we welcome in the journal, and so #5 above is of particular concern. If we do move forward with this, it will be my top priority to ensure the language and framing we use is not one of privileging particular forms of knowledge, but rather one of giving authors an alternate path that has some benefits of earlier feedback and more certainty of publication, but is by no means a preferred path, or the only necessarily path, to publication. So I very much disagree with Jens position of requiring it, now or ever, and very much share Sepehr’s concerns about the implications of creating this alternative. That said, part of the reason I’ve prioritized exploring this path is because I think that for some kinds of scholarship the journal has historically excluded (especially work that examines old questions on new populations), this could be an important shield against unjustified post-hoc critiques of significance.

[amyjko]

Sent out proposal for feedback to the SIGCSE-members list and Twitter.

[amyjko]

Finalized proposal, ready to vote

https://docs.google.com/document/d/1z7FeH_wfRvNPvZ4Ykk-EkO1oUIJ3EpOKmqNax-RSjPA/edit#heading=h.iivc4l8lnltk

[amyjko]

My latest update based on the vote:

I've heard from 18/32 of you, which I suppose is a simple majority.

Here's what I see in your anonymous responses: 11 of you support this unequivocally, 5 with reservations, and 1 does not support having this as a publication option. (I'm going to assume the other 14 of you are ambivalent or unavailable, since it's been two weeks).The key reservations expressed were 1) fears of increased community workload, 2) fears of increased complexity, and 3) fears that the existence of this publication option would disadvantage qualitative work published through the current model. (I must admit that I didn't quite understand this last one — it wasn't clear how an option to publish in different way would create bias in the traditional model, which isn't going anywhere. If that was you and you want to clarify, please reply so I don't miss a key point). There was also a general spirit of "let's try this and see what happens".

Based on the above, here's what I'd like to do: 1) post the option in the authors guide as something the journal wants to thoughtfully explore, but with a general sketch of the process rather than the full detail; 2) note that anyone who wants to try it will necessarily be helping grow a new process; but 3) not implement anything until we actually find authors who want to try it. This strikes a balance between courageous experimentation and the possible risks, while also clearly signaling to the community that it is not necessary a permanent option. What I like about this next step is that it helps us evaluate this with a concrete example, rather than just abstract hypotheticals. This is similar to what CSE did (though more conservative, since they did a whole special issue of registered reports submissions).

Let me know your thoughts if you have them. I'll wait at least a week before I do anything.

[amyjko]

I've included a summary of the above on the Author Guidelines page:

https://dl.acm.org/journal/toce/author-guidelines?pbEditor=true#registered-reports

I'm considering this issue closed for now.