search

emsejournal / openscience

Empirical Software Engineering journal (EMSE) open science and reproducible research initiative
https://link.springer.com/epdf/10.1007/s10664-019-09712-x
32 stars 11 forks source link

MSR - RR dates and timeline #15

Open neilernst opened 4 years ago

neilernst commented 4 years ago

Tentative timeline, based on other MSR tracks:

The period between initial plan and decision is only 4 weeks, which is pretty short if the registration is complex.

feigensp commented 4 years ago

Since this is new, I think we should start earlier. Can we do it at the beginning of January, because we also include a "rebuttal" phase? And I also think that reviewers might be having more questions for us, so more time for them might also be necessary.

neilernst commented 4 years ago

How about

feigensp commented 4 years ago

Yes, that sounds good.

snadi commented 4 years ago

I think the revised timeline sounds good to give you more time in the middle. Did any of you already try out OSF?

neilernst commented 4 years ago

We have been looking into the format. I think for now we will have a custom registration template, as the OSF one is pretty weighty. Then if accepted we ask the authors to put the registration on OSF.

snadi commented 4 years ago

Sounds good. Is there a backup plan if your custom template doesn't go through? Or perhaps I misunderstood: do you need some form of OSF approval to create a custom template?

neilernst commented 4 years ago

hmm, good question. @feigensp is working on the CFP/template. I think our model was the authors would then upload the registration using the "IPA" approach in the screenshot (but I will check with OSF).

osf form

snadi commented 4 years ago

So from that snapshot, it seems this is supported by default by the platform and doesn't need any approval from their part. I guess you just need to figure out the questions you want the authors to answer. So I guess it's all OK :-)

neilernst commented 4 years ago

Ok, in that case I’ll close this issue and you can use these dates for the MSR website - let me know if you want a different format or via email.

Date Milestone
January 10, 2020 study protocols and plans due
January 31, 2020 initial protocol reviews
February 14,2020 rebuttals/clarifications due
March 2, 2020 In Principle Acceptance (IPA) decision notifications
March 16, 2020 1 page summary plan / camera-ready
March 31, 2020 Reports registered with OSF registry
snadi commented 4 years ago

Just a clarification... why is it in principle acceptance? Aren't you simply accepting the registered report, not actually the study yet? Or is this the typical term used for registered reports?

neilernst commented 4 years ago

That’s the typical term used. In principle it can be accepted if the protocol is followed.

On Sep 19, 2019, at 15:18, snadi notifications@github.com wrote:

Just a clarification... why is it in principle acceptance? Aren't you simply accepting the registered report, not actually the study yet? Or is this the typical term used for registered reports?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

robertfeldt commented 4 years ago

One question (I think from Laurie Williams) when I mentioned pre-registration/RRs in my ESEM19 keynote:

"How are changes to be handled that come from comments on the pre-reg report when it is presented at MSR?"

I guess comments come either from reviewers during Stage 1 review but can also come from audience when presented at MSR. The former are handled in rebuttal/revision process pre MSR accept? While changes that happens between that and Stage 2 review must be explicitly explained and clarified in the paper submitted for Stage 2 review?

neilernst commented 4 years ago

That’s an excellent question. One solution would be open review but I believe @snadi looked at this and it didn’t meet the needs.

The problem is how we check that a study actually responded to the MSR comments (and which comments, sorry, ‘questions’ are worth responding to).

If the protocol is deviated from after IAP I think it can still be published, but would need more fulsome review.

On Sep 20, 2019, at 11:45, Robert Feldt notifications@github.com wrote:

One question (I think from Laurie Williams) when I mentioned pre-registration/RRs in my ESEM19 keynote:

"How are changes to be handled that come from comments on the pre-reg report when it is presented at MSR?"

I guess comments come either from reviewers during Stage 1 review but can also come from audience when presented at MSR. The former are handled in rebuttal/revision process pre MSR accept? While changes that happens between that and Stage 2 review must be explicitly explained and clarified in the paper submitted for Stage 2 review?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

ctreude commented 4 years ago

Would it be feasible to use (some of) the same reviewers for the versions submitted to MSR and the versions submitted to EMSE? And making comments made during the presentation at MSR available to the EMSE reviewers?

neilernst commented 4 years ago

I was sort of thinking the first part would be true .. that you would review it twice. But maybe that’s not necessary.

As for making comments available, I think this would be better served with a more deliberate mechanism like openreview.net or a github repo.

On Sep 21, 2019, at 19:39, ctreude notifications@github.com wrote:

Would it be feasible to use (some of) the same reviewers for the versions submitted to MSR and the versions submitted to EMSE? And making comments made during the presentation at MSR available to the EMSE reviewers?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

snadi commented 4 years ago

@neilernst one option for sharing reviews without using a specific system is to just make it a requirement that authors submit the reviews with their submission to the special issue. Alternatively, if you are worried they might tamper with them (Although I would assume everyone will be honest), you (as editors of the special issue?) can just export reviews with easychair and make it available to reviewers yourselves.

neilernst commented 4 years ago

oh good idea. the only catch is I'm not sure we own the reviews, but we could ask MSR reviewers to check a box when they submit right?

snadi commented 4 years ago

Yup, you need the permission of the reviewers and that checkbox works. Better yet, when inviting MSR reviewers, make it clear that if they accept the invite, they accept the fact that their reviews will be shared with the next phase reviewers. Otherwise, you may get in awkward situations where they decide not to check the checkbox, then some papers will have the reviews while others won’t. In the CFP, make sure it’s also clear for authors that the reviews will be passed on.