search

emsejournal / openscience

Empirical Software Engineering journal (EMSE) open science and reproducible research initiative
https://link.springer.com/epdf/10.1007/s10664-019-09712-x
32 stars 11 forks source link

add RR material for MSR initiative #10

Closed neilernst closed 5 years ago

neilernst commented 5 years ago

draft of the RR proposal. Please check the proposal is clear and understandable.

robertfeldt commented 5 years ago

We need more discussion on if we need to limit the number of phase 2 acceptance in the unlikely case where we get a lot of submitted and "in principle accepted" RRs in phase 1. These RR processes are new and one of the ideas of running pilots is so that we can discuss and find "good enough" solutions also for "edge cases". So I'm not committing to a standpoint/opinion yet but I want the sentence "EMSE J. Editors reserve the right to tighten eligibility criteria if necessary." to be discussed.

tzimmermsr commented 5 years ago

In case of a lot of submissions, isn’t it more efficient to limit the number of “in principle accepted” RRs in Phase 1?

Once a RR has been accepted in Phase 1 it should be guaranteed acceptance in Phase 2 as much as possible.

Not sure if we need “EMSE J. Editors reserve the right to tighten eligibility criteria if necessary.”

--Tom

From: Robert Feldt notifications@github.com Sent: Wednesday, June 5, 2019 12:53 AM To: emsejournal/openscience openscience@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [emsejournal/openscience] add RR material for MSR initiative (#10)

We need more discussion on if we need to limit the number of phase 2 acceptance in the unlikely case where we get a lot of submitted and "in principle accepted" RRs in phase 1. These RR processes are new and one of the ideas of running pilots is so that we can discuss and find "good enough" solutions also for "edge cases". So I'm not committing to a standpoint/opinion yet but I want the sentence "EMSE J. Editors reserve the right to tighten eligibility criteria if necessary." to be discussed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femsejournal%2Fopenscience%2Fpull%2F10%3Femail_source%3Dnotifications%26email_token%3DAATWV2SMSAFFODM2SC2I6P3PY5WGZA5CNFSM4HS7XPF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW642GA%23issuecomment-498978072&data=02%7C01%7Ctzimmer%40microsoft.com%7C78b6fe17caf04172865c08d6e98adf2f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953179992859290&sdata=Vby1IZNP%2FcLMNTcKGybsNRzbox5EluMPhrBQjjKuc2A%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAATWV2WNSWPWT53HYIIXRPTPY5WGZANCNFSM4HS7XPFQ&data=02%7C01%7Ctzimmer%40microsoft.com%7C78b6fe17caf04172865c08d6e98adf2f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953179992859290&sdata=5sCs2wbcdqPX6NWw4sEchBgLNgAlRNEo%2FTI4IAZ0WoQ%3D&reserved=0.

robertfeldt commented 5 years ago

Yes, putting a limit on Phase 1 seems simpler and clearer. I guess some "in principle accepted" papers in phase 1 might be withdrawn pre phase 2 but that is same as for normal Special issue.

@neilernst and Janet, any thoughts on this?

I agree we don't need the sentence if we put a limit.

feigensp commented 5 years ago

I agree that if we put a limit, we should do it in Phase 1. However, I do not think this is a good idea, because we want people to register their study. So if we have 30 Phase 1 submissions, this is great! How about instead, we stretch Phase 2 over several periods? So we have one deadline 6 months after Phase 1 acceptance, one 12 months, and another one 18 months after Phase 1 acceptance (or we use 3-month time windows or 4 months). I would not expect that all Phase 1 studies will be completed at the same time, especially qualitative studies could take more time to analyze.

robertfeldt commented 5 years ago

Ok, that's a new idea. :)

I think it is very hard to predict the interest for this track. My impression is that it has already made quite a "stir" in the community and I have heard even people that never really considered MSR as "their" target discussing sending an RR since this is such a new and exciting thing for the SE community. That is why I think we need to plan ahead and discuss what to do if we really get a lot of RRs submitted. But yeah, in principle, it is a nice problem to have if there are many so maybe I should just stop worrying and go with the flow... :)

mendezfe commented 5 years ago

Hi all,

I think we can all agree on that this is a luxury problem we are facing. :) My impression is also that, if at all, we are flooded with submissions in phase 1, maybe with many papers not even in scope of MSR itself but focusing on the broader area of EMSE.

What we should consequently do, in any case (my view), is to define a certain quality standard right from the beginning, maybe even making them explicit in form of coarse templates for what we expect. Another more pragmatic approach to this, or maybe in combination, is to have a rebuttal phase where reviewers can list some open questions to the authors just to make sure that we will not face any surprise in phase 2 + not reject otherwise good proposals just because some things would need some more clarification. This would reduce at least the problem of that we are simply not able to think ahead of all the various types of protocols we might receive.

Regarding the statement “EMSE J. Editors reserve the right to tighten eligibility criteria if necessary.”, what if we write something like “In dependency to the number of eventually accepted studies, EMSE J. Editors reserve the right to deviate from the originally planned publication timelime and / or to allocate accepted manuscripts to issues as they see fit.” (or something along that line)? I would for now leave some statement, but make it more open.

neilernst commented 5 years ago

is there a limit on number of papers per issue? In general I am guessing having 30 top quality papers is an acceptable problem for EMSE. I'm happy to change it to reflect timeline - most people put that stuff on Arxiv now anyway.

FYI I also forwarded to Sarah Nadi and Georgios for their input as MSR Chairs (30 studies would also pose a problem for MSR :)

tzimmermsr commented 5 years ago

Assuming standard acceptance rates, you would need 90-120 submissions to get to 30 top quality papers.

Even if you accept 50%, you still need 60 submissions.

In the unlikely case that we have 30 top-quality pre-registered reports, we will figure out a way to make it work. Likely we have to spread out across different issues.

--Tom

From: Neil Ernst notifications@github.com Sent: Wednesday, June 5, 2019 3:24 PM To: emsejournal/openscience openscience@noreply.github.com Cc: Tom Zimmermann tzimmer@microsoft.com; Comment comment@noreply.github.com Subject: Re: [emsejournal/openscience] add RR material for MSR initiative (#10)

is there a limit on number of papers per issue? In general I am guessing having 30 top quality papers is an acceptable problem for EMSE. I'm happy to change it to reflect timeline - most people put that stuff on Arxiv now anyway.

FYI I also forwarded to Sarah Nadi and Georgios for their input as MSR Chairs (30 studies would also pose a problem for MSR :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femsejournal%2Fopenscience%2Fpull%2F10%3Femail_source%3Dnotifications%26email_token%3DAATWV2VR56GIJ3UBEOY2ON3PZA4GTA5CNFSM4HS7XPF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXBGBXY%23issuecomment-499278047&data=02%7C01%7Ctzimmer%40microsoft.com%7C30ba2990c01947c1460b08d6ea0474ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953702184942839&sdata=WJDloBka8Vl2OtNgKGoWPKXyCxtZ9XabuRgp1Z8bePw%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAATWV2WHFRH6BKLP6TUIL4TPZA4GTANCNFSM4HS7XPFQ&data=02%7C01%7Ctzimmer%40microsoft.com%7C30ba2990c01947c1460b08d6ea0474ab%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636953702184942839&sdata=SbHrpOx%2F56rewe6f0yWxsSaTcB74clX5f2gTAqiVAfo%3D&reserved=0.

gousiosg commented 5 years ago

@neilernst I am not sure why submissions to the pre-registered track would pose problems to the main track... Are you of the opinion that people will opt for one over the other? I actually believe (or at least, hope :-)) that the tracks are complementary.

feigensp commented 5 years ago

I like Daniel's suggestion to include in the CfP the deviation from the planned timeline. This can set the expectations of the authors.

In case we do get flooded, would it be possible to move out-of-scope papers for MSR right to EMSE, so have the Phase I already at the journal? Then we are more flexible regarding the time line.

And I think we could also make a poll (e.g., on Twitter or by personal contacts) a few weeks or months before the deadline to ask who intends to submit. Then we would have a rough estimate about how many submissions to expect and we can react before the actual submission deadline.

tzimmermsr commented 5 years ago

Anything that is truly out-of-scope for MSR should be rejected (regardless of the number of submission). Remember this is a MSR track after all.

I’d also like to emphasize that RR should not be a backdoor to get lower-quality research published in EMSE.

--Tom

From: Janet Siegmund notifications@github.com Sent: Thursday, June 6, 2019 12:49 AM To: emsejournal/openscience openscience@noreply.github.com Cc: Tom Zimmermann tzimmer@microsoft.com; Comment comment@noreply.github.com Subject: Re: [emsejournal/openscience] add RR material for MSR initiative (#10)

I like Daniel's suggestion to include in the CfP the deviation from the planned timeline. This can set the expectations of the authors.

In case we do get flooded, would it be possible to move out-of-scope papers for MSR right to EMSE, so have the Phase I already at the journal? Then we are more flexible regarding the time line.

And I think we could also make a poll (e.g., on Twitter or by personal contacts) a few weeks or months before the deadline to ask who intends to submit. Then we would have a rough estimate about how many submissions to expect and we can react before the actual submission deadline.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femsejournal%2Fopenscience%2Fpull%2F10%3Femail_source%3Dnotifications%26email_token%3DAATWV2SKXRBTT2FSCEZIVMLPZC6QFA5CNFSM4HS7XPF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXCA3BA%23issuecomment-499387780&data=02%7C01%7Ctzimmer%40microsoft.com%7C20ef94d11f274af570ea08d6ea537e68%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636954041655456973&sdata=CKF3F8Vm3VN5Oee36Mvnp1uTGU68C%2Frnd%2FFh2lf%2BSEM%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAATWV2R3EBUSUMHO5WD6G3LPZC6QFANCNFSM4HS7XPFQ&data=02%7C01%7Ctzimmer%40microsoft.com%7C20ef94d11f274af570ea08d6ea537e68%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636954041655466981&sdata=sfmjPPYt1u8lmNKdu8fOuc%2F6YmVySOwWnnXqs%2FphCU0%3D&reserved=0.

feigensp commented 5 years ago

I completely agree, Tom. But I also believe that, if this RR track is a great success, we should use this momentum. So if there is a high-quality RR that is out of scope for MSR, couldn't we somehow encourage the authors?

neilernst commented 5 years ago

I agree ... this was more a scheduling issue of possibly N( with N > 5) presentations during the conference. I wasn't sure if this implied a cap or we just squeeze them in.

Probably this is a case of premature optimization and "let's deal with it when/if it happens"

@neilernst I am not sure why submissions to the pre-registered track would pose problems to the main track... Are you of the opinion that people will opt for one over the other? I actually believe (or at least, hope :-)) that the tracks are complementary.

tzimmermsr commented 5 years ago

Hi Janet. I agree that we should build on any momentum if the RR track is a success.

Out-of-scope authors can always pre-register their Phase 1 report in one of the pre-registration repositories. Authors can already do that, even without the RR track. The RR track adds publication of the Phase 1 report and presentation at a conference.

Authors can always submit a regular paper to EMSE with the Phase 2 results. If we want to guarantee acceptance of Phase 2, the Phase 1 has to be reviewed by experts, and this is difficult if a RR is out-of-scope.

Maybe it makes sense to explore partnerships with other conferences (e.g., ESEM) but first we should figure out what the interest is in RR…

--Tom

From: Janet Siegmund notifications@github.com Sent: Thursday, June 6, 2019 3:01 AM To: emsejournal/openscience openscience@noreply.github.com Cc: Tom Zimmermann tzimmer@microsoft.com; Comment comment@noreply.github.com Subject: Re: [emsejournal/openscience] add RR material for MSR initiative (#10)

I completely agree, Tom. But I also believe that, if this RR track is a great success, we should use this momentum. So if there is a high-quality RR that is out of scope for MSR, couldn't we somehow encourage the authors?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femsejournal%2Fopenscience%2Fpull%2F10%3Femail_source%3Dnotifications%26email_token%3DAATWV2SZES6MI5BNJ6GQAEDPZDN7FA5CNFSM4HS7XPF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXCLSTI%23issuecomment-499431757&data=02%7C01%7Ctzimmer%40microsoft.com%7Ce3bef320b94a4d5e377708d6ea65ee1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636954120885420639&sdata=ZNhlmgBpHUPj4tKb%2F1J6nTuSmItM8u6rvcEMIydkrXM%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAATWV2TTYERPHDZKFTM2OIDPZDN7FANCNFSM4HS7XPFQ&data=02%7C01%7Ctzimmer%40microsoft.com%7Ce3bef320b94a4d5e377708d6ea65ee1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636954120885420639&sdata=TjE6I4KYnZCfEe9PuaNTsHyRm6Ul%2BBfhEaLyhIjxcl8%3D&reserved=0.

robertfeldt commented 5 years ago

Based on our discussions I updated the page to clarify that papers need to be in scope for MSR and that we might have to split the phase 2 accepted papers up and schedule them over multiple issues (if that is an issue at that point in time).

Like Tom, I don't think we should expand outside of MSR scope at this point in time. We need to learn more about RR before we fine-tune processes around this.

snadi commented 5 years ago

Hi everyone, Sorry for chiming in late. I wasn't aware of this discussion here so thanks to Neil for pinging us.

I completely agree that it has to be a clear criteria that the papers are in MSR's scope. Otherwise, there's no point running this as an MSR track.

I went through the current CFP. Some notes/questions:

I apologize if these points have been discussed before but I didn't spot them while going through the above thread.

Sarah

neilernst commented 5 years ago

Thanks for pointing this out Sarah. I copied this flow chart from a journal, so the process is not in line with our approach. I'll endeavour to fix it soon.

My thinking:

mendezfe commented 5 years ago

Hi Neil,

Thanks! Just a small question / remark (I made it already but it might have gone unnoticed in the long conversation :)

Maybe consider one rebuttal phase? RRs are something new to the community and many things in a first submission might not be clearly described so that reviewers could fully understand the designs. Therefore, maybe give reviewers the chance to state some questions the authors can clarify in one further iteration? We could even focus on clarification questions only (I.e. no corrections / changes in the nearer sense)?

Again, this is only a suggestion as I feel that otherwise good proposals might not make it just because the authors did not consider every single detail reviewers might deem important. On the other hand, this rebuttal could of course also get a bit out of control ending with eventually accepting anything... at least I wanted to have brought this point up again ;-)

Best regards, Daniel

On 7. Jun 2019, at 22:07, Neil Ernst notifications@github.com wrote:

Thanks for pointing this out Sarah. I copied this flow chart from a journal, so the process is not in line with our approach. I'll endeavour to fix it soon.

My thinking:

we will have a PC there will just be the usual accept/reject phase, not multiple rounds the pre-registrations will have to attend and present the idea at MSR, like a NIER paper something will appear in proceedings, but maybe not the full registration. We can host that at OSF. I envisioned the EMSE reviews as 2 rounds (e.g. Major�→Accept/Reject) or straight accept/minor, if the study adheres completely to the protocol. The main obstacles will be deviations, or the usual lack of clarity and grammatical soundness issues. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

snadi commented 5 years ago

@neilernst your process sounds good to me!

The only other thing (and perhaps this addresses @mendezfe's concern too) is for the first phase, given that there is no perfect paper, reviewers are likely to point out some flaws/things to consider. It would be good to encourage the PC to clearly mark these points as "Must do for acceptance" versus "Here's a cool thing you can add in the methodology/evaluation". The first can be taken into account during the second-phase review.

feigensp commented 5 years ago

I'm not sure about a rebuttal phase, it might be over-engineering for the first pilot. But I also see that, if we accept a paper in the first phase, we must be certain it is really solid, so the protocol must be really clear, which might be difficult in a first try. So, I am really torn :)

tzimmermsr commented 5 years ago

We could also provide reviewer continuity in those cases.

Maybe there are three outcomes:

--Tom

From: Janet Siegmund notifications@github.com Sent: Tuesday, June 11, 2019 1:24 AM To: emsejournal/openscience openscience@noreply.github.com Cc: Tom Zimmermann tzimmer@microsoft.com; Comment comment@noreply.github.com Subject: Re: [emsejournal/openscience] add RR material for MSR initiative (#10)

I'm not sure about a rebuttal phase, it might be over-engineering for the first pilot. But I also see that, if we accept a paper in the first phase, we must be certain it is really solid, so the protocol must be really clear, which might be difficult in a first try. So, I am really torn :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Femsejournal%2Fopenscience%2Fpull%2F10%3Femail_source%3Dnotifications%26email_token%3DAATWV2RT6UJWA7S4E27W43DPZ5OK7A5CNFSM4HS7XPF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXMLACI%23issuecomment-500740105&data=02%7C01%7Ctzimmer%40microsoft.com%7C175e00d5966b4c1028dd08d6ee4630be%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636958382565108911&sdata=s4Y1VRRrB9GyIP%2FLbIjMP3BC7R5F4lKMW7GiOlq8AfM%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAATWV2SS26T66B4EQBKTSDDPZ5OK7ANCNFSM4HS7XPFQ&data=02%7C01%7Ctzimmer%40microsoft.com%7C175e00d5966b4c1028dd08d6ee4630be%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636958382565118907&sdata=biI7PBpY0%2F%2F4UJnxxqXq%2FDdDeFT4nl7ld2cba3hNIG4%3D&reserved=0.