search

emsejournal / openscience

Empirical Software Engineering journal (EMSE) open science and reproducible research initiative
https://link.springer.com/epdf/10.1007/s10664-019-09712-x
32 stars 11 forks source link

MSR RR - check with OSF on registration process #16

Open neilernst opened 4 years ago

neilernst commented 4 years ago

@neilernst will talk to OSF folks about getting EMSE/MSR approved(?) as journal following RR process, so authors can submit the IPA (in principle, stage 1 acceptance).

neilernst commented 4 years ago

Via @evomellor - David Mellor - at OSF, EMSE should decide on policies for RR studies. 1/2 are mandatory, 21 strongly recommended. The list includes

  1. Includes pre-study peer review: ✓
  2. Offers provisional pre-study acceptance: ✓
  3. Permanence of adoption: (permanent or special issue)
  4. Offered for novel studies:
  5. Offered for replication studies:
  6. Offered for meta-analysis:
  7. Offered for analyses of existing data sets:
  8. Publishes Registered Reports only:
  9. Allows reporting of unregistered analyses:
  10. Includes post-study peer review:
  11. Allows inclusion of unregistered pilot studies:
  12. Requires public data deposition:
  13. Specifies structured criteria for editorial decisions:
  14. Requires submitted protocols to have prior ethical approval:
  15. Specifies minimum statistical power requirements:
  16. Will publish ‘Withdrawn Registrations’:
  17. Publishes accepted protocols, in full or part, prior to study completion:
  18. Offers incremental (sequential) registration:
  19. Offers incremental addition of unregistered studies:
  20. Offered for qualitative research:
  21. Requires deposition of protocol in public registry following Stage 1 acceptance ✓

this is something EMSE and the open science panel should consider in more detail - pinging @robertfeldt @tzimmermsr

robertfeldt commented 4 years ago

Many things here but with a quick readthrough of the list, for me, a draft "selection" would be:

  1. Includes pre-study peer review: ✓
  2. Offers provisional pre-study acceptance: ✓
  3. Permanence of adoption: "special issue" (we can't commit to make this permanent until we have tried it, possibly even a few times)
  4. Offered for novel studies: Yes
  5. Offered for replication studies: Yes (but ok to say No to keep complexity down?)
  6. Offered for meta-analysis: No (to keep complexity down and not MSR common paper type?)
  7. Offered for analyses of existing data sets: Yes (but ok to say No to keep complexity down?)
  8. Publishes Registered Reports only: No
  9. Allows reporting of unregistered analyses: No (but not fully clear what this one means)
  10. Includes post-study peer review: Yes (if they mean stage 2 reviewing)
  11. Allows inclusion of unregistered pilot studies: No (not sure what this means)
  12. Requires public data deposition: No strong opinion, up to Guest Editors?
  13. Specifies structured criteria for editorial decisions: No strong opinion, up to Guest Editors?
  14. Requires submitted protocols to have prior ethical approval: Probably not since we don't do that in current review process
  15. Specifies minimum statistical power requirements: No strong opinion, up to Guest Editors?
  16. Will publish ‘Withdrawn Registrations’: Probably no but not clear what it means
  17. Publishes accepted protocols, in full or part, prior to study completion: Not in EMSE, maybe in MSR proceedings?
  18. Offers incremental (sequential) registration: No (Keep complexity down)
  19. Offers incremental addition of unregistered studies: No (doesn't fir MSRs process?)
  20. Offered for qualitative research: Probably no (to keep complexity down)
  21. Requires deposition of protocol in public registry following Stage 1 acceptance ✓
neilernst commented 4 years ago

I would default to "No" for most now as well.

@feigensp and I discussed the issue of meta-analyses and felt it was unlikely to be overwhelming, might be a good way to get started with registration, and we could use the topic filter to ensure they were relevant to MSR/EMSE. I wouldn't want to just have meta-reviews, but they are ok with us.

For 9, I think you mean "Yes" as my understanding is this is what is currently going on (all EMSE studies are unregistered by default).

For 17, I think that is the role of MSR/ICSME etc.

Personally I strongly believe in allowing qualitative approaches (20) if we get them ...

robertfeldt commented 4 years ago

9: Hmm, but isn't 9 redundant given 8?

17: Ok, yes.

20: Qualitative studies is fine to include for me but you must ensure you have proper reviewing expertise to review them then. I guess you do, just making sure.

Meta-analyses should be fine but I'm more sceptical towards sysrevs and sysmaps; reviewing their protocols is fairly trivial (method already described in detail) and hard to see what it adds. Meta-analysis I can see a value, yes, but rarely is a sysrev actually a (statistical) meta-analysis IMHO. Maybe hard to make the distinction clear though?

evomellor commented 4 years ago

Hello, I'm chiming in here for a few clarifying points.

9 is typically "yes", as it allows authors to report the results of exploratory, unregistered tests (with the clear expectation that they are indeed clearly marked as non-registered). This assuages many concerns about the preregistration process and is a good way to encourage clear exploration without an author feeling pressured to over emphasize their importance.

You can see what other journals are doing for each of these policies here https://docs.google.com/spreadsheets/d/1D4_k-8C_UENTRtbPzXfhjEyu3BfLxdOsn9j-otrO870/edit#gid=0

When the decisions are finalized, we can provide you with author instructions that correspond to your selections (ie this completed template https://osf.io/pukzy )

robertfeldt commented 4 years ago

Ok, the we are ok with "yes" on 9. Thanks for the clarification, David.