search

REStud / guidance

4 stars 3 forks source link

Refactoring the rules somewhat #26

Closed larsvilhuber closed 2 years ago

larsvilhuber commented 2 years ago

This probably breaks the scripts, they are not debugged. An additional column appears that allows for alternates (for a given item/subitem). This should be used as a selector in the pattern, eg. restud.csv, but it could also be used in a selector app to enforce that one of a set of alternates is chosen.

korenmiklos commented 2 years ago

Merging into a separate branch for experimentation.

korenmiklos commented 2 years ago

@larsvilhuber please see this word document. It looks quite confusing to me. I understand journals could pick one of the alternatives, but I want the main document to be clear to authors, not journal editors.

I would eliminate small differences in policy stance (like 12c), and come up with a different representation for policy alternatives. Add a separate column listing journal specific options (like 12a or 15)?

korenmiklos commented 2 years ago

https://github.com/REStud/guidance/blob/03be441229e6bb3b8f554dc4e68aabb043ae376e/RRS/Reproducible-Research-Standard-1.0.docx

larsvilhuber commented 2 years ago

Miklós,

I may have misunderstood your intent. The basic document is a list of all POSSIBLE clauses. A journal then tags them as "required", "suggested", "verified", or "not applicable". Not applicable ones can be filtered out. The journal-specific version is then the checklist that the author sees.

As a "Reproducible Research Standard" to be directly provided to the author (by whom?) it does not make a lot of sense to me. It must​ bear a relation to a specific journal's policy, and yet be such that it can be compared across journals.

So the flow I saw was:

Alternatively, you can figure out from the Standard as it currently stands a small number of template Policies (Policy RRS-1, Policy RRS-2, ...) which has various configurations of "required", "verified", "suggested", in a spreadsheet like fashion. Corresponding to each is checklist version of it. ReStud states "we follow Policy RRS-1", and that's the checklist the author picks. If EJ also uses "Policy RRS-1", then the same checklist works there.

I do not see the author as a direct end-user of the document as it stands.

Lars

-- Lars Vilhuber, Economist Cornell University / Labor Dynamics Institute/ ILR School / Department of Economics American Economic Association - Data Editor J-PAL IDEA Initiative - Co-Chair Journal of Privacy and Confidentiality - Managing Editor

@.*** | http://lars.vilhuber.com/ p: +1.607-330-5743 | https://twitter.com/larsvil

Assistant: @.*** | +1.607-255-2744

From: Miklós Koren @.> Sent: Monday, November 29, 2021 07:52 To: REStud/guidance @.> Cc: Lars Vilhuber @.>; Mention @.> Subject: Re: [REStud/guidance] Refactoring the rules somewhat (PR #26)

@larsvilhuberhttps://github.com/larsvilhuber please see this word document. It looks quite confusing to me. I understand journals could pick one of the alternatives, but I want the main document to be clear to authors, not journal editors.

I would eliminate small differences in policy stance (like 12c), and come up with a different representation for policy alternatives. Add a separate column listing journal specific options (like 12a or 15)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/REStud/guidance/pull/26#issuecomment-981605739, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABVSQ6CGHJPWVOFAJGH3TRLUONZQFANCNFSM5IXP3MQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

korenmiklos commented 2 years ago

Lars,

This looks like a great workflow indeed. But then how do we communicate to the author that this AER branded document is "the same" as the Restud branded document? Maybe having a reference and link to the Standard gives reassurance that this is something commonly used.

I am happy to work out a backend for journals (.csv -> .md -> .docx whatever workflow) so that journal policies can easily be created. We still need, however, a frontend to the standard (a nice looking .pdf or website) so if anyone checks to see what it is, they get a clear answer.

I wouldn't do Policy 1, 2, ... 6 because the differences are relatively small and this would create confusion. Let me think of a nice way of visually representing "either/or" and "pick one" choices.

As an alternative, we could limit rules to only include the commonly required elements and letting each journal add their own special requirements. For example, Rule (1) could read in the standard "Data are made publicly available in a manner specified by the journal." Then the journal policy could list the Rule and add details on Implementation "(1) Data are included in the replication package."

This is probably much simpler conceptually, but requires very careful wording of rules so that they are neither too general nor too specific.

Miklos

korenmiklos commented 2 years ago

@larsvilhuber I tried the verbal generalization alternative: https://github.com/REStud/guidance/blob/alternatives/RRS/rules.docx

I removed all implementation details (like emailing the editor or explaining special hardware, long-running scripts) and reformulated rules to leave some discretion to journals. See an example:

https://github.com/REStud/guidance/blob/alternatives/RRS/implementation-sample.docx

larsvilhuber commented 2 years ago

This looks like a great workflow indeed. But then how do we communicate to the author that this AER branded document is "the same" as the Restud branded document? Maybe having a reference and link to the Standard gives reassurance that this is something commonly used.

Haven't had time to look at the sample, but just wanted to get at this point of "branding". My thought was that the AEA checklist is "co-branded" with RRS. So there's a RRS logo (say, on the left), and a AEA logo (say on the right). It links back to the general checklist website, which lists all possible policy elements. The rule to allow for co-branding is that you have to re-use the verbiage as-is (i.e., the AEA checklist is a true subset of the general checklist website), except where verbatim deviations are explicitly allowed (as you noted, this might be necessary). (Note that the constraint about "not modifying" applies only to the checklist, not the journal's actually posted policy).

So there is (technical words) a provenance chain from the general checklist to the specific checklist that is visible to the user.

The other part is the Policy 1, 2,... vs. Policy AEA, Policy EJ, etc. One of the goals of the checklist is to convey that the journal requirements are similar. For that, I would want to see that Element 2.a is the same across all journals, but Element 3.c is only used by some journals, say. That is easiest if we have Policy 1, 2, 3... rather than a much larger set of journal-specific policies. But another way might simply be to have a dynamic table on the website, that lists each journal that uses RRS and its checklist (so the website has to be the checklist-generator for each journal), and you can then compare individual journals, something like that. It's technically more challenging, but also more general.

Lars

-- Lars Vilhuber, Economist Cornell University / Labor Dynamics Institute/ ILR School / Department of Economics American Economic Association - Data Editor J-PAL IDEA Initiative - Co-Chair Journal of Privacy and Confidentiality - Managing Editor

@.*** | http://lars.vilhuber.com/ p: +1.607-330-5743 | https://twitter.com/larsvil

Assistant: @.*** | +1.607-255-2744

From: Miklós Koren @.> Sent: Thursday, December 2, 2021 10:02 To: REStud/guidance @.> Cc: Lars Vilhuber @.>; Mention @.> Subject: Re: [REStud/guidance] Refactoring the rules somewhat (PR #26)

Lars,

This looks like a great workflow indeed. But then how do we communicate to the author that this AER branded document is "the same" as the Restud branded document? Maybe having a reference and link to the Standard gives reassurance that this is something commonly used.

I am happy to work out a backend for journals (.csv -> .md -> .docx whatever workflow) so that journal policies can easily be created. We still need, however, a frontend to the standard (a nice looking .pdf or website) so if anyone checks to see what it is, they get a clear answer.

I wouldn't do Policy 1, 2, ... 6 because the differences are relatively small and this would create confusion. Let me think of a nice way of visually representing "either/or" and "pick one" choices.

As an alternative, we could limit rules to only include the commonly required elements and letting each journal add their own special requirements. For example, Rule (1) could read in the standard "Data are made publicly available in a manner specified by the journal." Then the journal policy could list the Rule and add details on Implementation "(1) Data are included in the replication package."

This is probably much simpler conceptually, but requires very careful wording of rules so that they are neither too general nor too specific.

Miklos

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/REStud/guidance/pull/26#issuecomment-984708802, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABVSQ6EECULXLBUEQXNG7NLUO6C7NANCNFSM5IXP3MQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.