Q17 Coding - Replication Attempts

[Peter-Kedron]

@josephholler I did a first pass a identifying and coding the motivations of the replication attempts actually made by our respondents. The results are in /data/derived/public/q17_coding_pk.xlsx.

I did this inside the entire survey response data sheet because it is useful to cross-ref back to user provided definitions and other questions in a few instances when interpreting unclear responses. The relevant columns to the coding are BI->BO. The columns denote the following:

• Rep_flag: 1=is a replication attempt matching our provided definitions, 0=may be a replication but merits brief discussion
• Flag_Reasoning: Justification for potential exclusion, or a open question worth discussing

I then matched the flags we used to code definitions as respondent motivations for the attempt, but I changed the epistemology flag. Rather than identifying a problem with replication, it now identifies when replication was explicitly identified as epistemological fundamental to the research process.

Flags of interesting themes:

• open-repeatable: emphasizes quality of research as reproducible or repeatable based on publication, code, data, etc.
• validate-external: There is an emphasis on externally validating or generalizing with results from new contexts and studies, or an emphasis on comparing findings / conclusions based on results from new studies. Most definitions including new context and similar results will get this flag.
• unique-space-time: recognition of challenges that spatial / temporal heterogeneity will cause for replication
• validate-internal: Explains replication as a process internal to the research project by repeating and comparing samples/measurements/locations within a population/experiment/landscape, or explicitly references internal validation as a goal of replication through internal repetition, reanalysis, and/or assessment of construct validity.
• epistemology: discusses epistemological and ontological as fundamental to knowledge creation

@josephholler please review these, so we can discuss conflict and come to common coding for subsequent analysis.

[josephholler]

In reproducibility survey, we coded each reason as one of the following categories (copied, with notes about this replication survey after the dash)

1 Verification/Peer-Review -- relates to new validate-external and/or validate-internal 2 Self-check/Promote transparency of own work -- relates to open-repeatable, though this should rarely be the main motivation of a replication 3 Replication -- now rep-flag and prerequisite for coding anything else 4 Teaching/Learning -- let's add a column for this 5 Missing

So after checking against the previous survey, I think we should add a learning motivation. Additionally 1 - Verification was often motivated by suspicious / questionable results in other research in the reproducibility survey. Do we want a code for this?

The reproducibility coding scheme did not allow for multiple motivations: we chose one primary motivation. This approach is different so far, with a different binary code for each motivation, but I think that is ok.

@Peter-Kedron -- let me know what you think about this before I dive though the rest of the responses.

[Peter-Kedron]

@josephholler, I am fine with these changes. Would you mind implementing in your review and flagging any rows where the new coding departs from my coding. The schemes seem to mostly overlap, if you flag departures and give reasoning, I can review and resolve like we did for Q6 coding.

[josephholler]

switched rep_flag to 0 from 2 if it may not be a replication

[josephholler]

@Peter-Kedron : I have gone through Q17. There are two new codes: learning and extending. My notes are in q19-jh column, and if I recommnded adding or removing a code, the excel cell has text add 1 or remove 1, highlighted with conditional formatting.

Please see if the data dictionary (q17_codebook.md) makes sense.

Some thoughts on the coding scheme:

The open-repeatable code shouldn't be very interesting for the motivation of a replication study. This is the domain of reproduction. It is interesting to note when respondents find that a lack of reproducibility was a barrier to replication. However, since the question is about motivation, not barriers, I wouldn't use it in any statistical summary or analysis. I had a higher bar for this code, looking for concerns specifically about reproducibility as a quality of the prior study.

Many definitions are clearly aimed at validating prior findings or attempting to invalidate prior studies perceived to be flawed. However, it is not always clear whether this (in)validation is internal or external. Furthermore, validation is implicit in any motivations with nods to the epistemological function of replication in science. Therefore, the most reliable analysis of these codes might combine the information from internal validation, external validation, and epistemology. The particular codes may still be useful for additional context or for finding qualitative quotes. We could also tease out the internal/external distinction a bit with Q18.

I had a much higher bar for epistemology, looking for statements concerned about the accumulation of knowledge in a field rather than any concern about validity of a particular study.

For several responses, the answer clearly fit a motivation to extend a prior study, while any other codes were implicit. I think adding this code allows us to remove assumptions about other codes.