Closed Peter-Kedron closed 6 months ago
In reproducibility survey, we coded each reason as one of the following categories (copied, with notes about this replication survey after the dash)
1 Verification/Peer-Review -- relates to new validate-external
and/or validate-internal
2 Self-check/Promote transparency of own work -- relates to open-repeatable
, though this should rarely be the main motivation of a replication
3 Replication -- now rep-flag
and prerequisite for coding anything else
4 Teaching/Learning -- let's add a column for this
5 Missing
So after checking against the previous survey, I think we should add a learning motivation. Additionally 1 - Verification
was often motivated by suspicious / questionable results in other research in the reproducibility survey. Do we want a code for this?
The reproducibility coding scheme did not allow for multiple motivations: we chose one primary motivation. This approach is different so far, with a different binary code for each motivation, but I think that is ok.
@Peter-Kedron -- let me know what you think about this before I dive though the rest of the responses.
@josephholler, I am fine with these changes. Would you mind implementing in your review and flagging any rows where the new coding departs from my coding. The schemes seem to mostly overlap, if you flag departures and give reasoning, I can review and resolve like we did for Q6 coding.
switched rep_flag to 0
from 2
if it may not be a replication
@Peter-Kedron : I have gone through Q17. There are two new codes: learning
and extending
.
My notes are in q19-jh column, and if I recommnded adding or removing a code, the excel cell has text add 1
or remove 1
, highlighted with conditional formatting.
Please see if the data dictionary (q17_codebook.md) makes sense.
Some thoughts on the coding scheme:
The open-repeatable
code shouldn't be very interesting for the motivation of a replication study. This is the domain of reproduction. It is interesting to note when respondents find that a lack of reproducibility was a barrier to replication. However, since the question is about motivation, not barriers, I wouldn't use it in any statistical summary or analysis. I had a higher bar for this code, looking for concerns specifically about reproducibility as a quality of the prior study.
Many definitions are clearly aimed at validating prior findings or attempting to invalidate prior studies perceived to be flawed. However, it is not always clear whether this (in)validation is internal or external. Furthermore, validation is implicit in any motivations with nods to the epistemological function of replication in science. Therefore, the most reliable analysis of these codes might combine the information from internal validation, external validation, and epistemology. The particular codes may still be useful for additional context or for finding qualitative quotes. We could also tease out the internal/external distinction a bit with Q18.
I had a much higher bar for epistemology
, looking for statements concerned about the accumulation of knowledge in a field rather than any concern about validity of a particular study.
For several responses, the answer clearly fit a motivation to extend
a prior study, while any other codes were implicit. I think adding this code allows us to remove assumptions about other codes.
@josephholler I did a first pass a identifying and coding the motivations of the replication attempts actually made by our respondents. The results are in
/data/derived/public/q17_coding_pk.xlsx
.I did this inside the entire survey response data sheet because it is useful to cross-ref back to user provided definitions and other questions in a few instances when interpreting unclear responses. The relevant columns to the coding are BI->BO. The columns denote the following:
1
=is a replication attempt matching our provided definitions,0
=may be a replication but merits brief discussionI then matched the flags we used to code definitions as respondent motivations for the attempt, but I changed the epistemology flag. Rather than identifying a problem with replication, it now identifies when replication was explicitly identified as epistemological fundamental to the research process.
Flags of interesting themes:
@josephholler please review these, so we can discuss conflict and come to common coding for subsequent analysis.