Closed josephholler closed 6 months ago
Note: please wait until I have the chance to go through more of these to stabilize a coding scheme.
I've run through a full coding with a scheme that we can discuss before moving forward. These are the main codes of interest:
Flags of interesting themes:
From these codes, we can derive different types of responses. E.g. a respondent understands replicability as reproducibility if they only have codes for open/repeatable, same/similar results, same data, same methods, and same context, or a similarity to the NASEM definition can be derived by combining "new data", "same methods", "new context", and "same" or "similar results".
We may want to combine "same" and "similar" results together into one indicator, because it is sometimes difficult to discern the difference.
see data/derived/public/q6_coding_jh.csv
I revised column headers to good column names and adjusted the list of variables accordingly. I also saved this as q6_coding_jh.xslx
and ordered the columns to parallel above: essential variables for calculating similarity to NASEM definition first, followed by other interesting flags.
I calculated replicability-nasem = sum(data-new, methods-similar, context-new, results-similar) - sum(data-same, methods-varied, context-same, results-same)
and sorted on this variable. It gives a good first pass of placing definitions on a gradient from replicability to reproducibility.
I avoided trying to infer codes from ambiguous definitions unless it was fairly obvious (e.g. an archeologist commenting that no one else could re-dig obviously implies they are thinking same data and same context).
@Peter-Kedron : the combination of context-new
and results-similar
informs us about confirming results in new contexts, whereas we could reserve the validate-external
flag for higher order thinking about validating claims, findings, understanding of phenomena, etc. I'm not sure where that would put the climate reconstructions. Thoughts? It's just a flag for finding interesting definitions, as opposed to a variable that ascertaining similarity to the NASEM definition hinges upon.
Let's keep this comment up-to-date with coding definitions: https://github.com/HEGSRR/OR-Replicability-in-Geography-Survey/issues/5#issuecomment-1432059649
@josephholler, Did my definition checks last Thursday and forgot to load them until today. pk_code column 0 is agreement with joe, 1- flags a def for discussion. Let's go through those next full meeting (3/ 16).
@SarahBardin, I would say don't code these yourself. Just join in our discussion of the 'conflicts' and weigh in there.
the last work on this is in data/derived/public/q6_coding_jh.xlsx
@josephholler when you come back online, I have rechecked the q6 coding and updated the most recent file as data/derived/public/q6_coding_jh_pk.xlsx. The last column of that file now has my explicit suggestion and reasoning for that suggestion on the remaining conflicting definitions.
There are only 31 conflicts remaining. You can filter to them using pk-flag = 1
When you are back, please go through that file and Approve/Disapprove the recommendations. If after that review there are still a few you want to discuss, let me know and we can finish this task.
@Peter-Kedron I looked over the 31 remaining conflicts and changed codes according to my note in jh-response column. The majority of notes are ok
and accept your suggestion while a minority justify keeping the code as-is.
If it looks good, I think we need to remove the other codes from excluded cases, or make sure we exclude them when we analyze.
@josephholler, Thanks Joe. I went through and resolved all of the definitions. Should be good to go now. I'll proceed with the initial analysis and write-up of this section in overleaf.
@josephholler, I am setting up the results summary for Q6. What do you think about the following?
I want to code the match to the NASEM definition along three dimensions as follows:
NASEM Definition: Replicability to mean obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data
My plan is to sum across those three. I am omitting context from that sum because the definition is not clear on that. Unlike replication where same data necessitated same context, we don't have that implicit que here. So I want to leave it out of the summary measure, but will discuss it and provide percentage stats to contextualize the collective thinking.
So you propose that: nasem-replication = data-new + methods-same + (results-same OR results-similar)
That's good; your reasoning is well justified. There are two versions of this data: a working excel sheet, a CSV saved from the excel sheet, and there is an explanation of the measure in the metadata markdown file.
There are multiple parts to these workflows, so here's a separate issue to track them.
We defined replication as:
We may code responses for:
Note that code for
reproducibility
without a code forresults
andnew data
indicates confusion between the termsreproducibility
andreplicability