josephholler
commented
1 year ago - Q11: The main interest does seem to be a new location, so the question is now "How do each of the following characteristics of the phenomenon being examined affect the chances of replicating a study in your sub-field in a different location from the prior study?"
- Q13-other sliders: Sure. For simplicity, I think I'd go with an instruction prompt, "Please estimate the percentage of studies ... "
- Q15: Ok. I just simplified the responses because "Affects" is already part of the question. Makes it even easier / clearer.
- Q23: Maybe Q18 and Q23 should just be qualitative then... . I can't imagine many respondents would mark that "difficulty publishing" studies in their sub-field often impacts decisions not to do replications, and then not check that as an influence on themselves not doing replications or vis a versa. Open-ended qualitative space for folks to express their personal primary motivations for doing replications (or not).
- Q24: I'm nervous about this question because it is a high cognitive load. I'm afraid that we might cause incomplete responses with both the definition and this Q at the top of the survey. My aim in the revision was to make it like simple & obvious skip logic. I think this is a really interesting and complicated problem for which a large percentage of respondents probably do not have an answer.
Peter-Kedron
Some standing question that we didn't get to or finish in our meeting today: Q11 - do we want to constrain this question only being about replications in different locations? Right now it covers both same location and different location. That may influence answers. Asking it twice will be to much burden. SO I think we want to focus on what we care about. Is that replications in different locations? Q13-25 (sliders) - What do you think about swapping 'in your opinion' to 'by your estimation'? Q15 - I added fabrication of data/results in response to a pre-test comment. I also think this needs to go back to being a frequency estimating questions (very often, sometimes...) because it could be read as do I think if the statement is true it will do what to the chances. everyone will say if there is pressure to publish original research there will be less replication attempts. We need to ask them how frequently we think a pressure to publish original research affects replication attempts in their sub-field. Then they can reasonably answer, I don't think that pressure exists or matters and mark low. Also with the direction now baked into the statement, I think we can read the frequency scale. Make sense? Q23 - I think we want to keep this question. While I agree with Joe it overlaps somewhat, the skip logic makes this a specific response to what they themselves did not attempt a replication. That makes it different from their overall beliefs about the function/place/incentives of replication. They report here their specific motivation. Q24 - Joe reworked this and it is confusing to me, but I would like Sarah's take? In my mind we want to ask 'What criteria would you use to assess whether a study has been replicated', or some variation on that. I also think this Q doesn't have to be here in the survey. We can move it. Joe's question gets there, but seems to have extra steps to me. I suggest we move Q24 to either 'end of Factors that affect replication block' or 'right after we give the definition'. I think I prefer the latter. High cognitive load Q right up front, but also right after we fix the definition in their heads. Ok, that is what I have for now.