jodeleeuw
commented
3 years ago I think we have a lot of data already, so I'm in favor of being judicious about how much we try to add to this. So I'm trying to think carefully about the value of adding additional data:
Replication (in-lab): Would tell us something about the degree to which technology vs. population/environment matters here. If we find that in-lab samples are similar, it's a simple takeaway. If we find that in-lab samples are better (or worse, I suppose!), then it's more complicated. In that case the most valuable thing would be to know why and to try and come up with recommendations that bring online subjects closer to in-lab subjects. I suppose there's also value in knowing whether WebGazer/jsPsych can be used as a substitute for expensive eye tracking in a lab that doesn't have those resources.
Replication (online): I think the value here would be that we could pre-register a variety of decisions that we make during analysis of the first set of data. It would be relatively simple to do this, probably only requiring a day or two of work to collect the data. But the analysis and writing, even with scripts in place, would take at least a week or more I imagine. BTW, we did pre-register the first 5 experiments, but those registrations were somewhat naïve about the problems we would encounter with the analysis work.
Posner task: This could serve as a sort of pre-registered implementation of any recommendations that we come up with from analyzing the first 5 experiments. Probably the most work logistically. I'm on the fence about whether it is worth the added time.
raryskin
arieljames
jkhartshorne
Will we replicate any of the studies in the lab or online and, if so, which ones?
Will we run the posner task?