Welcome!
This project also lives on the OSF.
General project information can be found here.
We recruited collaborators interested in forming data analysis teams to examine the same dataset. The data analysis teams wrote a journal-ready methods and results section and then peer-reviewed each others analyses. Next, we used meta-analytic techniques to explore the data analysis teams' findings.
Recent efforts to replicate published findings have uncovered surprisingly low success rates across disciplines. Moreover, several studies have highlighted the large degree of analytic flexibility in data analysis which can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches that derives not only from aspects of statistical modelling but also from decisions regarding the quantification of the measured behavior. In the present study, we gave the same speech production data set to 30 teams of researchers and asked them to answer the same research question. Using Bayesian meta-analytic tools, we have observed substantial variability between teams and submitted analyses, with analytic and researcher-related predictors having little or no effect on the reported effects.
The first MSA project paper is available as a pre-print on PsyArXiv. You can read it here: https://psyarxiv.com/q8t2k/.
Want to play around with the data? We also have a shiny app available here.