Check RR diff

[stefanocoretta]

Please, have a look at https://github.com/many-speech-analyses/many_analyses/blob/rerun/RR_manuscript/RR_diff.pdf

• [x] Reviewed by @troettge
• [x] Reviewed by @jvcasillas

[jvcasillas]

• [ ] I'm still going through this, but I think it looks pretty good. Something to consider, we might want to put the up-to-date version on psyarxiv and tell author's to double check their author/affiliation info. We can tell them we will assume everything is correct unless they contact us.
• [x] #313
• [x] I think this sentence could be better:

  Of the analyses submitted by the 46 teams, the initiating authors identified analyses from 33 teams to be eligible to be included...

  My suggestion:

  Of the analyses submitted by the 46 teams, the initiating authors identified 33 teams with submissions meeting the criteria for inclusion in the meta-analytic model. 

• [x] 3rd level headings, e.g., ### Review ratings, are really hard to see. I assume this won't matter when the Rmd (or latex) is used by the journal.
• [x] I still don't think the x-axis label for figure 3 makes any sense. 🤷🏼‍♀️ Also, @troettge you asked me to remove the compelling/not compelling dichotomy from the shiny app plots. Did we intend to do that here too?
• [x] RE figure 4, the y-axis labels for plot c are now capitalized. I think this should also be the case for the x-axis labels of 4a and 4b.
• [x] Removing the transition sentence ("In light of the observed...") of the second paragraph of page 48 (line 2080 in the Rmd file) makes this paragraph rather odd without context. Perhaps because it now starts with "First of all". I suggest we add a transition or delete "First of all".
• [x] "Second, making analytic pathways completely re-traceable and preregistering them in advance do not change the fact that different analysts might apply different analytic approaches (preregistered or not)." My brain says this needs to be "does" rather than "do". In other words, it seems to me that the subject is singular.

[stefanocoretta]

I just realised now that we could have used the PR review workflow for this... Maybe next time?

[jvcasillas]

Yeah that would have been ideal. I didn´t think of it either. Next time.

[timo-b-roettger]

• [x] figure 1 needs updating. ForkingPaths.png should say ‘bear’ not ‘hope’ in A-2
• [x] “Prosodic encoding of redundant referring expressions” is a title of a osf project so maybe in quotation marks or some other typographic marker @stefanocoretta this looks fine in RR.pdf but not in RR_diff.pdf
• [x] “The mean of the number of different predictors included in teams’ models was 2 “ —> provide decimal point, else it looks to round for an exact number
• [x] “Non Preregistered” is not typographically highlighted so it looks out of place (p.37). Make cursive and maybe with a color? Again @stefanocoretta this looks fine in RR.pdf but not in RR_diff.pdf
• [x] ”into milliseconds by multiplying those values with the meta-analytic estimates standard units” —> “estimated standard units”?
• [x] check the f0 and word window part on p. 43. This has changed now. The text needs to change to:

"For example, five teams measured f0 in the noun and predicted f0 based on typicality as a categorical predictor. Their standardized effect estimates ranged from -0.35 to 0.19 standard deviations. While these teams in principle measured the same thing, they differed in analytical details of how f0 was operationalized (i.e. mean, minimum, maximum, point or range) and how their statistical model was constructed (i.e. the number of predictors ranged from 1-2 and the number of random effect terms ranged from 1-10)."

• [x] "is can be” in discussion --> "can be"
• [x] add sankey diagram reference?

[timo-b-roettger]

@jvcasillas you write: " still don't think the x-axis label for figure 3 makes any sense." that is an uninformative sentence. Can you be more specific ;) Also "RE figure 4, the y-axis labels for plot c are now capitalized. I think this should also be the case for the x-axis labels of 4a and 4b." I cannot see capitalization at all. I will fix it although I am not sure why anymore want capitalization, it makes it harder to read!

[timo-b-roettger]

@stefanocoretta the sankey diagram in supplentary_materialias.pdf is not the updated one based on the recoded sheet. We should not have many unclear categories for duration on right hand side. Can you look into this?

[jvcasillas]

@stefanocoretta the sankey diagram in supplentary_materialias.pdf is not the updated one based on the recoded sheet. We should not have many unclear categories for duration on right hand side. Can you look into this?

I mentioned this here. I'm pretty sure it was because at the time we still hadn't pulled in the newest version of the google sheet (we were still testing). Should be a simple fix by just rerunning the code.

[jvcasillas]

@jvcasillas you write: " still don't think the x-axis label for figure 3 makes any sense." that is an uninformative sentence. Can you be more specific ;)

Indeed it is, but it wasn't when I mentioned it here 🤣 As is, the x-axis label of figure 3 says "Submitted typicality effect". To me, this indicates that the x-axis should plot a continuous variable, but it doesn't. It plots the team name for every individual model used in the meta-analysis. In fact, I think one could make the argument that "Submitted typicality effect" is actual what is plotted on the y-axis (in grey). In the shiny app I labeled the x-axis as "All models", which isn't ideal either, but seems more informative to me.

While we are talking about figure 3, I also mentioned in the comment referenced above that you preferred we don't use the compelling/not compelling dichotomy (at least not by default) in the shiny app, but we are still doing this in figure 3. Do we still want this?

Also "RE figure 4, the y-axis labels for plot c are now capitalized. I think this should also be the case for the x-axis labels of 4a and 4b." I cannot see capitalization at all. I will fix it although I am not sure why anymore want capitalization, it makes it harder to read!

I'm fine either way, but it should be consistent. I initially pointed it out because I think capitalizing the first letter is pretty common. Whatever everybody wants is 👍🏼, just needs to be consistent.

[stefanocoretta]

I'll go through this tomorrow. Please next time create new issues for individual major points because like this it is difficult to follow, whether comments are informative or not.

[stefanocoretta]

In the meantime, @troettge @jvcasillas make sure your repo is up to date, you made pull requests as necessary and all that so that tomorrow we can finalise everything.

[timo-b-roettger]

I will try to fix the plot related things today. update to follow!

[stefanocoretta]

@troettge Are your edits complete? Have you rerun the code to pull the latest coding sheet? Or shall I do that? Shall I re-run other things too? Is the flow plot up to date?

[timo-b-roettger]

@stefanocoretta I did not rerun, pull or knit. So please run the whole thing!

[stefanocoretta]

Thanks! I am glad I asked