Appendix S4 - database selection

[ASanchez-Tojar]

Here is my update. So far, I went through about 40% of all studies (so a guestimate of 55-60% of all studies that provided data before author correspondence, which is the ones I wanted to focus first so that we do not need to ask the authors for permission?). I briefly wrote the effect size type each study used, and then I checked further those using r or Zr. Some of those were excluded because of no phylogeny, no moderators, involving one of the co-authors or being more on the ecological side (which I understood would be the focus of the SMD/lnRR example). After that, I tried running the code on 4 or 5 datasets. Only two datasets seemed potentially useful for our purposes.

Below I describe them both shortly and provide a draft html file for each of them, so that you can decide whether any or both could be used or instead we need to keep searching. In short: (i) the only issue with the first example (behavioural syndromes) is that the first test of the two-step approach, i.e. the meta-regression with SE, shows "only" a marginally statistically significant slope (p=0.086, which anyway becomes p=0.043 in the meta-regression including other moderators), otherwise I think it is an excellent example with potential for a nice comparison of ours and the original results, including the difference in the effect size adjusted for publication bias; (ii) the second example (condition & parasites) could be a good add-on example to show that our approach can also avoid false positive findings regarding publication bias since the original study found strong evidence of publication bias when using traditional methods, and we do not.

ft044: on behavioural syndromes (Garamszegi et al. 2012, Evol Ecol, 113 citations)

• Contains phylogeny and moderators of interest, and originally found evidence of publication bias using Begg’s method (Kendall’s tau = 0.126, P = 0.028) and trim-and-fill (15 studies missing; adjusted Zr = 0.201).
• Equation 21 finds a marginally statistically significant slope (p=0.086), so I proceeded to next equations.
• Equation 22 finds an adjusted overall effect size, i.e. the intercept, equal to 0.248.
• Equation 23 does not find evidence of time-lag bias.
• Equation 24 including year and two other moderators that were available for the whole dataset (rather than for a subset of the effect sizes only) shows that evidence for small-study bias is a bit clearer even after accounting for some heterogeneity (sei slope p = 0.0426).
• IMPORTANT: I did not model phylogeny yet to avoid investing time on a dataset that we may not use, so that could change some results slightly.

ft177: on the relationship between body condition and parasite infection (Sánchez et al. 2018, Ecol Lett, 44 citations)

• Contains phylogeny and moderators of interest, and originally found strong evidence of publication bias using simple Egger’s regression (z = –2.77, P = 0.006) and trim-and-fill (99 studies missing; adjusted Zr = 0.01).
• Equation 21 finds no evidence whatsoever of small-study effects, which contrast with the original results from the authors.
• Equation 23 does not find evidence of time-lag bias.
• IMPORTANT: I did not model phylogeny yet to avoid investing time on a dataset that we may not use, so that could change some results slightly.
• Despite not finding evidence of publication bias, we could use this example to show that using the right methodology can also avoid finding false positives when testing for publication bias.

html_examples.zip

Please, let me know if those are good datasets for the worked examples, or whether I shall keep searching for a better one.

[itchyshin]

@ASanchez-Tojar - just looking at this now - can you upload the Rmd file here - so I can edit directly? Thanks - (@Yefeng0920 seems to have the file here but we need to merge all into one). I will chat with Yefeng soon to sort this out

[ASanchez-Tojar]

Here they go. Rmd + data, each in its corresponding folder. Let me know if you need something else. Also, I'm less overwhelmed this week than last week, so I can push this quicker, particularly at the beginning of the week. worked_examples.zip

[itchyshin]

@ASanchez-Tojar - an excellent job - I think we should go for the personality dataset as @Yefeng0920 's ecological one does not have phylo but yours would. I will try to get some outline together now - let's see but so far so good. Sorry for taking so long - many little urgencies - you will understand once you become a Prof. Thanks again.

[itchyshin]

Also, @ASanchez-Tojar and @Yefeng0920 - please do not touch the sutff in this repo - till I say please do. I will set up a Supporting Information folder here for all us to work on. Also, I may make a new issue with a to-dolist

[ASanchez-Tojar]

Attached the updated version including phylogeny plus a little of polishing here and there. I did not work on section 1.3.3 because I wasn't sure which moderators you'd like to add at the end for that meta-regression. The attached .zip now contains all necessary files for reproducing the results in the future (basically: all phylogenetic information, some of which could potentially change in future updates of the Open Tree of Life, which could then break the code (true story). That is why we should also provide the phylogenetic data that I added to the folder). Also, I know use the I2 function written by Dan Noble because that one deals with phylogenies and also provides 95% CI (now added). Let me know what else I could do tomorrow. Cheers. behavioural_syndromes.zip

[itchyshin]

OK - finally looking at this @Yefeng0920 says

moderators for dataset ft027: Disturbance_category + Biome + Species category (invader)

ft030: variance_component + trait_class

ft078: Country1 + Des_type + Veg

ft109: plantNPcr + Plant-microbial-relationship

ft027 is the most potential dataset based on your first expression

Probably use ft027 And @ASanchez-Tojar - to use ft044

[itchyshin]

@ASanchez-Tojar - I just gonna use R2 and I2 without CI for this - as MetaAidR's CI for I2 is not quite validated.

[itchyshin]

@ASanchez-Tojar - have a good read of this paper

https://ecoevorxiv.org/su4zv/

You fitted the model which we do not recommend (I think I made the same mistakes in the past) - here the latest version with a worked example:

Phylogenetic_multilevel_meta_analysis__a_simulation_study_on_the_importance_of_phylogeny.pdf

(I will fix this up - I am hoping to send an outline to you and @Yefeng0920 by the end of my Monday)

[itchyshin]

BTW - what you have done so far is excellent @ASanchez-Tojar - hope you do not mind - me chaning things around and deleting some bit

[itchyshin]

@ASanchez-Tojar - also note that I2_phylo is not equivalent to H2 - it is related tho Also I take out Q as it is meaningless without DF and DF is hard to get in a mixed model as you know

@Yefeng0920 and @ASanchez-Tojar - I am getting there but I will need one more day and I will arrange to meet you so we can go through the stuff together

[ASanchez-Tojar]

@itchyshin Thanks for all the comments, I'm learning a lot and that's the best way to start a week. Thanks! And yes, please, change anything you want and let me know how I can help implementing stuff. I'm here to help. Looking forward to it.