search

FBartos / zcurve

zcurve R package for assessing the reliability and trustworthiness of published literature with the z-curve method
https://fbartos.github.io/zcurve
10 stars 1 forks source link

relationship between z value and replication rate #18

Open Yefeng0920 opened 10 months ago

Yefeng0920 commented 10 months ago

Hey @DominikVogel @FBartos , I am addicted to your method. Just curious about whether there is a way to construct the relationship between z value and replication rate. I am meant to make a plot with x-axis as the z value and the y-axis as the replication rate estimate. My main point is that in the case of non-normal distributions, it is not that meaningful to calculate the average power or the so-called expected replication rate. Rather, we should visualize the relationship between z value and the replication rate estimate

FBartos commented 10 months ago

I think Ulrich Schimmack did some work along these lines, I will let him point him here.

DominikVogel commented 10 months ago

Sorry, I have no merits in developing the method or the package. All fame should go to @FBartos, Ulrich Schimmack, and others.

Yefeng0920 commented 10 months ago

@FBartos Thanks for bringing Ulrich Schimmack and would like to look at his solution. Another relevant question I have is that the replication rate derived from the z-curve seems to be an observational level estimate that does not account for sampling error variance. Because z = mu_observation / sd, while mu_observation is often overestimated. And the power calculated using this way is a kind of post-hoc power or observed power.

FBartos commented 10 months ago

I'm not sure I followed your question sampling error question correctly. The common problem with post-hoc power is a) selection on statistical significance and b) the high uncertainty of power of a single study. Z-curve handles a) by modeling only statistically significant z-scores and accounting for selection on significance via truncated likelihood and b) by aggregating across many test statistics. Both z-curve papers explain this in more detail.

Brunner, J., & Schimmack, U. (2020). Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology, 4. Bartoš, F., & Schimmack, U. (2022). Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology, 6.