search

FBartos / zcurve

zcurve R package for assessing the reliability and trustworthiness of published literature with the z-curve method
https://fbartos.github.io/zcurve
10 stars 1 forks source link

Option to change the default CI #7

Closed DominikVogel closed 2 years ago

DominikVogel commented 2 years ago

First of all, I want to thank you for this great package!

I have a question regarding the confidence intervals for the observed discovery rate (ODR) and the expected discovery rate (EDR). It would be great to test if EDR and ODR differ significantly. To do so, one could test if the 90% confidence intervals overlap. However, zcurve() only offers 95% CIs. Is there a way to get 90% CIs? The alpha options does not seem to help, since it changes the whole estimation.

FBartos commented 2 years ago

Hi Dominik,

Glad that you like the package!

The 90% CI for ODR should be directly obtainable from the prop.test function that is used to obtain the CI for the proportion of significant studies out of all observed studies (e.g., prop.test(x = 80, n = 100, conf.level = .90) for 80 out of 100 significant studies).

The 90% CI for EDR is a bit trickier since we calibrated the 95% CI to provide the appropriate coverage (by extending the lower and upper bound of the 95% bootstrapped quantile intervals for EDR by 5 percentage points, see out pre-print for more details).

You could theoretically obtain a similarly adjusted 90% CI by extending the 90% boostrapped quantile intervals, however, we did not test or assess the behavior for this modification. You can access the bootstrapped EDR estimates by:

m.EM <- zcurve(OSC.z, method = "EM", bootstrap = 1000)
m.EM$coefficients_boot[,"EDR"]                                        # vector of the boostrapped EDR estimates
quantile(m.EM$coefficients_boot[,"EDR"], probs = c(0.025, 0.975))     # 95% quantile interval

Cheers, Frantisek

DominikVogel commented 2 years ago

Hi Frantisek,

Thanks a lot for your quick response! I think I understood most of it :-). Just one question to make sure that I got it right: To get the same results as with summary(m.EM), I need to run quantile(m.EM$coefficients_boot[,"EDR"], probs = c(0.025, 0.975)) and subtract five percentage points from the upper bound and add 5 points to the lower point, right?

So when I want to get correct 90% CIs I run the following code and subtract/add five percentage points?

m.EM <- zcurve(OSC.z, method = "EM", bootstrap = 1000)
quantile(m.EM$coefficients_boot[,"EDR"], probs = c(0.05, 0.95))     # 90% quantile interval

Output:

       5%       95% 
0.1415391 0.6310292

Result: The correct 90% CIs are 0.092 to 0.681.

Or do I need to adjust for only 2.5 percentage points?

FBartos commented 2 years ago

Thanks a lot for your quick response! I think I understood most of it :-). Just one question to make sure that I got it right: To get the same results as with summary(m.EM), I need to run quantile(m.EM$coefficients_boot[,"EDR"], probs = c(0.025, 0.975)) and subtract five percentage points from the upper bound and add 5 points to the lower point, right?

Yes, that's correct indeed.

Result: The correct 90% CIs are 0.092 to 0.681.

That would be the analogous way of obtaining 90% CI, yes. However, I just want to point out that we did not test coverage of any other CI than 95% (and since we needed to correct the bootstrap with the +-0.05 addition I cannot guarantee the performance).

DominikVogel commented 2 years ago

Alright. Thank you very much for the clarification!