search

mpt-network / MPTmultiverse

An R package for the comparison of MPT analysis approaches
3 stars 6 forks source link

Monte Carlo standard error for posterior predictive p values #5

Open crsh opened 6 years ago

crsh commented 6 years ago

Hi. I was wondering whether we want to plot the Monte Carlo standard error or confidence interval for the estimated posterior predictive p values (PPP) in the summary plot. This would give us a sense of how reliable that estimate is in terms of a dichotomous decision about model fit (as this is often how this is used in practice). I'm happy to add this. All I would need is the effective sample size for the sampling statistic used to calculate the PPP. Is this stored somewhere in the MPTmultiverse objects?

danheck commented 6 years ago

This information is currently not available in the fitted object. The MCMC standard errors are currently only available for the parameters.

This is also due to the fact that TreeBUGS does not provide SEs for the p-values as output. However, it might be worth adding that (however, I have not yet seen this in any paper). How exactly would you compute the SE? Simply by: SE(ppp) = sqrt(ppp*(1-ppp) / n_eff) ?

crsh commented 6 years ago

Yes, as per p. 267 in Gelman, et al. (2015). The confidence interval could then be calculated either asymptotically by relying on CLT or as upper bound by multiplying the asymptotic interval by 2.3 (Rosenthal, J. S. (2017). Simple confidence intervals for MCMC without CLTs. Electronic Journal of Statistics, 11(1), 211–214. https://doi.org/10.1214/17-EJS1224). Do you happen to know whether it's sensible to rely on CLT in this case?