search

stan-dev / posterior

The posterior R package
https://mc-stan.org/posterior/
Other
167 stars 24 forks source link

[Feature] Add relative_eff function #280

Closed n-kall closed 6 months ago

n-kall commented 1 year ago

For some diagnostics, such as pareto-k diagnostics, we may want to calculate the relative efficiency (r_eff). The loo package has an implementation of this loo::relative_eff. The documentation states that it 'computes the MCMC effective sample size divided by the total sample size', but the output is not equal to ess_bulk(x)/ndraws(x) nor ess_basic(x)/ndraws(x), so perhaps the method in loo could be adapted and moved to posterior?

n-kall commented 1 year ago

I now see that the recommendation for calculating r_eff is to use the split-chain ESS ("[...] R{eff,MCMC} = S{eff,MCMC}/S is the estimated relative efficiency of the MCMC draws, and the effective sample size S_{eff,MCMC} for r(θ) is computed using the split-chain effective sample size estimate method" from Appendix H in Vehtari et al, 2022).

It seems that loo:::ess_rfun is not using the split-chain estimation, so I guess this issue is actually about whether a relative_eff function which is based on the split-chain ESS would make sense to include in posterior (and then potentially used by loo). Would it just be ess_bulk(x)/ndraws(x)? EDIT: The correct calculation is ess_basic(x)/ndraws(x).

n-kall commented 6 months ago

As relative eff should be quantity specific, I don't think it makes sense to have a single function for it. Closing this for now