search

bbolker / broom.mixed

tidy methods for mixed models in R
229 stars 24 forks source link

Change default value of CI/HDI for Stan-/Bayes-Models? #40

Open strengejacke opened 6 years ago

strengejacke commented 6 years ago

The authors of rstanarm (see ?rstanarm::posterior_interval) or John Kruschke (see Kruschke JK. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan. 2nd edition. Academic Press, 2015, pp183-85) recommend reporting 90% credible intervals for models with less than 10.000 samples, instead of 95%.

Since the default of modelling functions like brm() or those functions in rstanarm are 4.000 samples, would you say it make sense to set the argument conf.level in tidy() to .9 instead .95?

bbolker commented 6 years ago

I see the point, but I think I'm more concerned about consistency (to the extent that it's possible!) across model types. (I'm a little bit surprised that it takes that big a sample to get reliable 95% CIs - is that counting samples rather than effective sample size, so that ESS might be << 10,000 ??) If it were possible I would want to do something so that users would get a warning if they tried to combine tidied objects with different confidence levels, but I think that's too hard ...

strengejacke commented 6 years ago

Kruschke says

For reasonably accurate and stable estimates of the limits of the 95% HDI, an ESS of 10,000 is recommended.

Maybe, if you want to address this issue in any way, the least complicated way is just to print a message when the default (i.e. .95 CI) is returned for Stan-Models - but maybe it would be even enough just to add a Note to the docs.

This is rather a more "educational" or "awareness" issue, maybe comparable to missing p-values in the lmer()-output.