search

paul-buerkner / brms

brms R package for Bayesian generalized multivariate non-linear multilevel models using Stan
https://paul-buerkner.github.io/brms/
GNU General Public License v2.0
1.28k stars 184 forks source link

k-fold grouped #619

Closed jayrobwilliams closed 5 years ago

jayrobwilliams commented 5 years ago

Currently kfold with folds = "stratified" places observations in the same group in different folds. The development version of loo now includes a function loo::kfold_split_grouped that will construct folds such that entire groups are either included or excluded in the folds. I've modified kfold_internal to allow k-fold cross-validation that leaves entire groups out using loo::kfold_split_grouped.

I've also modified kfold_internal to return groupwise instead of pointwise elpd standard errors following @avehtari's demo for rstanarm here.

I've done some limited tests and so far it doesn't look like anything I've done has broken kfold_predict. The modified function can be viewed on my forked repo under the kfold-grouped branch. Would you have any interest in incorporating this functionality into brms?

paul-buerkner commented 5 years ago

Thanks! This is definitely an important addition to brms and I will surely implement it. Currently, the up-to-date version of kfold_internal is on the loo-compare branch, which I will merge into master once the new loo version is on CRAN (hopefully tomorrow or so). Once that is done, I will add "grouped" as an option in argument folds.

jayrobwilliams commented 5 years ago

Awesome!

paul-buerkner commented 5 years ago

This feature is now available on the loo-compare branch, which is to be merged soon.