search

paul-buerkner / brms

brms R package for Bayesian generalized multivariate non-linear multilevel models using Stan
https://paul-buerkner.github.io/brms/
GNU General Public License v2.0
1.27k stars 183 forks source link

Time-series cross validation #608

Open ghost opened 5 years ago

ghost commented 5 years ago

Currently, loo and kfold facilitate model comparison for models without a time dependency. To my knowledge, brms does not support time-series cross-validation in the same way. Are methods such as approximate leave-future-out cross-validation (http://mc-stan.org/loo/articles/loo2-lfo.html), fixed window rolling forecast, or increasing window rolling forecast features that might be implemented in the future? For instance, I'm thinking kfold might be augmented to include support for rolling forecasts through (maybe) the folds option. Perhaps something along the lines of: folds = "rolling". Further arguments that determine the initial window size, horizon, and whether the window is fixed would add further value. Hoping this post isn't out of place. Thanks!

paul-buerkner commented 5 years ago

We actually have a full paper on approximate leave-future-out cross-validation available on arXiv now (https://arxiv.org/abs/1902.06281) and I am definitely planning on implementing this, and perhaps other methods in brms as soon as I find the time. @avehtari any thoughs on this?

jgabry commented 5 years ago

Definitely in favor of implementing the LFO-CV stuff at some point. I think we should try to build as much of the time series stuff into the loo package as possible so that all the R packages can share to the extent possible.

paul-buerkner commented 5 years ago

I agree. @topipa and I are doing the same with the moment matching loo method, that is abstracting away all the specifics so that the main algorithm can go into loo.