search

stan-dev / loo

loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)
https://mc-stan.org/loo
Other
150 stars 34 forks source link

Add hierarchical stacking #174

Open yao-yl opened 3 years ago

yao-yl commented 3 years ago

Given the new research progress on "hierarchical stacking", it would be useful to implement this new tool into the loo package.

This new tool needs more user customization: after a few models are fit, the user needs to

  1. Run pointwise model evaluation. For IID data, this is highly automated.
  2. Specify what variable that stacking weight should depend on. This cannot be automated.
  3. Come up with a stacking model. A template may look like https://github.com/yao-yl/hierarchical-stacking-code/blob/main/well_switch/optim.stan. But this model could be more complicated too.
  4. Specify hyper prior.

The challenge for putting this package into the present package lies in that

  1. Extra dependence on stan.
  2. May even need stanglm/brms syntax to write stacking models.
jgabry commented 3 years ago

Yeah it would be nice to support this. Regarding the extra dependence on Stan, would the loo package have to run the Stan code from step 3 for the user?

yao-yl commented 3 years ago

Perhaps in this case the loo package can serves as a stan-model-generator: compute pointwise loo, suggest a default stacking model, and generate a stan file. Then user can run/modify this generated stan file on their own.

Is it possible that there could be dependence on rstan, but such dependence is only checked/brought when certain loo::function is called?

jgabry commented 3 years ago

Perhaps in this case the loo package can serves as a stan-model-generator: compute pointwise loo, suggest a default stacking model, and generate a stan file. Then user can run/modify this generated stan file on their own.

Interesting idea. Would be curious to know what other people think of this option, but yeah that's a possibility.

Is it possible that there could be dependence on rstan, but such dependence is only checked/brought when certain loo::function is called?

In theory yes, but RStan isn't a typical package so we'd need to check if that works (but I think it should). Do you need to call rstan::log_prob or rstan::grad_log_prob? If not then I think it would actually be easier to do this with CmdStanR than RStan.