search

stan-dev / loo

loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)
https://mc-stan.org/loo
Other
148 stars 34 forks source link

Add R2 #201

Open sims1253 opened 2 years ago

sims1253 commented 2 years ago

It would be nice if the loo objects weren't hardcoded to use elpd. For a study I wanted to include rmse and R2 and had to build fake loo objects in order to use the loo_compare functionality. Code is here.

avehtari commented 1 year ago

I had missed this issue Thanks for the code! PRs #202 and #203 are adding more utilities/losses, but we still need to add options for loo_compare to know which utility/loss to use brms part of the code needs to go to brms package

avehtari commented 1 year ago

@sims1253 do you need something else than what is in the PRs mentioned above, or can we close this issue?

avehtari commented 7 months ago

We do have rmse, but we're still missing R2 (which is available in brms and rstanarm, but not as plain loo package version)

sims1253 commented 7 months ago

...which is available in brms

At least the brms version does not use psis, which is why I wrote the version from my pr.

avehtari commented 7 months ago

brms loo_R2 does use psis based on the code https://github.com/paul-buerkner/brms/blob/d1a050803670f4f4647324ce9a84780591a40692/R/loo_predict.R#L202

sims1253 commented 7 months ago

True. Sorry, I'm currently on vacation and only have my phone and only found the Bayes r2 function. Maybe it was about the pointwise estimate for the SE then that we built the new version. I don't quite remember to be honest.