Calibration/Refinement Decomposition of LOO Scores

stan-dev / loo

loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)

https://mc-stan.org/loo

Other

148 stars 34 forks source link

Calibration/Refinement Decomposition of LOO Scores #179

Open ParadaCarleton opened 3 years ago

ParadaCarleton commented 3 years ago

I think a breakdown of LOO scores into calibration/refinement would be interesting and useful to users. Mentioning @avehtari because Jonah mentioned you'd proposed adding this decomposition and said you might have some related work.

avehtari commented 1 year ago

@ParadaCarleton can you add here a reference to "calibration/refinement"

ParadaCarleton commented 1 year ago

It has now been 2 years and I have no idea what I was talking about then :smile:

But I think this was the relevant paper that inspired this suggestion, though I'm not 100% sure.