search

arviz-devs / arviz

Exploratory analysis of Bayesian models with Python
https://python.arviz.org
Apache License 2.0
1.61k stars 407 forks source link

Is there a LOO R2? #1931

Open hadjipantelis opened 3 years ago

hadjipantelis commented 3 years ago

I am just playing with ArviZ and it is great. There is a r2_score to get Bayesian R²; is there a LOO equivalent akin to the one discussed in https://avehtari.github.io/bayes_R2/bayes_R2.html or in loo_R2 for brms?

OriolAbril commented 2 years ago

Missed that message somehow. We don't have that yet, and I'm not too familiar with the method so I'm not sure if it should be here or at bambi cc @aloctavodia @tomicapretto what do you think?

Also, would you be interested in contributing it to either ArviZ or bambi @hadjipantelis ?

tomicapretto commented 2 years ago

It sounds interesting, and it looks like something we could have in Bambi. However, I'm not very familiar with the method and it would require me to study it a little.

I can add it to our TODO list, but I don't think we're going to tackle it very soon.

hadjipantelis commented 2 years ago

Hmm... OK, @OriolAbril.

Would you guys prefer this in bambi or arviz? (Just so I know from which repo to clone.) Downstream I would need a bit more feedback regarding where this functionality needs to reside module-wise, testing, etc.

tomicapretto commented 2 years ago

I think ArviZ has a much wider adoption than Bambi. However, as far as I've read so far, you need more information about the model for which you're computing R^2. And in Bambi you have all that information so it seems like a better place.

ahartikainen commented 2 years ago

What are the needed parts for loo_r2?

tomicapretto commented 2 years ago

I'm not sure. But the code in brms seems to do different things for at least two different families of regression models. https://github.com/paul-buerkner/brms/blob/master/R/loo_predict.R

hadjipantelis commented 2 years ago

@ahartikainen: Reading through brms::loo_predict the immediate thing missing in arviz is loo::E_loo functionality (that is assuming that arviz.psislw can replicate the output of loo::psis).

@tomicapretto: It seems to me that the implementation errs for unordered categorical models and/or gives some rather strong-worded warnings if used with ordinal families. Family type doesn't look like to be a show-stopper here. (Most people would probably look twice at an R^2 for a non-continuous variable anyway...)

OriolAbril commented 2 years ago

It looks like E_loo or equivalent should go in ArviZ and use the data in posterior predictive group. loo_predict (and similars) would go in bambi (if the posterior predictive group is not present, ArviZ can't populate it, bambi can).