arviz-devs / arviz

Exploratory analysis of Bayesian models with Python
https://python.arviz.org
Apache License 2.0
1.56k stars 388 forks source link

Unclear documentation of rank and weight in arviz.compare() #2305

Open imranashraf opened 6 months ago

imranashraf commented 6 months ago

Short Description

I am confused about how arviz.compare() reports the 'rank'. In the documentation it says:

rank: The rank-order of the models. 0 is the best.

Is this best decided taking into account both predictive accuracy as well as simplicity OR is it only based on the predictive accuracy of the models?

And the same question applies to the weight as well.

Relevant documentation or public examples

Here is the link to the documentation I am referring to.

aloctavodia commented 6 months ago

The rank is based on the elpd (expected log pointwise predictive density). The both the predictive accuracy and simplicity are already considered in the elpd. The same goes for the weights, but notice there are different methods to compute the weights, and stacking and BMA has different intepretations. Here you can read more details https://bayesiancomputationbook.com/markdown/chp_02.html#model-comparison