search

arviz-devs / arviz

Exploratory analysis of Bayesian models with Python
https://python.arviz.org
Apache License 2.0
1.61k stars 406 forks source link

Trying to understand quantile ESS plot for emcee chains #977

Open fbartolic opened 4 years ago

fbartolic commented 4 years ago

Thank you for building this excellent library!

I'm using Arviz to plot quantile ESS plots in order to check convergence properties of different samplers in the tails of the target distribution. When I make the plot for emcee output I get the opposite behaviour from what I'd expect, namely, the ESS is lowest in the bulk of the posterior and highest in the tails as seen here: image

I see the same pattern independent of whether I import the trace using the from_emcee method or if I manually construct an InferenceData object using a single walker.

Does anyone have intuition for why this is happening with emcee?

OriolAbril commented 4 years ago

I have a hunch on what is happening, but I don't currently have time to check this is actually what is happening.

The ESS estimates rely on the chains being independent, which is not really the case for emcee's walkers. However, for long chains, the estimate generally ends up converging to the real ESS value, hence the recommendation in emcee docs to use chains longer than 50 * tau. Both in the tutorial in emcee docs and in some extra examples I ran (link below) the autocorrelation time for short chains tends to be lower than the real autocorrelation, yielding higher ESS estimates (ESS is calculated as nchains * ndraws / tau).

image

My suspicion is that trying to use quantile ESS ends up triggering the same behaviour. The ESS estimates for extreme quantiles are not reliable and yield smaller autocorrelations. Using much more draws may solve the problem, however, I would probably advise against using quantile/local ess for emcee. At least until you have had to run some simulations to check is behaviour.

To read more on ESS calculation with emcee samples: https://discourse.mc-stan.org/t/does-effective-sample-size-estimation-require-independent-chains/10061

fbartolic commented 4 years ago

Thanks for the link to the Stan discourse discussion, I missed that.

I made the same plot including tail ESS estimates (code here) image

Solid lines are Arviz estimates of the autocorrelation time assuming walkers are independent chains, dashed lines are estimates constructed for each walker separately and then averaged which is what's suggested in the emcee docs.

It seems that both the median autocorrelation and the tail autocorrelation estimators are converging to a constant value, but the the autocorrelation in the tails is still consistently lower :S. I wonder if it's possible to check the calibration of the quantile ESS using an approach similar to the one suggested by Bob Carpenter in the Stan thread.

OriolAbril commented 4 years ago

Wow, this looks very good, and is quite surprising but not totally unexpected. As emcee walkers are not independent, there is no guarantee that ess estimates will converge to the real value.