Add WAIC and PSIS-LOO

[carlkl]

PSIS-LOO

Pareto smoothed importance sampling. Aki Vehtari, Andrew Gelman and Jonah Gabry (2016). https://arxiv.org/abs/1507.04544 see also https://github.com/avehtari/PSIS

Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Aki Vehtari, Andrew Gelman, Jonah Gabry 29 June 2016 http://www.stat.columbia.edu/~gelman/research/unpublished/loo_stan.pdf

WAIC http://watanabe-www.math.dis.titech.ac.jp/users/swatanab/waicwbic_e.html see also http://www.stat.columbia.edu/~gelman/research/published/waic_understand3.pdf

Both methods use the MCMC chain for evaluation.

[Samreay]

Hi Carl,

These both look useful to add, so will try and get on them when I have time. Have my PhD confirmation coming up in a few weeks, so won't be a lightning fast addition though, so if you or your group want it in asap please feel free to make a pull request.

Cheers

[carlkl]

It is not urgent (at least for me). My feeling was, it should be added as comparison to AIC/BIC/DIC. Recently I found an pymc3 implementation here: https://github.com/pymc-devs/pymc3/blob/master/pymc3/stats.py

Good luck for your PhD.

[Samreay]

Ah yeah, thats a nice compact implementation that I can credit over, cheers. Out of curiosity, what group do you work with?

On Wed, Apr 5, 2017 at 10:12 PM, carlkl notifications@github.com wrote:

It is not urgent (at least for me). My feeling was, it should be added as comparison to AIC/BIC/DIC. Recently I found an pymc3 implementation here: https://github.com/pymc-devs/pymc3/blob/master/pymc3/stats.py

Good luck for your PhD.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Samreay/ChainConsumer/issues/19#issuecomment-291842101, or mute the thread https://github.com/notifications/unsubscribe-auth/AGb5Wd3OzW9qSV6zv16pMWtrUsqJgQw8ks5rs4UWgaJpZM4M0GsS .

[carlkl]

Your Q: accelerator physics.

BTW: bayesian analysis ist just a hobby project for me, for prokrastination and self-learning.

[Samreay]

Ah fair enough. And thats interesting, didn't realise ChainConsumer had done outside of the Dark Energy Survey yet!

On Wed, Apr 5, 2017 at 10:31 PM, carlkl notifications@github.com wrote:

Your Q: accelerator physics.

BTW: bayesian analysis ist just a hobby project for me, for prokrastination and self-learning.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Samreay/ChainConsumer/issues/19#issuecomment-291846774, or mute the thread https://github.com/notifications/unsubscribe-auth/AGb5WViyWEWaiGPKNCCPFGPn7Xb7kQUWks5rs4mrgaJpZM4M0GsS .

[carlkl]

BTW: There is a nice comparsion (in german): https://epub.ub.uni-muenchen.de/31977/1/BA_Erdemir.pdf with an easy to understand explanation why WAIC/WBIC should be preferred over AIC/BIC. I didn't get the point with the original WAIC papers.

[Samreay]

Excellent, time to brush up on meine Deutsch. Danke!

On Wed, Apr 5, 2017 at 10:44 PM, carlkl notifications@github.com wrote:

BTW: There is a nice comparsion (in german): https://epub.ub.uni-muenchen. de/31977/1/BA_Erdemir.pdf with an easy to understand explanation why WAIC/WBIC should be preferred over AIC/BIC. I didn't get the point with the original WAIC papers.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Samreay/ChainConsumer/issues/19#issuecomment-291849708, or mute the thread https://github.com/notifications/unsubscribe-auth/AGb5WT-tH6Nye1gg6ozNZ8IiGeySTbi9ks5rs4yWgaJpZM4M0GsS .

[Samreay]

Closing this as an old issue I never got the time to work on, sorry Carl.