stats.bpic does not calculate Bayesian Predictive Information Criterion (BPIC)

[ghost]

Docs for bpic in pymc3/stats.py state:

"Calculates Bayesian predictive information criterion n of the samples in trace from model Read more theory here - in a paper by some of the leading authorities on model selection - dx.doi.org/10.1080/01966324.2011.10737798"

The function calculates the information criterion from the stated paper correctly; however this is not the BPIC. The information criterion in the paper (as far as I know it does not have a name) is an attempt to strike a balance between the better bias correction of BPIC which overcomes the tendency of Deviance Information Criterion (DIC) to overfit, and the easy computability of DIC. As it turns out this information criterion doubles the penalty term of DIC.

The docs/function name should be updated with to avoid misleading users.

[aloctavodia]

Thanks @abish91 for opening this issue.

There is some reason do you want to use BPIC over WAIC/LOO? I have been thinking for a while that we should remove BPIC (and maybe DIC) and just keep WAIC and LOO.

[junpenglao]

I agree we should remove one (or both) of them as well, see also: https://github.com/pymc-devs/pymc3/issues/938#issuecomment-313425552

[springcoil]

I think we should keep WAIC and drop the others. And I think we are using PSS loo right?

On 13 Dec 2017 12:51, "Junpeng Lao" notifications@github.com wrote:

I agree we should remove one (or both) of them as well, see also: #938 (comment) https://github.com/pymc-devs/pymc3/issues/938#issuecomment-313425552

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc3/issues/2752#issuecomment-351382227, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8DiAKwc0_7nXfsVCQPIzSwG9vJTVz1ks5s_8hEgaJpZM4RAXzP .

[aloctavodia]

OK, so I will add a deprecation warning to BPIC and DIC and remove references to them in the docs. BTW @springcoil you are right now we have WAIC and PSIS-LOO_CV (loo for the friends), and also a couple of functions and plots to compare and average models based on them.

[ghost]

@aloctavodia There was no reason I wanted to use BPIC - I just noticed it was there and wondered how pymc calculated it.

[fonnesbeck]

Let’s definitely update or remove it if it does not do what it advertises.

[aloctavodia]

Closing since BPIC and DIC has been marked as deprecated and will be removed after 3.3 release.