search

StatisticalRethinkingJulia / StatsModelComparisons.jl

Model comparison functions for statistical models as used in StatisticalRethinking. Version 2 will likely use or be based on ParetoSmooth.jl.
GNU General Public License v3.0
6 stars 3 forks source link

Suggestion: Remove AIC, BIC, and DIC #9

Open ParadaCarleton opened 3 years ago

ParadaCarleton commented 3 years ago

AIC, BIC, and DIC have all been fully supplanted by WAIC, WBIC, and LOO-IC, and there's no longer any real reason to use them. I would suggest removing them from the package to avoid confusing beginners. Someone seeing them used in the package may be led to believe that using them is a good idea, that they provide information different from that provided by the newer information criteria, or that there must be some reason they're in the package. BIC is especially problematic: The name simultaneously gives new users the false impression that the BIC is actually Bayesian and that WAIC/LOO-IC are not Bayesian.

It might be a good idea to implement WBIC to have a replacement for BIC, since BIC has a different use case from the other information criteria in this package (asymptotically maximizing the probability of selecting the correct model, rather than maximizing the expected predictive accuracy of the model). I would also suggest referring to it as the Watanabe-Schwarz Information Criterion to avoid giving the impression that WBIC is "Especially" Bayesian.

goedman commented 3 years ago

Carlos, can you describe how you compare the weights between Julia and R. Just for my education.

ParadaCarleton commented 3 years ago

Carlos, can you describe how you compare the weights between Julia and R. Just for my education.

Just using count(r_weights .≈ jul_weights), which indicated all the weights were approximately equal.

Makes little difference. I think, given above your check on the weights, your code at this point is more trustworthy than the old code.

The weights and pareto_k values are the same, but the code in Bombe has an error somewhere that I'm trying to find -- the LOO-CV is off from the value returned by R, by about 4. I suspect we're both making a similar mistake somewhere in our Loo code.

goedman commented 3 years ago

Noticed you just did a major addition.

Thanks for above suggestions, this gives me a better target to look for if the new version doesn't solve it!

Thanks for this work, in my opinion a major step forwards.

ParadaCarleton commented 3 years ago

Noticed you just did a major addition.

Thanks for above suggestions, this gives me a better target to look for if the new version doesn't solve it!

Thanks for this work, in my opinion a major step forwards.

Yep, I found the mistake and fixed it. Because the LOO code ended up being smaller than I expected, I've decided to incorporate it into ParetoSmooth.jl instead of spinning it off. The code can be found here. However, I think it doesn't run at the moment -- I'm trying to sort out a bug that keeps me from running tests on my own computer...