pymc-devs / pymc-examples

Examples of PyMC models, including a library of Jupyter notebooks.

https://www.pymc.io/projects/examples/en/latest/

MIT License

259 stars 211 forks source link

update Model Averaging example to run on pymc v5 #67 #600

Open alporter08 opened 7 months ago

alporter08 commented 7 months ago

update Model Averaging example to run on pymc v5

https://github.com/pymc-devs/pymc-examples/issues/67

[x] Notebook follows style guide https://docs.pymc.io/en/latest/contributing/jupyter_style.html
[x] PR description contains a link to the relevant issue:
- a tracker one for existing notebooks (tracker issues have the "tracker id" label)
- or a proposal one for new notebooks
[x] Check the notebook is not excluded from any pre-commit check: https://github.com/pymc-devs/pymc-examples/blob/main/.pre-commit-config.yaml

Helpful links

https://github.com/pymc-devs/pymc-examples/blob/main/CONTRIBUTING.md

:books: Documentation preview :books:: https://pymc-examples--600.org.readthedocs.build/en/600/

review-notebook-app[bot] commented 7 months ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

alporter08 commented 7 months ago

Hi @OriolAbril @cluhmann , I will look over the text to make sure it syncs with the code changes and look at the Jupyter Style guide. I also noticed there is an issue currently with the colors and axes in the final plot so will investigate that.

cluhmann commented 7 months ago

Great, thanks again for your the help!

alporter08 commented 7 months ago

@OriolAbril @cluhmann I got the code to work with the new PyMC version, and the results match the old notebook. I also edited some of the text for clarity.

One point to mention however is that the notebook is supposed to demonstrate that the weighted model approach has more uncertainty than an individual model. But looking at the hdi, it is actually pretty much equal for the weighted approach and the compared individual model. I think this is because the model we are comparing the ensemble method to has a weight of one, and the other two models are effectively zero weights. So the ensemble method is really just one model, which we are comparing to itself. So maybe that should be specified in the commentary? Right now it says that the weighted model uncertainty is larger than the single model uncertainty, which isn't really the case.

OriolAbril commented 7 months ago

Given this is taken from an example in the book, we should reproduce it as it is. Then if we deem necessary, add extra comments. I have just checked, and the book uses the pseudo-BMA method to compute the weights. We should then use the method argument when calling az.compare, and use those weights to generate the weighted posterior predictive samples.

Afterwards, it is probably a good idea to use az.compare with the stacking method and use the results+reference to the section on stacking to explain why it is ArviZ's default now. What do you think? Tagging also @aloctavodia who authored the notebook