search

arviz-devs / arviz

Exploratory analysis of Bayesian models with Python
https://python.arviz.org
Apache License 2.0
1.6k stars 398 forks source link

Explain plot interpretation in example gallery #2078

Open michaelosthege opened 2 years ago

michaelosthege commented 2 years ago

I often visit the example gallery & tell others to do so to learn new visualization techniques for interpreting my MCMCs.

The pages show how to create each plot & how it looks like which is great.

It would be even better if there was a short summary on how to interpret the plots:

Bonus points for "this is good" and "this is bad" examples.

I'm selflessly making this docs feature request because I have no idea how to read the plot_ppc figure.

Also after reading https://python.arviz.org/en/latest/api/generated/arviz.plot_ppc.html#arviz.plot_ppc I can't tell: Is the plot shown in the gallery an example of a good, or a bad fit?

aloctavodia commented 2 years ago

Very timely comment as we are actively working on improving the documentation. One option to solve this issue will be to link plots in the gallery to sections in https://github.com/arviz-devs/Exploratory-Analysis-of-Bayesian-Models (WIP).

OriolAbril commented 2 years ago

I would also document this in EABM website and not in the API nor example gallery pages. First and foremost because the interpretation is not a property or concept tied to the ArviZ functions. Multiple plots are interpreted in the same or very similar way and at the same time, a single "arviz plot" can be interpreted in completely different ways depending on the arguments given to it. Also, interpretation will be common to python, julia, R, javascript or any language used, so I think it is best to keep it independent and link to the right page from the python docs to cover this issue.

However, to still try and keep docs attractive and as useful as possible as a quick reference resource, we could add an "interpretation" directive after the function description in the API docs. We discussed in generaly with @aloctavodia to try and use a progressively deeper and more extensive page style in EABM, so the start for these pages could be a single sentence on how to check if the plot is ok (in the plot_ppc case for example could be "the curve corresponding to the observation should be indistinguishable from the multiple posterior predictive curves") which could also be explicitly copied to the api docs so they look like (using numpydoc section names)

short description

extended description

interpretation directive (as last element of the extended description). 
An admonition highlighted box with a sentence and a link to EABM

Paramenters

Returns 

...

Also, this is tangentially related to #2076 in trying to integrate the example gallery and the API docs better. We could start by making the link to the API page much more prominent.

canyon289 commented 2 years ago

@sarinac and I talked about this literally today. She was very confused about forestplot, rightly so its like 4 different things in one between MCMC sampling checks, parameter estimates, model comparisons, and hierarchical checks. +10000 that we explain this

sarinac commented 2 years ago

I like the structure of what to add (what am I looking at? What's the good/bad to look out for?). In my opinion, I think it's better to put these explanations in the example gallery (more than API docs) because an actual example is close to how you would communicate data to your audience. Plus, it eliminates most of the uncertainty around how to read a chart, and once you know how to read it then you'll know how to use it.

For example: for plot_density, the description can be something along the lines of (but as specific as)

This chart shows the 94% high density interval for the theta parameter determined for each of the eight schools in the centered_eight and non_centered_eight datasets. For Choate, the mean from the Centered data is less than the mean from the Non Centered data.

When I first looked at the example gallery without reading the API, I did not know this displayed 94% and the means (defaults in the code). This will probably be more useful for the more complicated graphs.

OriolAbril commented 2 years ago

I think the root issue is also related to the different expectations each of us has from the documentation, made worse by the fact the the documentation in general still needs a lot of work and many pages and resources are still missing.

In my opinion, both the example gallery and the api docs are bad places to explain how to interpret the plot. For me, both the gallery and the api are reference content, places for dry and succint description. They are also pages that are not written to be read as much as consulted. The api pages describe which objects are part of the ArviZ library, what arguments to they take, the types and defaults of such arguments... the example gallery is a visual reference, it visually shows what plots are supported by arviz so people can get a quick idea of the different available plots, can check if that nice plot they saw in a paper recently but whose name they don't know is there in one of the thumbnails...

Consequently, I think example plots for the example gallery should be chosen for visual reasons and "coverage" mostly (hence my suggestions in #2080 for example), and example plots in the API docs should be chosen for "argument description" reasons (i.e. add a plot showing the right format of the lines argument in plot_trace).

I also think adding the "what am I looking at?" description to the example gallery would be great, something along the lines of the plot_density description above proposed by @sarinac. plot_density is a relatively simple plot, but still, to be able to correctly interpret it one needs to know what is a probability density function, what is a KDE, even what is the kde algorithm used by arviz, but it doesn't have "good" or "bad" examples so it is simpler than other plots like plot_ppc.

On the other hand, how to interpret a plot, especially in cases where there are "good" and "bad" examples, is not something I see can be done in a descriptive manner. It needs to explain the reasoning behind the choices made, how do the multiple elements relate to each other so that users can use that reasoning on their own plots. In the case of plot_ppc, we can write a 1-2 sentence description (like what I added in the previous comment: "the curve corresponding to the observation should be indistinguishable from the multiple posterior predictive curves") so that if your plot checks this you are done with the plot; but that is barely the tip of the iceberg.

Explaining how to interpret the plot and make the most out of it needs significant explanation on is background and elements, it needs many examples of good and bad plots, and detailed explanation on why the bad plots are bad, I don't see how adding that explanation on an example gallery page with a single plot will help users. I did write about loo_pit and plot_ppc a while ago, and I think the quality and understandability of the blog post would decrease significantly with each example plot that is removed from it.

Also, not directly related to the latest comments but something we need to take into account given our limited capacity and time constraints when it comes to working on and maintaining ArviZ. The interpretation of plot_ppc can be tied to the interpretation of plot_loo_pit, and also to the interpretation of plot_bpv, and using them together is much more powerful than using them independently. So including the "how to interpret the plot" explanation in either api or example gallery pages wouldn't only mean a significant ammount of duplication (with the consequent maintenance burden) but it would also fail to properly cover the workflow-like use and synergies between the plots (unless again we duplicated everything again).