Organize example gallery into sections and re-scale images

[sarinac]

Tell us about it

Current State: The ArviZ example gallery currently lists plots in alphabetical order (based on API name), but there are a few critiques:

1. It gets difficult to browse or search by visual where there are ~50 graphs to browse through. This makes discovery pretty difficult (users might think a very specific chart doesn't exist in ArviZ because they can't find it easily). a. Also, some charts (az.plot_dist_comparison) from the API Reference might be missing from the example gallery?
2. Some plots are hard to recognize by picture (e.g. "Compare Plot", "ELPD Plot", and "Quantile MCSE Error Plot" are too zoomed in or not cropped optimally).
3. The hover event is not helpful. For example, if I hover over the "Autocorrelation Plot" tile, I see the text overlay "example_plot_autocorr".

Recommendation:

1. Group all the current charts by use case.
   • Some great examples of galleries that do this are matplotlib and python graph gallery. Both of these examples have a section header followed by relevant charts, so that it helps break up the information and makes it easier for the user to digest. For Matplotlib, you can browse sections using the right panel, which would also be great for ArviZ.
   • See bottom Thoughts on implementation section for a starting point on how I would section these, but I think it would great to get multiple opinions on the organization.
2. Resize the images of example charts to fit in the thumbnail entirely (probably edit doc/sphinxext/gallery_generator.py file). Most charts in the example gallery should default to figsize=[8, 6] wherever makes sense (e.g. not "Separation Plot" because it makes more sense as a thin bar).
3. Change hover to show text from documentation or example from gallery. For example, if I hover over the "Autocorrelation Plot" tile, seeing the text "Bar plot of the autocorrelation function for a sequence of data" is more helpful. FYI Matplotlib does this with a tooltip instead of overlay. See screenshot below:

Thoughts on implementation

1. Grouping charts by use case- are some sections I can think of but definitely open to other ideas. With my current approach, the "Distributions" section is pretty overloaded, and I'm not sure which section each graph belongs to.
   • Distributions - density, dist, dot plot, trace plots
   • 2D Distributions - 2D KDE plots
   • Correlation - autocorrelation, plot HDI
   • Inference - bpv, posterior plots
   • Style Sheets (at the end)
2. Here's an example of showing an explanation on the hover event.

[OriolAbril]

Thanks! I love the ideas!

I have no idea how hover things work, but we can add custom classes to the elements when generating them and we can also load custon css and even js content to the website (being careful on the maintenance burden we are assuming when doing this). The gallery_generator is a bit messy, but we can go through it at some point so explain what is happening at each step and how does this integrate with the website generation with sphinx process (I have no idea how familiar you are with sphinx, if you already saw this clearly while reading the script ignore the comment).

My main concern with the hover is avoiding duplicating information as much as possible. In my opinion the example gallery is first and foremost a visual reference page, not a how-to use the plots. But it would be good to have better integration with the API docs (name/text based reference page) instead of showing only a link, not really sure how though

Most charts in the example gallery should default to figsize=[8, 6]

I think this can be part of the style used (in matplotlib, would have to look for options with bokeh whose plots generally overflow the screen). I think it would also be perfectly ok to have a specific examples style with things like this we might not want to add to the general packaged style.

Grouping charts by use case- are some sections I can think of but definitely open to other ideas.

I would use different groupings. Still not completely clear, but here are some ideas. Warning, I think it is perfectly fine to have some calls to the same arviz function in different groups, examples below.

• Mixed (or summary or something similar) plots. Plots that combine into a single figure multiple smaller/atomic plots. Main ones are plot_trace and plot_forest (when plotting the rhat and ess).
• Distributions (we could have 1-2d subsections, but given how arviz is geared towards bayesian models where we only ever visualize marginals, I am not sure it is a very relevant distinction): plot_dist, plot_kde, plot_ecdf, plot_posterior, plot_density (single idata input), plot_dot, plot_pair, plot_violin, plot_forest
• Distribution comparisons (I think it is different enough to be its own section): plot_density (with multiple idata input), plot_dist_comparison, plot_pair (overlay version), plot_violin (single side version), plot_ecdf (with reference samples and bands, I think it is available, not sure if it has an example though), plot_forest (multiple models)
• Inference diagnostics: plot_autocorr, plot_ess, plot_mcse, plot_energy, plot_rank, plot_parallel, plot_pair (with divergences),
• Regression-timeseries: plot_hdi, plot_lm, plot_ts
• Model comparison: plot_compare, plot_elpd, plot_khat
• Model checking and criticism: plot_ppc, plot_bpv, plot_loo_pit, plot_separation
• Styles

Extra idea: How about adding extra pages to the example gallery that don't necessarily use an ArviZ plotting function directly? It might be interesting. For example we could add something like https://github.com/arviz-devs/arviz/discussions/2075 to the regression/timeseries section to showcase it.

[sarinac]

Hey @OriolAbril - thanks for the feedback! I have some additional comments:

1. CSS in example gallery (for hover text) - I can make changes to CSS, and my preference would be to move it to an external CSS file.

   • Right now, I believe the gallery_generator is creating this CSS internally in the HTML of the example gallery page. ArviZ already has a custom.css file (linked in conf.py(sphinx doc)) that we can add to. I would probably add a "example-gallery-" prefix to all the CSS classes (so they only affect the example gallery objects) and update the class names in the gallery_generator here.
   • I think we can get by without additional JS :)
   • For the hover event, I am not sure what description is good, and I agree with your point on avoiding duplicate descriptions / serving as visual reference. Currently, I do not think the text shown on hover (the "example_{module_name}") is that helpful or consistent with anything else in the documentation. So a low-lift improvement would be to get rid of this hover text. Instead, on the hover event, I can put some border/shadowbox visual that'll make it look like the graph is clickable. I can play with the css on this.
     • Depending on how the API doc structure changes we could have the option to add some description text on hover, but I haven't explored the lift on that.

2. Resize images to fit in the thumbnail - I think I can accomplish this with CSS, and it'll remove some complexity from the gallery generator. I looked at matplotlib's example gallery, and they do not crop their images to be a square. I can do this at the same time after item 1 (move css externally) because it involves editing the HTML/CSS. That might mean we don't need the create_thumbnail function inside the gallery_generator because it scans through the matplotlib example py files and crops them. We can also remove the _thumb: .6, .5-like settings inside the py files too.

3. From a developer standpoint, do you think it's better to have examples in mutually exclusive groupings or tags that can overlap? Both can reference the same ArviZ function multiple times. I'll also elaborate on these 2 options in case if I am describing this badly:

   • Option 1 (mutually exclusive grouping): For example, there will be files examples/model_checking/matplotlib/plot_trace.py, examples/model_checking/bokek/plot_trace.py, examples/mixed/matplotlib/plot_trace.py, and examples/mixed/bokeh/plot_trace.py if you want to show plot_trace for some mixed or model_checking category. This does kind of duplicated unless the plot is customized (e.g. having different arguments or data).
     • In my head, this is the cleaner and more intuitive approach which I like. However, if future plotting functions are created to cover multiple use cases then multiple example files will be required (although variations are nice).
     • Matplotlib uses this approach (file structure here). They used some other sphinx extensions (sphinxext.gallery_order) to organize/order the example gallery.
   • Option 2 (tags that overlap): For example, plot_trace will have "model checking" and "mixed" tags in a single example py file. I think this can be implemented in the docstring (lowest effort) because the gallery_generator runs some regex to extract these values. The downside is that it's hard to catch typos if the tag is misspelled and gets omitted/looks bad.
   • Finally, with either approach, I think showcasing other useful non-ArviZ functions would be nice to fit the ArviZ users' needs/inspirations. We can also assign it to a certain existing category or tags. Is this already being done in the example gallery? I don't see a restriction with non-ArviZ functions in the gallery generator, and if the result of identifying non-ArviZ functions is consistent with this attribute (apiname) then I can also add some CSS to tag it as a non-ArviZ/inspirational/other in the example gallery thumbnails.
   • Also, @canyon289 had an idea on organizing categories based on steps taken in a workflow. I think the groupings you two came up with have decent overlap. Regression Timeseries and Styles could be their own section, but the rest could probably be combined in some grouping according to the workflow (e.g. model comparison, checking; distributions section could be assigned to different steps).

[OriolAbril]

1. Moving the css to our custom.css would be great, thanks.

   It might also be possible to simplify it using grids and/or cards via sphinx directives.

   • It is not as flexible as using raw html though, so using sphinx things is not a hard constraint and look and functionality should come before that, if possible to do the same thing we want without raw html it would have some advantages
   • I have edited all of these files at some point, but I don't really know what I'm doing. I do remember though, that when we updated the theme we use for our docs (from one built on bootstrap3 to the current one using bootstrap4 part of the example gallery broke, I am not sure how much dependended or still depends on bootstrap classes in addition to the custom css but using sphinx directives for the grids would mean we don't have to worry about this in the future if we need to update again bootstrap (there is little raw html anymore, so it might even be using bootstrap5 already) or theme. I can help with sphinx-related things and using sphinx doesn't mean not being able to use custom classes.

   I have a vague idea about how css works, no idea whatsoever about JS, so this is great news :)

   Agreed that the current hover text is not useful. And the images don't have alt text either. It might be a good idea to start with a dry and concise description of the generated graph to then use as hover and/or alt text. That would also be something that is specific to the example and not to the function/type of plot, and even if we manage to restructure everything better, should still be available as alt text at least once we enter the example page to make the docs more accessible.

   The border efect sounds good, the card elements I mentioned above already have those, so if we got rid of the span html element I think using a card or grid-item-card would achieve the same visual and include the link effect with minimal work on our side.

2. Showing the full plot instead of a cropped thumbnail would be much better conceptually and if we can get rid of the thumbnail generation code even better.

3. I think that depends on the subsections we decide to use.

   • If we use something along the lines of my proposal above, then we'd have sections that are mutually exclusive when it comes to examples not when it comes to functions. That is, all examples are unique and generate slightly different images, even if they use the same arviz function, so we can't have a py file that is shared between sections because the example using plot_trace that goes in the mixed section uses different arguments and/or data than the one that goes in the model checking section
   • If we use something more close to steps in the bayesian workflow then it's probably best to use overlapping tags and continue to keep all py scripts in a single directory (even if the generated page has sections). For example, evaluating the prior/posterior predictive distribution are both model checking and criticism tasks.
   • I suggested the distinctions above because my expectation for the example gallery is to be a visual reference first and foremost, so I tried to start with visual groupings then use more conceptual ones when this is not possible. I love the idea of having a bayesian workflow related page, but I'd make it a different page whose focus in on the workflow, e.g. the page shows the workflow image and when you hover on a node of the diagram a modal floating window opens with advise and examples of using ArviZ for that task.
   • We don't have any of these such examples yet, but the style showcase one doesn't use any "arviz plotting function" and can't be tied to an api page. With the current configuration, the same would happen for those, nothing would break, it would show this "No API Documentation available" message: https://python.arviz.org/en/latest/examples/styles.html. If we want to showcase a specific link, it might be better to optionally add a custom apiname attribute on the python script (like the thumbs thing we have now) so for example, the styles example page could link to https://matplotlib.org/stable/api/style_api.html#matplotlib.style.use or https://matplotlib.org/stable/gallery/style_sheets/style_sheets_reference.html. But as a general solution, it is probably better to get codeautolink working on the example code samples (currently works only on notebooks not sure why, but see how code samples have clickable links that point to the corresponding api page)

[canyon289]

From the discussion above theres a lot going on so here's my suggested split of tasks

1. Split the css out into its own css. This shouldnt be too bad and will get you more familiar with sphinx
2. For the next task of building the new page I would manually prototype the new example gallery, using a subset of examples, outside of Sphinx. That way its faster for you to iterate Sarina
3. We figure out how to include it in sphinx. Sphinx is a bit complex but at the end of the day its just open source code so we can figure out how to get sphinx to generate the page in the new style
   • https://www.sphinx-doc.org/en/master/templating.html

Also this is the first time ive ever heard that @OriolAbril is not good at something! From everything ive seen Oriol is great at everything 😱

[sarinac]

For CSS and example gallery (points 1 and 2 above), moving CSS to external file was pretty straightforward. Then I tried a combination of Sphinx's grids/cards and ended up deleting all the original custom CSS because Sphinx has its own CSS classes and elements.

The code and gallery look cleaner (no more thumbnails, got rid of hover text), but the generated HTML/CSS for Sphinx grid-item-card is pretty limited and hard to modify (i.e. lack of <div>s to space things correctly, can't insert own classes while creating elements, using flex instead of grid, etc.). Even if I tweak the page with the chrome developer, I don't actually know if I can implement the HTML I want with Sphinx. I'm overwriting a lot of their classes in the custom CSS file to make sure they're sized and spaced right. In the current gallery_generator, it's really easy to customize both the HTML/CSS although it's not fun to look at in a .py file (would still prefer sphinx).

Another thing I could try is sphinx-gallery which looks more customizable, but I'm not sure yet how much change that involves. I do like how their template is structured (better example here). I would probably figure this template out first before splitting into categories based on Oriol's recommendation (also like the idea of a separate workflow page).

...

[OriolAbril]

but the generated HTML/CSS for Sphinx grid-item-card is pretty limited and hard to modify (i.e. lack of

s to space things correctly, can't insert own classes while creating elements, using flex instead of grid, etc.).

It is not possible to change the name or the number/hierarchy of the generated html elements (cards and grids are generated all from div elements) but is is possible to add extra classes in addition to the ones the extension provides by defining them as arguments of the grid/grid-item/card directives.

For example, the page: https://www.arviz.org/en/latest/our_team.html (whose core contributor grid is generated from https://raw.githubusercontent.com/arviz-devs/arviz-project/main/core_contributors.md) uses the p-2 class so unlike the default, their edges are a bit rounded. If you want you can try opening a PR with the changes you have done so far and I can try reviewing to see if I can help. Or try to have a quick call to discuss this

---

Externalizing the gallery so everything depends on sphinx-gallery instead of a custom extension would be great, but I don't think it is possible to combine multiple python scripts into the same example using tabs.

I am also not sure what you mean by the sphinx gallery template. Do you mean so code and explanations can be combined in the same page? If so we can look at multiple alternatives, I don't really like the sphinx-gallery approach, I find it's too standalone (either you use everything from sphinx-gallery or you use nothing) and not well integrated with the rest of sphinx or jupyter ecosystem. It looks like now it is possible to use the sphinx-gallery css alone though, which is something.

[sarinac]

Hey @OriolAbril and @canyon289 - I'm going to start suggesting layouts for categories.

In the mockup below, I'm suggesting the following:

• Table of Contents on the left panel is organized by categories (starting point based on Oriol's previous comment). Within each category, the charts are arranged alphabetically.
  • Oriol mentioned that there can be overlap between categories, so the mockup shows Forest Plot appearing in "Mixed" and "Distributions" as an example. They should link to the same page, but Forest Plot will just show up twice in the Table of Contents + gallery card. Is that okay?
  • Implementation-wise, I'm thinking of reusing this regex code from the gallery generator to get the categories from each MPL plot file.
  • Last category in table of contents is "Miscellaneous" which dumps any chart that doesn't have a category assigned 🗑️ (just kidding, I don't know a better way to handle this).
• Content Cards will be organized based on category to match the table of contents. I have some style suggestions:
  • Wrap each chart with a light gray border (see "Mixed" section, the "Distributions" section does not have this just for comparison). I think this will make the layout look more structured/symmetrical because not all charts take up the same amount of real estate. I put the chart titles below (also experimenting with all uppercase which I tend to like for short titles, but I don't feel strongly for this).
  • On hover, plot title turns blue + chart gets a blue overlay with the function name in white text.
  • I changed the gray background to white for the charts so it looks "cleaner"/minimal. If you're okay with me adding a custom style to ArviZ then I can play around with chart aesthetics (e.g. change blue to arviz blue) 😄 I'm not sure if users would prefer white over gray background, but I personally like charts that blend in with the background.
• Other thoughts:
  • We can put descriptions under each category that can say which plot(s) is most commonly used. I think that will be a good entry point for new users when they are exploring the gallery. The python graph gallery is a rough example of what I'm describing (header, description, image tiles, repeat).
  • We can add documentation on "best practices" in the styles section? I'm not too sure what it'll include yet, but if this is the right place for it then I can add some ideas on it to the next mockup.

[canyon289]

Oriol mentioned that there can be overlap between categories, so the mockup shows Forest Plot appearing in "Mixed" and "Distributions" as an example. They should link to the same page, but Forest Plot will just show up twice in the Table of Contents + gallery card. Is that okay?

Duplicates are fine for now. I wonder if itll make things too cluttered but lets try it out

On hover, plot title turns blue + chart gets a blue overlay with the function name in white text.

Sounds good

I changed the gray background to white for the charts so it looks "cleaner"/minimal.

Also good

If you're okay with me adding a custom style to ArviZ then I can play around with chart aesthetics (e.g. change blue to arviz blue)

I would be thrilled if proposed changes to the styles. Youre really good at this

We can put descriptions under each category that can say which plot(s) is most commonly used. I think that will be a good entry point for new users when they are exploring the gallery. The python graph gallery is a rough example of what I'm describing (header, description, image tiles, repeat).

I think this will be tougher for ArviZ because some plots require so much context on both modeling and how samplers work. Lets open another issue ticket to discuss how we may meet this intent.

We can add documentation on "best practices" in the styles section? I'm not too sure what it'll include yet, but if this is the right place for it then I can add some ideas on it to the next mockup.

Lets open another issue for this as well at least to track we want to have a discussion

Thank you for this thought and discussion. We have some concrete actions and also some new threads to think about. Appreciate all the knowledge and improvements youre adding here

[OriolAbril]

will try to add more comments soon, some quick ideas for now

Duplicates are fine for now. I wonder if itll make things too cluttered but lets try it out

My idea was to have arviz funcions appear in multiple sections but to aim for no duplicated images/examples. plot_forest will appear in multiple places, but in one it will have the rhat and ess dots, in another show only the intervals for each variable, yet another plotting mutiple models... so the images would be mixed, ditribution, comparing distributions, but the functtion is always plot_forest. We can probably also have overlap between categories using the same image but that might be confusing to visitors, we'd have to check somehow.

I changed the gray background to white for the charts so it looks "cleaner"/minimal.

is that updating the use of arviz-darkgrid style to arviz-white or arviz-whitegrid? or something else? I think the long term goal here is to have an arviz-doc style we use throughout the documentation and that is also available as the other arviz themes.

If you're okay with me adding a custom style to ArviZ then I can play around with chart aesthetics (e.g. change blue to arviz blue)

That would be ok but you will probably need to create a whole new colorscheme from 0. The current one is colorblind friendly, so if you only change one color chances are it will become similar to another of the existing ones for some colorblindness type.