New diagram proposal for Overviews page

[nicolaskruchten]

Hi folks,

On and off for the past few years, I've been working on a slightly different, more up to date, more structured, diagram to the one that's currently at https://pyviz.org/overviews/index.html ... I had a chance to present my work-in-progress at a meetup this week. It's not really quite ready for prime-time, mostly because I need to figure out a reasonable default view and interaction style, but I wanted to solicit some feedback before putting a lot more work into it :)

The repo for the diagram itself is at https://github.com/nicolaskruchten/pyviz_diagram and the diagram is at https://nicolas.kruchten.com/pyviz_diagram/ (note: the default view shows all links, which is clearly not the intention! the idea is to make it less overwhelming than this!)

I'd really love some feedback on this structure, so please comment here or add issues in the repo :)

[jbednar]

Nic, thanks for getting this going! The original diagram is definitely in need of updating, and this does seem like steps in the right direction. I've come back to this several times trying to see a way to make this less complex, with fewer overlapping and ambiguous lines, but the underlying structure is even more complex than shown here, so it's tough!

I think one simplification that could help from the start would be to collapse all native-GUI toolkits into a single box (tk/MacOS/GTK/Wx/Qt), the same way you collapse vector formats, since they have the same destination and mostly the same source. But the rest of my suggestions all make things more complex! :-) (Well, other than updating the page title from pyvis_diagram to pyviz_diagram.)

One thing I definitely didn't get until watching the video was precisely what the arrows meant, so I think there should be text on the page or an "About" button that explains that the arrows represent directed communication, generally of rendered plots going to the right and plot-space events going back to the left. Even with that interpretation, without watching the video I think it will be tough to see even that e.g. Mayavi fundamentally sends data to a windowing toolkit, while in the diagram because of how the lines overlap it looks strongly like Mayavi is only a target, not a source. I hate to say there should be more lines, but maybe the backward lines could be separate from the forward lines and made thinner and/or dotted and/or a dim color and/or some other indication that the backward ones have a different status than the forward ones? The backward paths generally represent events rather than data or renderings, and showing that explicitly might help people see the fundamental left-to-right flow better. Plus then maybe there could be a legend and/or the backward paths could be turned on and off explicitly to make it clear how those differ, letting people answer the question "which pathways allow bidirectional communication" without obscuring the other relationships shown here.

Alas, most of my other suggestions end up adding even more arrows:

• Datashader can directly generate raster and HTML output, so I think it's missing forward arrows to those targets. It also provides fully interactive Matplotlib artists, so there's a missing backward arrow from Matplotlib to Datashader
• Altair generates HTML, via Vega/Vega-Lite, so it presumably needs a forward arrow
• I can't quite tell where the backwards arrow leading into Plotly comes from, but presumably it should be from Panel and Dash? If so, seems like there should be a similar backwards arrow into Bokeh, as Panel (unlike streamlit) can send events back to Bokeh plots the same as it does to Plotly plots
• Bokeh and Panel objects can be used as ipywidgets using jupyter_bokeh and thus served with Voila, so there's a missing "forward" arrow from Panel to ipywidgets and from Bokeh to ipywidgets, and also backwards arrows along the same path since ipywidgets have bidirectional comms.
• HoloViews plots in Plotly can send events back that trigger re-rendering of Datashader plots, so I think there should be an arrow going back from Plotly to Datashader or at least from Plotly to HoloViews

There are also some other dimensions or axes here that might make sense to separate into other plots or make clear in some other way:

• Should the web server component of these libraries be considered separately? E.g. Bokeh and Panel use Bokeh Server, a separate component, as their bidirectional communications platform when used outside of Jupyter. Does it help anything to pull that out so that Bokeh as a plotting library shows up in a different bubble than Bokeh as a server, the same way Voila as a server is separated from ipywidgets as a widget library and ipywidgets-based plotting libraries like bqplot? Not sure how that sort of separation applies to Dash or Streamlit.
• Should Datashader be treated separately (overlaid in some way or as a separate plot)? It really tangles everything up if all the correct arrows are drawn to and from it, because it can be used in different ways by many different libraries.
• Should pandas .plot() be moved alongside seaborn/plotnine/cartopy, omitting the other arrows from it? Seems like trying to convey that pandas (or xarray) .plot calls Matplotlib but can be configured to call HoloViews (via hvPlot not shown), Bokeh (via pandas-bokeh not shown), Plotly (via cufflinks not shown), or Vega-Lite (not shown, via pdvega not shown) greatly complicates the story, and would need a separate figure to really convey what it means.