Allow solutions to be in multiple sectors

[DentonGentry]

Currently, data/overview/solutions.csv holds the sector names as published in the 2017 Project Drawdown book. The Drawdown Review in 2020 updated the sectors, now called Areas, with new names and organization. A number of the solutions are now present in multiple Areas. For example, Waste to Energy was formerly in the Electricity Generation sector and is now in both the Electricity and Industry areas.

There are several things which will need to be updated in order to adopt the new area definitions:

• A CSV in data/overview/solutions.csv may no longer be the best way to do this, it may instead be appropriate to use JSON in order to support lists and allow a solution to be in multiple areas.
• the data visualizations in the overview page of the Jupyter UI, a donut chart and treemap, will need to be updated. It isn't immediately clear how to support a single solution being present in multiple sectors in either of these visualizations, we'd either need to come up with a way to modify them or replace the dataviz with something else.

This issue is marked as good-first-issue because it does not require coming up to speed in the Drawdown models, but does require some experience working with data visualizations.

[Sunishchal]

With regard to the CSV structure changing to JSON, I believe this could be handled instead by adding a column to the table:

This would allow us to keep the existing vega interfaces, since it's straightforward to read a CSV into a pandas dataframe. A drawback to this approach would be if in the future we expect solutions to be in 3 or more sectors, we'd need to add yet another column to the table.

I think this is a reasonable tradeoff, because if at any point a solution falls into more than 2 sectors, it would likely require us to rethink the layout of the data viz. As such, perhaps it would be best to decide on viz layout first and then determine the ideal table structure.

To brainstorm new viz types which support dual sectors, here is something I made to see how many solutions fall into two sectors and if there may be any adjacencies we can leverage:

It seems the highest overlap sectors are Electricity / Buildings and Food, Agriculture, and Land Use / Land Sinks. I'm thinking we could create something like a clustered bubble chart where each cluster is a sector and each bubble is a solution. Like a venn diagram, the clusters with adjacencies will overlap with eachother to accommodate these dual sector solutions. The rest of the solutions can be in single sector clusters surrounding the overlapping ones.

Could you explain what the inner and outer circles represent in the donut chart? Is there a reason there are two separate charts (treemap & donut), or could they be consolidated?

[DentonGentry]

The donut chart outer ring is individual solutions like Refrigerants or Regenerative Agriculture. The inner ring are the Sectors from the 2017 book like Food or Electricity Generation. The donut chart was constructed as an homage to the printed collateral put together for the book, as seen here. The treemap similarly was constructed to resemble the checkerboard-like collateral. The intent was that clicking on a solution in either the treemap or donut chart would check/uncheck that solution in the list to be rendered, though handling clicks means setting up a Channel for I/O which was never implemented.

Neither donut nor treemap has to be retained, and both could be replaced by a single visualization.

A clustered bubble chart certainly sounds interesting and has potential. How would one construct such a thing? d3.js? The researcher UI currently is constructed as a Jupyter notebook, and served using Voilà (which can be seen at https://solutions.geekhold.com/). Ideally, anything we use for dataviz would work in both the traditional Jupyter Notebook and in Jupyterlab, where Jupyterlab requires an extension be constructed to allow JavaScript to be added. I know I've seen a d3.js labextension, though we currently only use it indirectly by way of ipyvolume.

[Sunishchal]

Thanks for clarifying. Yes, I believe the two visuals can be consolidated.

Here is a more tangible mockup of what I have in mind. You may interact with it here: https://public.tableau.com/views/ProjectDrawdownSolutions2020/VennDiagramBubbles?:display_count=y&publish=yes&:origin=viz_share_link

I don't believe using Tableau will be a long term solution as it will probably lack interoperability with the rest of the UI in Jupyter to allow filtering by clicking on the bubbles. Also, this is not a standard viz type Tableau handles, I had to do a lot of manual layout editing in order to get the venn diagram effect. You'll also notice the scale of bubbles between clusters are not synchronized.

Hopefully this mockup is helpful for evaluating if a "clustered bubble chart" is the direction we would like to go. At the very least, this viz is available in Tableau Public in case it's useful for other purposes.

I searched around for viz frameworks which can generate a chart like this programatically, and had little luck, so it may require some custom engineering. I suspect we could use plotly to create overlapping bubble charts, but fear the manual effort will be nontrivial. I'm also unsure how easy it would be to set up the I/O backchannel in plotly.

While we contemplate, I will try and brainstorm other viz types aside from this clustered bubble chart which can serve the purpose of showing solutions in multiple sectors.

[DentonGentry]

Tableau was considered when initially working on the researcher UI, but its pricing model is per-user which is problematic: we want people to be able to interact with the system, even add data sources or adjust assumptions in their local view, and be able to evaluate the model without having to contact Project Drawdown.

This is hoped to allow a larger reach to climate-interested researchers by keeping friction low. They only have to begin corresponding with Project Drawdown if they want to contribute something, like new data points in the VMA or adoption data, to be incorporated into the published results.

I don't see a way to use per-user licensed components in that kind of environment. Access would have to be granted explicitly in order to ensure license compliance. Requiring people to sign up to get initial access would drive a portion of potential users away.

---

As far as software which might be able to produce a bubble chart like this:

• Altair Charts is used for most current visualizations, and its scatter charts look like they might be able to do this.
• d3.js seems capable of producing something like this, and does have a Jupyterlab extension. We don't currently use it directly, but some of the components we use rely on d3.js for their rendering.

[Sunishchal]

Tableau's per user pricing is only for an enterprise deployment where data privacy is a concern. Since Project Drawdown is fully open (if I'm not mistaken), then we could leverage the free solution Tableau Public provides (up to 10GB/user and 15M records per workbook). Researchers can download and modify existing dashboards (or even edit within browser) without needing any support from Project Drawdown. </end sales pitch>

That being said, Tableau may not be an ideal solution due to the interoperability limitations with the rest of the Jupyter UI. I wonder if the entire UI could be hosted in Tableau? But that's probably a whole separate discussion.

On the point of Altair, this library looks quite feature rich. I think the multifeature scatterplot may be a candidate building block for the clustered bubble chart. I'm still concerned about how we will get the bubbles to all fit within the larger sector clusters and overlapping portions. There will also be challenges synchronizing the scale of CO2 reduction units across all the bubbles. https://altair-viz.github.io/gallery/multifeature_scatter_plot.html

I haven't worked with d3 and don't know javascript, so I may not be too useful if we go down that path. I've seen how much customization that framework offers though, so I wouldn't be surprised if that's the only way to build this specific view.

Perhaps if we have time in tomorrow's meeting, we can get Chad's thoughts on the best viz type from an end user perspective?