Sankey Crashes

[kenny5033]

With certain queries, the sankey maps begin to crash. The browser tab takes up an inordinate amount of resources.

Example: CLPFU a2 Germany PCM Energy Final IEA Not including NEU 1970

To look into:

• [x] See if Professor Heun can create the sankey diagram in R given the same query.
• [ ] Look for a number of rows threshold that makes the rendering crash.

[kenny5033]

Possible fix:

• Assuming the cause is too much JavaScript being executed client side at once. Output diagram as an image and then send that back to the client.

Cons: Lose interactivity provided by Plotly.

[rpruim]

How much javascript is associated with each plotly plot vs with the plotly library? Does the library get reloaded with each graphic?

[kenny5033]

Each plotly plot comes with a chunk of script that actually renders the plot using the plotly library, which is only included once when the whole page loads. From my testing, the issue is in the plot being rendered client-side and the only thing changing client side, besides a few new divs for the plot to live in, is the chunk of javascript being delivered with a new plot.

[kenny5033]

Here is an example of the javascript delivered with a plot:

[kenny5033]

After today's meeting we know that the sankeys we currently have are slightly wrong (#35). I don't know or think that it will have a large impact on this problem, but it is a possibility.

[MatthewHeun]

With certain queries, the sankey maps begin to crash. The browser tab takes up an inordinate amount of resources.

Example: CLPFU a2 Germany PCM Energy Final IEA Not including NEU 1970

To look into:

• [ ] See if Professor Heun can create the sankey diagram in R given the same query.
• [ ] Look for a number of rows threshold that makes the rendering crash.

Here is the Sankey diagram that I constructed in R for the requested example:

I noticed that it takes a while (22 seconds) to download the DEU, 1971 data from the database. Making the Sankey diagram is very fast (< 1 second).

For posterity, the code I used to download the DEU, 1971 data is

foo <- PFUPipelineTools::pl_filter_collect("PSUT", 
                                           countries = "DEU", 
                                           years = 1970, 
                                           methods = "PCM", 
                                           ieamws = "IEA", 
                                           last_stages = "Final", 
                                           energy_types = "E", 
                                           includes_neu = FALSE,
                                           conn = conn, 
                                           collect = TRUE, 
                                           matrix_class = "matrix")

[rpruim]

Another possible (temporary?) solution here would be to create the sankey diagrams in R.

See, for example,

• https://christophergandrud.github.io/networkD3/
• https://rviews.rstudio.com/2022/05/25/calling-r-from-python-with-rpy2/

[kenny5033]

After some testing:

The SanKEY.js library does not crash in making this report. Server side it runs in around 0.1 seconds and it renders pretty quick client side--maybe a second or less on my machine on Firefox.

[rpruim]

So is SanKEY.js our provisional option of choice? Any missing features/issues that we already know about?

[kenny5033]

We haven't run into any issues / missing features yet...

From what I've read on their documentation, the plots as a whole are fairly customizable as to what is displayed and callbacks when something happens. That is, desired functionality should be reasonably easy to add.

Given that we've pretty much moved over to that library, I'm marking this bug as fixed