[Sim] Output sim_engine plots -> streamlit/plotly -> webapp

[trentmc]

Background / motivation

We're on a path to output predictoor analytics plots to webapp, using streamlit + plotly.

Currently the sim engine plots are output on-the-fly directly to the desktop using matplotlib defaults. They're not going to the browser. This was a fine enough starting point.

But there are issues:

• Because it keeps updating the plot in realtime, and takes up so much screen real estate, it's always popping over everything else. Annoying when you want to do other things at the same time
• A user reported ^ as well, in #744
• It doesn't work for Calina, because she needs to run compute remotely. What could work for is that the py code runs remotely, and then she can see the results in her local browser.
• Hard to add more plots, because of limited real estate. Yet we have more plots we want to add.
• Haven't tried interactivity yet, but it'll be hard

The solution: use streamlit.

That will address the issues above, and, a bonus: it helps us prototype for the analytics dapp. (We could even have them all in the same dapp. Pros and cons to this, decision is still TBD.)

Pre-requisites

• [x] "Refactor plots in sim_engine" #750

TODO

• [x] Migrate sim flow plots to a streamlit architecture, like the one we're going for with analytics app
• [x] Complete pickle/separation of concerns/overall architecture

Resources

Key ref: https://plotly.com/python/creating-and-updating-figures/

Maybe useful: https://github.com/Paulescu/real-time-data-pipelines-in-python/tree/main

[idiom-bytes]

How to organize all of this towards updating plots in a backend process + serving this via FE webapp

process 1 - etl.py - fetch + etl + update plots
process 2 - sim.py - running sim + updating plots
process 3 - streamlit frontend w/ different pages serving data + artifacts
> page a - predictoor net income
> page b - sim

[trentmc]

process 1 - etl.py - fetch + etl + update plots

This can run as part of sim_engine.py, as it does now. No need to make things more complicated. KISS.

Also: page a & b should be switched. It's the order that the user does things.

And: it shouldn't be "predictoor net income", it should be "predictoor", where net income is the top level metric.

Therefore, how I see it:

process 1 - sim.py - running sim + updating plots
process 2 - streamlit frontend w/ different pages serving data + artifacts
> page 1 - sim
> page 2 - predictoor

[trentmc]

@calina-c fyi I have completed the pre-requisite (refactor sim plot code)

[idiom-bytes]

There is a fork that we have to decide on:

• Use matplot lib interface
• Use streamlit native plotting (interactive, etc...)

Options:

• streamlit charts
• matplot lib
• plotly

[trentmc]

We want interactive.

It's ok to move away from matplotlib interface.

[calina-c]

Current behaviour:

• Static full-size image generated with matplotlib, containing multiple subplots

• Difficult to separate and analyse the graphs, everything is displayed in one single picture and the user must zoom in and out

• No possibility for real-time updates, i.e. see the changes instantaneously during the simulation

• No interactivity with the data itself, the scale, etc.

Possible streamlit approaches

1. Streamlit using the matplotlib generated subplots

Advantages:

• We can have this fully merged by tomorrow
• We can later transition the pieces (currently matplotlib generated) into native interface
• Can be used in real-time with refreshing, but can not be made interactive

Downsides:

• Requires breaking up the current matplotlib figure into multiple figures (one per plot) because streamlit can’t display it otherwise

2. Streamlit using its native interface

Advantages:

• Would be real-time AND true interactive

Disadvantages:

• We have to redo all the graphs using streamlit. Easy for line charts, but might be a challenge with the scatter and contour plots. Current matplotlib code will be lost, used only as inspiration
• Takes much more time and code. Such tasks are also slowed by constant conflict resolution -> that’s why I propose starting with the first approach, then transitioning each graph

Samples:

• here is the current version on my branch, a hybrid between the existing matplotlib code and using streamlit to display the graphs in real-time:

https://github.com/oceanprotocol/pdr-backend/assets/4101015/96579e90-0ebd-4405-9ea9-be58bb822bfd

• here is a simple line chart using streamlit's native plotter for the first graph (profit vs time). You can see there's better zooming/scaling options, exports etc.

https://github.com/oceanprotocol/pdr-backend/assets/4101015/f31fb54a-288a-4a85-9421-81648acc5b4f

My recommendation

• keep matplotlib figures for now and rework them piecemeal later, converting each plot to a native streamlit plot, as we go. It will prevent conflict resolution issues and it's still an improvement to real-time display and the canvas itself.

Oh, just one more thing...

(to quote Peter Falk's well-known character)

To run with streamlit, we need to modify commands, i.e. we can not run just pdr. For sim, it would be streamlit run pdr -- sim my_ppss.yaml instead of pdr sim my_ppss.yaml. Is that an issue? I couldn't find another way to start the streamlit server.

[trentmc]

keep matplotlib figures for now and rework them piecemeal later, converting each plot to a native streamlit plot, as we go. It will prevent conflict resolution issues and it's still an improvement to real-time display and the canvas itself.

Sounds good to me

[trentmc]

To run with streamlit, we need to modify commands, i.e. we can not run just pdr. For sim, it would be streamlit run pdr -- sim my_ppss.yaml instead of pdr sim my_ppss.yaml. Is that an issue? I couldn't find another way to start the streamlit server.

That hurts ux for perhaps the most commonly-called command.

Let's think about ways where can we can preserve pdr run __. Eg maybe right after installation, the user defines a bash-level alias mapping "pdr" -> "streamlit run pdr"?

[calina-c]

To run with streamlit, we need to modify commands, i.e. we can not run just pdr. For sim, it would be streamlit run pdr -- sim my_ppss.yaml instead of pdr sim my_ppss.yaml. Is that an issue? I couldn't find another way to start the streamlit server.

That hurts ux for perhaps the most commonly-called command.

I was thinking about that, yes. That's why I thought to ask.

But we can't alias every pdr run, because not all commands need streamlit. Plus, streamlit opens a port and runs the server. Users should take note of that and how to close/kill those processes when they finish using them.

I will check if I can mimic the functionality of streamlit run inside pdr-backend, if possible. I will also look into some sort of aliasing, but if possible just for sim.

[idiom-bytes]

I have moved the discussions around altair and plotly from here, to this other ticket. https://github.com/oceanprotocol/pdr-backend/issues/842

This will enable us to close this ticket and move forward with the altair/plotly discussions separately.