search

BSC-ES / autosubmit-gui

The Autosubmit Graphical User Interface (GUI) is the web-based Autosubmit frontend, allowing users to discover, monitor, and analyze experiments. It is based on ReactJS and relies on the Autosubmit API as the middleware to get experiment information.
MIT License
2 stars 0 forks source link

Graph Visualization Improvement #41

Open kinow opened 2 years ago

kinow commented 2 years ago

In GitLab by @mgimenez on Oct 6, 2022, 09:48

A while back @mcastril asked me to look into the graph visualization in the GUI. I Learned some techniques in one of my master's subjects which could be useful.

But before I get into coding I am arranging some meetings with users to get a grasp of how they use all visualization tools in Autosubmit. So far I have talked only with Aude and Fransesca. My idea is that in the following weeks I reiterate our interest in arranging this short meetings with users.

In this issue I will report on the main takeaways of each meeting.

kinow commented 2 years ago

In GitLab by @mgimenez on Oct 6, 2022, 09:52

Last week @cgutierr and I met with Francesca. She showed us this experiment a4dd that had a splitted job with dependencies. This caused that two sequential levels where fully conected, creating this thick bush apearence with overlapping edges.

We could explore increasing the vertical spacing between levels of the graph, which seems to be fixed.

image

kinow commented 2 years ago

In GitLab by @mgimenez on Oct 6, 2022, 09:59

Two weeks ago I met with Aude, which only worked with workflows with 5k+ jobs. Although the graph was indeed rendered, Aude found it useless since exploring the graph meant panning a lot left and right to find nodes.

This issue happend because, since we have DAGs, the renderer forces that each level is at the same vertical coordinate making very long horizontal lines of nodes (figure 1). This strategy has the flip side of logically organizing the workflow in each level (e.g. chunk 1 -> chunk 2 -> chunk3). But when we have such massive workflows it makes the graph a long long line (figure 2)

figure1

figure2

kinow commented 2 years ago

In GitLab by @mgimenez on Oct 6, 2022, 16:01

Had a meeting with Eric. He is kind of an outlier for our case, since he tipically works with "small" (150 jobs) workflows. So he usually uses graph view and he is pleased with it. We know that our library can handle appropriately this "small" graphs.

Even mesier graphs, that is with more complicated dependencies, when constraint to the hundreds of nodes are still understandable.

He wasn't aware of the data-member and status buttons.

kinow commented 2 years ago

In GitLab by @cgutierr on Oct 10, 2022, 07:57

From this meeting with Eric, he had requested "square" selection (see image). I think this could be added in the next GUI iteration.

image

kinow commented 2 years ago

In GitLab by @mcastril on Oct 11, 2022, 09:44

Thanks guys for this discussion and the reflections. Here you have another beautiful example. The positioning is fine, I think, but improbable:

https://earth.bsc.es/autosubmitapp/experiment/a510

One problem is the CLEAN_EXPERIMENT at the end, creating a lot of edges.

Then it's odd that the library is putting some members on an inferior level to balance the graph better. That makes better use of the space, but it can be confusing if you don't look at the edges, and there are a lot.

In any case, some grouping functionality I think would facilitate having the general picture.

kinow commented 2 years ago

In GitLab by @cgutierr on Oct 20, 2022, 09:27

The mouse multi-selection is ready. Working in local, this afternoon will deploy in into Rocky Dev env.

Maybe this could also go to the release @mcastril ?

Made this quick demo for small graph and demo for big graphs video.

kinow commented 2 years ago

In GitLab by @mcastril on Oct 20, 2022, 09:45

Yes I thihk so, given that we may have been testing the legacy API (however, I don't understand how some of the last developments could have been tested if that's the case) and not the production one, we should re-do the tests.

kinow commented 2 years ago

In GitLab by @mcastril on Oct 20, 2022, 09:46

Thanks for the demo @cgutierr, it's impressive. I think that it will help a lot.

kinow commented 7 months ago

In GitLab by @ltenorio on Mar 27, 2024, 13:37

Recently, I have been testing some other visualization libraries to replace the one used in the GUI (vis-network).

At first, React Flow seemed to be an amazing library for this as it allows rendering Sub-Flows. Unfortunately, it only renders the workflows as SVG which is really expensive and will not work with our large-size graphs (>1k nodes).

Another amazing option is Cytoscape.js. It is actively maintained by many institutions, it is extensible and has friendly documentation. It uses the canvas API like vis-network which is great for rendering large graphs. I've tested it with one large graph and it performs better, but not significantly. Even so, it provides way more options than vis-network allowing also to declare Compound graphs which is awesome for visualizing wrappers. Already made a POC and it looks promising:

Screencast_from_27-03-24_13_12_28

Also, @bdepaula suggested looking at Ogma. This one uses a more powerful web API than Canvas, which is WebGL. It seems to be the final solution but it is a commercial product :cry:

kinow commented 7 months ago

In GitLab by @bdepaula on Apr 2, 2024, 09:52

Great summary!

We tried ^1 to use Cytoscape in Cylc but if I remember well it was hard to re-use the existing Vue components with Cytoscape and the performance with large graphs wasn't very good.

Their graph view was finalized after I left the project using SVG + canvas to re-use existing Vue components and to manage reactivity/performance ^3. But we don't need to follow that path if there are other options that perform well.

Also, @bdepaula suggested looking at Ogma. This one uses a more powerful web API than Canvas, which is WebGL. It seems to be the final solution but it is a commercial product

I think they provide a commercial support (their open source business model), but the code and libraries are all under permissive Apache license https://github.com/Linkurious/ogma-react

kinow commented 7 months ago

In GitLab by @ltenorio on Apr 2, 2024, 11:06

We tried 12 to use Cytoscape in Cylc but if I remember well it was hard to re-use the existing Vue components with Cytoscape and the performance with large graphs wasn't very good.

Their graph view was finalized after I left the project using SVG + canvas to re-use existing Vue components and to manage reactivity/performance 3. But we don't need to follow that path if there are other options that perform well.

Interesting! Cytoscape.js indeed feels more difficult than vis. Even so, it has many performance options that make significant improvements in rendering large graphs. Maybe it was slow in Cylc because it was also used to calculate the positions on the client-side?

I'm already working on integrating it into the Autosubmit GUI because of all the perks it provides:

I think they provide a commercial support (their open source business model), but the code and libraries are all under permissive Apache license https://github.com/Linkurious/ogma-react

The catch there is that the React wrapper is open-source. To download the core Ogma package you need to generate a secret key from your user panel in Linkurious.

kinow commented 7 months ago

In GitLab by @bdepaula on Apr 2, 2024, 11:10

Maybe it was slow in Cylc because it was also used to calculate the positions on the client-side?

Might be! We indeed did all the work on client side. The server side just served JSON/Protobuf via GraphQL.

The catch there is that the React wrapper is open-source. To download the core Ogma package you need to generate a secret key from your user panel in Linkurious.

Ah, hadn't seen that coming :disappointed: (literally disappointed :sweat_smile: ).