search

cylc / cylc-ui

Web app for monitoring and controlling Cylc workflows
https://cylc.github.io
GNU General Public License v3.0
37 stars 27 forks source link

static graph visualization #82

Open hjoliver opened 5 years ago

hjoliver commented 5 years ago

We probably want to replace cylc graph with web technology too, not just the control GUI graph view.

hjoliver commented 5 years ago

This graph demo (found by @MartinRyan ) allows manual node repositioning (as many do, I think): https://cytoscape.org/cytoscape.js-euler/. Could be useful when automatic layout fails us.

MartinRyan commented 5 years ago

Cytoscape looks great, it has a dagre extension, and a navigator 'bird's eye view and pan control' extension (mini-map). It consumes JSON though there are convertors for graph objects.

hjoliver commented 5 years ago

Also from @MartinRyan py2cytoscape might be useful for static vis. (Cytoscape itself might be useful for the live control GUI as well)

oliver-sanders commented 5 years ago

This is an interesting challenge because:

The UI-Server is only concerned with the installed copy and is not necessarily able to see the working copy.

It is OK to require users to register a suite for cylc graph functionality (as the suite wouldn't appear in the web UI otherwise) but we wouldn't want users to have to repeatedly re-install a suite in order to visualise it.

hjoliver commented 5 years ago

Well, yes and no.

If you're working on the working copy then you are able to install it repeatedly pretty easily, and if the payoff is kick-ass web base viz that seems a small price to pay, And besides you currently have to do that anyway if your suite.rc takes Jinja2 inputs from rose-suite.conf (it won't validate or graph without being installed first).

Also what's the alternative? We don't want to keep a GTK-based static graph GUI. An electron app might allow us to kill both birds with one stone, I guess.

matthewrmshin commented 5 years ago

Assuming that we have done enough logical migration from rose suite-run to Cylc in the coming months, then we should have enough logic in Cylc to deal with the rose-suite.conf stuffs. The web UI can have an local file system browser so users should be able to open a suite from its source location. Otherwise, we can also build support for the UI to search for named suites from various configurable locations on the file system. (E.g. If a suite cannot be found under ~/cylc-run/SUITE/ then look at ~/roses/SUITE/.)

oliver-sanders commented 5 years ago

If you're working on the working copy then you are able to install it repeatedly pretty easily

But what if the suite is running?

Also this is the situation we have at the moment, I advise people to do this:

$ rose suite-run -l; cylc graph mysuite

Unfortunately this causes confusion and is a barrier to usage. As a result very few people here use essential Cylc utilities such as:

I wrote this page to try and break down the barrier but sadly it has made little difference.

I still regularly see users trying is running the entire suite to test changes.

Also what's the alternative?

No nice solution:

hjoliver commented 5 years ago

Fair point on barriers to easy use. So let's do better than that if we can. We do need a non-PyGTK static vis tool with cylc-8 though, one way or another.

I still regularly see users trying is running the entire suite to test changes.

Same, and even worse, running an entire beast of a suite repeatedly to test how some cylc feature works, when a 5-line 1-task test suite would do. You can lead a horse to water, but you can't make it change its spots. Or something.

kinow commented 5 years ago

See comments from (now closed) #81 about graphs in Cylc 8:

@hjoliver

A Vue component to draw directed acyclic graphs in the browser leveraging dagre-d3v4 (https://github.com/smbolton/dagre-d3v4) - https://github.com/vdidonato/vue-dagre-d3v4 - seems to be broken.

https://github.com/cylc/cylc-ui/issues/81#issuecomment-423513023

@oliver-sanders

I'm currently looking at Javascript libraries for graphing, I'll update this issue with conclusions as I investigate. I'll probably put up my experiments as a GitHub repo when more complete.

Graphing consists of two components:

The Layout Engine

The bit which arranges nodes / edges spatially. See the layout engines supported by cytoscape for examples.

  • Directed hierarchical graphs are hard
    • Layout is slow, there isn't a whole lot of choice when it comes to layout engines. Dagre is the standout but is a fairly stagnant project.
  • More arbitrary layouts are easy
    • We might want to automatically swap to a less ideal layout engine for larger graphs.

Requirements

1) Directed & hierarchical 2) Must handle edges spanning multiple levels 3) Width of node considered in graph (i.e. nodes cannot overlap) 4) Speed!

Results

Engine 1 2 3 4 Supported Engines
Dagre :ok: :ok: :ok: 4.1s cytoscape, d3
Klay :ok: :partly_sunny: :no_entry_sign: 6.4s cytoscape, d3
D3-DAG :ok: :ok: :no_entry_sign: [2] >> 60s [3] None? [4]

[1] Benchmark using the first cycle of MO global opp suite [2] Possibly using the separation function though I can't get this to work [3] Tested "Vert", "Greedy" and "Spread" layout methods [4] Visualisation constructed by user provided functions, i.e. user provides the engine

The Graph Engine

The bit which renders the graph, provides an interface for manipulating and interacting with it. This is the component we will end up working with / extending to form the graph view.

Requirements

1) Mobile friendliness 2) Hackability of the graph:

  • Job status indicators (require more than just text labels)
  • Sub-workflows?
  • Advanced collapsing (e.g. parametrisations using an ellipsis symbol)? 3) Not a massive RAM hog 4) Support for the desired layout engine(s) 5) Node / family collapsing

Results

Engine 1 2 3 4 5
cytoscape :ok: Dagre, Klay
d3 Dagre, Klay
vis.js
arbor

Hybrids

Systems where the graph engine and layout engine are integrated:

Results

Engine L1 L2 L3 L4 G1 G2 G3 G4 G5
viz.js :ok: :ok: :ok: :ok:

https://github.com/cylc/cylc-ui/issues/81#issuecomment-423555699

@hjoliver

Interactive arbitrary grouping: select a bunch of nodes (individually, or by drawing a rectangle around them) and group them into a single node. Layout algorithms for visualizing directed acyclic graphs - https://github.com/erikbrinkman/d3-dag - new-ish? demo https://jsfiddle.net/ye2xanf9/12/

https://github.com/cylc/cylc-ui/issues/81#issuecomment-423662282 https://github.com/cylc/cylc-ui/issues/81#issuecomment-423686731