Graph View

[kinow]

The GraphQL data model is stable now, so we can create the initial views for the Vue.js app. The first one will be the Graph View, listing all the tasks found in the task pool.

It is important to list all the tasks, without filtering at this point.

It is also important to simulate the change in the tasks, as we need to confirm that the view can be used with a dynamic graph.

While other web interfaces may be used as reference, there are things in the Cylc 7 GUI that can be used for the initial implementation, like the task colors.

• [ ] write mocked data
• [ ] add a JS library to plot a graph
• [ ] respond to changes in the mocked data, updating the graph

[kinow]

write mocked data

Python has a project called vcrpy: https://github.com/kevin1024/vcrpy, which I believe was based on the Ruby vcr: https://github.com/vcr/vcr.

And like the VCR, you can use it to record HTTP request/responses. It is used to record interactions client-server, and replay them later (like a VCR). Which is useful when writing integration tests for web applications.

Looks like there are JS ports too:

• https://www.npmjs.com/package/fetch-vcr
• https://github.com/linkedin/sepia

So we should be able to start a suite with some tasks and interaction (e.g. the old complex found in the tutorial or examples of Cylc 7 I think). That suite also takes a while to complete.

And then change the existing SuiteService to return the current workflow state. And write a setTimeout call, to advance the state every 5 or 10 seconds.

EDIT: moved to #77

[oliver-sanders]

he first one will be the Graph View

I would strongly advise implementing the Tree View first. It is the simplest view and for most users [here at least] it is "the Cylc GUI".

[kinow]

Cylc 7 Graph view:

To the right of the screen shot, is the Dot view.

[oliver-sanders]

tldr; From a quick look I think dagre.js with a custom presentation layer is probably the way to go but some further research would be a good idea. Performance is likely to be poorer than the Cylc7 graphviz approach.

The graph view is a combination of:

• A layout engine
  • This is the hard part, proper nasty algorithms, we should use an existing library.
  • The prime candidate is dagre.js
• A presentation layer
  • I've called this the "graph engine" elsewhere as some of them are large "graph theory" packages with algorithms for graph traversal etc (which we don't actually need).
  • Due to the advanced graphical things we may want to do with the graph this component may need to be custom.

I summarised the requirements of the layout and presentation layers and performed some analysis back in September.

[kinow]

That's an excellent and extremely helpful feedback!

I think dagre.js with a custom presentation layer is probably the way

Will do some further research, but having one option to start with is really helpful.

Performance is likely to be poorer than the Cylc7 graphviz approach.

Agreed. Though we may have more resources than with a desktop client, like caching in multiple on server and/or client.

I summarised the requirements of the layout and presentation layers and performed some analysis back in September

A ha, I remembered seeing that somewhere, but couldn't remembered where. I am hoping to have something to show in June and collect more requirements and constraints in person.

But if you have more pointers or links to share, please keep them coming :-D

[oliver-sanders]

But if you have more pointers or links to share, please keep them coming :-D

I think that's everything important, the only other bits I can think of which are/will-be in sketchups anyway:

• The ability to focus on a task in the graph and view only the nodes N edges from it (to allow the graph view to work with really big suites).
• A mini-map for the graph view!
• Possible visual alternatives to family un-grouping
• Task/Job visual separation

[hjoliver]

Performance is likely to be poorer than the Cylc7 graphviz approach.

I'm hoping we can get around this problem by no longer displaying the full graph (at least not by default, for suites that are sufficiently big). For really big suites the full graph is not just slow, it's also useless as a monitoring and control interface. (Static visualization - like cylc graph - is another matter, because performance is less relevant there).

[hjoliver]

(I like the mini-map idea).

[kinow]

Re-assigning to @MartinRyan :+1:

Martin, not sure if you will be doing a small test project first, or changing directly here in cylc/cylc-ui. If the latter, this may be helpful I think:

• I work normally with @dwsutherland 's branches for GraphQL (https://github.com/cylc/cylc-uiserver/pull/34 & https://github.com/cylc/cylc-flow/pull/3122)
  • I have cylc-uiserver from the pr/34 above, with a virtualenv...
  • In the cylc-uiserver folder I create a virtualenv, change jupyterhub_config.py settings, fix some imports and any other issues, then remove cylc-flow dependency from setup.py, and install via pip install -e ., then navigate to cylc-flow, and do a pip install . too.
  • I start jupyterhub in this folder
• Then in another terminal I have npm run build:watch running from cylc-ui
• I will prepare a pull request later to rename the current Suite.vue to TreeView.vue, but you can use that one and create something like GraphView.vue I reckon.
• I have created services a-la angular, and the data for the TreeView.vue and other views can be retrieved using the SuiteService
• @dwsutherland is the best person to help with the query for this view, but I think you will have to query { workflow { edges {} } } and create a list of nodes and edges, and use it in some graph library. The query is also in the SuiteService. Just create a new method, and check the current tests with mocks to cover that code
• The actions to alter a Suite are being proxied via the Service. The Service accesses the Vuex Store (global state manager for Vue)... but the views access the Store directly to retrieve data (we could go through the service for that too, but it was more an organic design, happy to get some review and feedback about it later)

If you have suggestions for improvements, feel free to file new issues. I created the current architecture/design without thinking too much, as I wanted to test other things (does it work with graphql? cors? what we needed for tornado. how much JS or TS would be helpful. etc).

Hope that helps Bruno

[hjoliver]

On graph edges etc.: https://cylc.discourse.group/t/meeting-notes-ui-workflow-views-and-graphql/44

[oliver-sanders]

Here is a quick dump of the work I had done on the graph view prior to Melbourne:

• Requirements:
  • Layout and graph engine(s).
  • We expect performance to suffer a bit but it should scale simiarly to GraphViz, we should be able to render the complex suite but would expect it to take a few seconds.
  • We need the rendering to be really hackable so we can write our own custom nodes / edges.
  • Some example layouts we would expect to graph nicely:
  • task matrix: <a, b> => <a + 1, b>; <a, b> => <a, b + 1> (should have straight lines)
  • multi-level deps: a => b => c => d; a => e (ideally a-d in a straight line)
    • ordered-tasks: a_01 & a_02 ...& a_NN (ideally tasks should be in alpha-numeric order)
• Tests / Profiling:
  • GH Repo
  • Live Demo + Profiling (springy example only works locally, huh)
  • Profiling Results.
• Initial conclusions:
  • Fairly hard to match GraphViz
  • Should take a look at JS GraphViz implementations
  • Dagre is probably the best layout engine for purpose
  • We may want to just write our own presentation layer for maximum control

[hjoliver]

Efficient methods and readily customizable libraries for managing complexity of large networks - uses Cytoscape.js

(Same but better?): https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0197238

Outputs:

• https://github.com/iVis-at-Bilkent/cytoscape.js-expand-collapse
• https://github.com/iVis-at-Bilkent/cytoscape.js-view-utilities

[MartinRyan]

https://github.com/cylc/cylc-ui/issues/187#issuecomment-547179333 expand and collapse

[hjoliver]

There is a Vue app for visualising CWL workflows:

• https://www.npmjs.com/package/vue-cwl

And an all SVG library here:

• https://www.npmjs.com/package/cwl-svg

[oliver-sanders]

(note this is a static SVG visualisation, interesting to learn from)

[hjoliver]

(Chat room convo on dagre edge routing):

the layout engine (dagre) will have to generate enough control points - and in the right places - to enable the renderer to make nice curves out of them. So the jury is still out IMO.

I would think edge routing layout (control point positions) might be as hard or harder than node layout.

A-ha: https://github.com/dagrejs/dagre/issues/239

Edge routing seems to be a bit extreme:...

That is an artifact of the Brandes-Kopf layout algorithm. I've gone over it a few times and believe it is working as expected. It would be interesting to someday to use the network simplex solution from graphviz, which would not have this problem, but would be slower.

Not a blocker obviously, but a bit disappointing. Most of the Cylc 8 UI is going to look like beautiful eye candy compared to the old GUI - but it seems we can't get graphs that look as good! (hierarchical ones, at least).

Context: dagre layouts seem to look a bit cheap compared to graphviz, because of sub-par edge routing.

[hjoliver]

(From meeting discussion and chat room)

• IMO, with post-SoD low-n windows, there is less need for a hierarchical layout a la dagre (if you just see a few edges out from active tasks it doesn't matter too much where the edges point exactly)
• we will still want hierarchical for n-max windows in small to medium flows though, and static visualisation

[hjoliver]

Graphviz in the browser:

• https://github.com/mdaines/viz.js/ (archived project)
• https://github.com/magjac/d3-graphviz (active?)
• https://github.com/hpcc-systems/hpcc-js-wasm (web assembly!)

[hjoliver]

Notes from @oliver-sanders in Element chat:

A bit of investigation work:

• I think the plan is now try to try an SVG implementation probably using Dagre as the layout engine. I wrote a quick demo a while back which went some way to suggesting that the performance of SVG in-browser should be sufficient, however, it would be good to confirm that. Our graph will potentially involve a lot of SVG elements (shapes, task icons, text, arrows), what's the scaling?
• I'm guessing we would have to go with the SVG 1.1 standard?
• SVG supports the concept of element re-use which could potentially be a big win for performance so it would be worth looking into whether/how that might work for the graph nodes.
• Can we avoid re-implementing the task/job icons (would be good to share the same component with HTML/SVG).
• Styling, I guess we would do this via CSS by modifying the class of SVG objects?
• How best to implement arrows in SVG (SVG line endings are pretty useless 😠).
• Finally the graph layout engine, Cytoscape provided us with not only the GUI but also the graph layout engine abstraction layer, which was really rather useful. It would be well worth investigating whether we can steal this or if there are other libraries which support it as this would save us having to individually implement different layout engines.

[hjoliver]

Graph library for interactive CWL editor: https://github.com/rabix/cwl-svg/

[oliver-sanders]

Review 2022:

Not a whole lot has changed, the front-runners remain:

• dagrejs/graphlib
  • Grass roots Graphviz-dot-like javascript layout engine.
  • Last release 2017.
  • Dagre is still marked as deprecated with no new releases.
  • Graphlib is showing signs that it is starting to be maintained again.
• hpcc-js
  • Graphviz compiled to web assembly by a group not affiliated with Graphviz.
  • Last release 2022.
  • A fairly complete port, nice, but overkill. We only need a small subset of functionality.
  • Release cadence is confidence giving.
  • Now with a JSON interface (don't need to render to SVG directly)!

[ColemanTom]

I think https://cylc.github.io/cylc-doc/latest/html/7-to-8/caveats.html#browser-based-ui needs updating as this Issue is closed?