Tree View

[kinow]

Implement something similar to what we had in Cylc 7:

EDIT: more recent screenshots, taken with 7.8.1.

And a few features in the current Tree View:

• Users can adjust font size
• Users can choose themes (color blind, high contrast, etc)
• Users can collapse / expand all (though it changes once the family/tasks are updated)
• Users can filter by status
• Users can filter by a regex (not sure what is searched, task name?)
• Users are displayed a legend of what each task color means in the tree view
• Users have access to a context menu with actions such as
  • Browse task URL (if URL is defined, otherwise a warning is displayed)
  • Trigger (if possible)
  • View logs, prereqs, outputs, call cylc show
  • Reset
  • Force spawn
  • Hold
  • Release
  • Remove
  • Enable/disable family grouping

Family grouping doesn't seem very simple. Unless we alter the underlying data, but still not sure if the Vue.js component will behave OK with that.

[sadielbartholomew]

For the basic tree-view code, there are some really nice pre-existing 'Tree' components in the Awesome Vue listing under the 'Components & Libraries -> UI Components -> Tree' heading. Too many to pick from! But as some good (while not necessarily the best, as I don't have time to look at them all) examples:

Tree component: name	homepage	demo
Sl-vue-tree	here	here
Bosket	here	here
vue-draggable-nested-tree	here	here

Features supported by some or most/all of these tree components which could be really useful to have ready out-of-the-box are:

• Drag and drop for ordering items (so that e.g. users could move families & tasks they care most about to the top of the relevant nested tree level)
• Asynchronous loading and caching of children (could help with efficiency!)
• Search, sort and filter (as we have in our old GUIs, but sounds hard to implement from scratch)

If we make use of one of these components (or similar), it seems to me the main challenge would be working out how to query the GraphQL schema to create the necessary data structure populated with the relevant workflow info content from the native workflow data structure.

[hjoliver]

Thanks for that @sadielbartholomew. I like the look of Bosket. It seems well documented too.

There's also a tree view example on the main Vue site: https://vuejs.org/v2/examples/tree-view.html

These examples seem to be all expandable lists, without the full table (i.e. without the multiple columns we need associated with each list row) ... but hopefully not hard to extend to that?

[hjoliver]

Bosket's asynchronous child loading (with timing info available too) could be great with GraphQL (I imagine) ... the UI could request - on the fly - just the tasks and families that it has been asked to display. The old UI has holds all the suite data even if not displaying it.

[kinow]

Work started, first adding mocked data. Found one issue when trying to create the query to populate the items for this view, posted here to see how it can be fixed, or if there's a workaround.

First will populate the UI as is, with an ugly table. Then will check the libraries shared by @sadielbartholomew :point_up: and hopefully get a working demo soon.

[dwsutherland]

The following are a couple of example GraphQL queries to provide the tree data: Flat Structure query

query tree($wIds: [ID], $nIds: [ID], $nStates: [String]) {
  workflows(ids: $wIds) {
    id
    name
    status
    stateTotals {
      runahead
      waiting
      held
      queued
      expired
      ready
      submitFailed
      submitRetrying
      submitted
      retrying
      running
      failed
      succeeded
    }
    treeDepth
  }
  familyProxies(workflows: $wIds) {
    name
    cyclePoint
    state
    childFamilies {
      name
    }
    childTasks(ids: $nIds, states: $nStates) {
      id
      state
      latestMessage
      jobs {
        id
        host
        batchSysName
        batchSysJobId
        submittedTime
        startedTime
        finishedTime
        submitNum
      }
    }
  }
}

variables

{
  "wIds": ["baz"],
  "nIds": ["20*/*"],
  "nStates": ["succeeded", "waiting", "held"]
}

Tree Structure The following query will give you the provision for data driven visualisation... You would need two queries; one every so often (on workflow reload) to find out the treeDepth, and the second would be a recursive query from root constructed to depth (by the client (or sent from UIS?)) using the treeDepth... (we could just nest the query to some arbitrary depth, it will just fill it out to the data depth) query

fragment treeNest on FamilyProxy {
  name
  cyclePoint
  state
  depth
  childTasks(ids: $nIds, states: $nStates, mindepth: $minDepth, maxdepth: $maxDepth) {
    id
    state
    latestMessage
    depth
    jobs {
      id
      host
      batchSysName
      batchSysJobId
      submittedTime
      startedTime
      finishedTime
      submitNum
    }
  }
}

query tree($wIds: [ID], $nIds: [ID], $nStates: [String], $minDepth: Int, $maxDepth: Int) {
  workflows(ids: $wIds) {
    id
    name
    status
    stateTotals {
      runahead
      waiting
      held
      queued
      expired
      ready
      submitFailed
      submitRetrying
      submitted
      retrying
      running
      failed
      succeeded
    }
    treeDepth
  }
  familyProxies(workflows: $wIds, ids: ["root"]) {
    ...treeNest
    childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
      ...treeNest
      childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
        ...treeNest
        childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
          ...treeNest
          childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
            ...treeNest
          }
        }
      }
    }
  }
}

The depth variables just give you the minimum information require for folding. variables

{
  "wIds": ["baz"],
  "nIds": ["20*/*"],
  "nStates": ["succeeded", "waiting", "held"],
  "minDepth": 0,
  "maxDepth": 4 
}

For example, the data would look like this:

{
  "data": {
    "workflows": [
      {
        "id": "sutherlander/baz",
        "name": "baz",
        "status": "held",
        "stateTotals": {
          "runahead": 0,
          "waiting": 0,
          "held": 5,
          "queued": 0,
          "expired": 0,
          "ready": 0,
          "submitFailed": 0,
          "submitRetrying": 0,
          "submitted": 0,
          "retrying": 0,
          "running": 0,
          "failed": 0,
          "succeeded": 3
        },
        "treeDepth": 4
      }
    ],
    "familyProxies": [
      {
        "name": "root",
        "cyclePoint": "20170101T0000+12",
        "state": "held",
        "depth": 0,
        "childTasks": [
          {
            "id": "sutherlander/baz/20170101T0000+12/baa",
            "state": "succeeded",
            "latestMessage": "succeeded",
            "depth": 1,
            "jobs": [
              {
                "id": "sutherlander/baz/20170101T0000+12/baa/01",
                "host": "localhost",
                "batchSysName": "background",
                "batchSysJobId": "3582",
                "submittedTime": "2019-05-28T20:43:57+12:00",
                "startedTime": "2019-05-28T20:43:58+12:00",
                "finishedTime": "2019-05-28T20:44:28+12:00",
                "submitNum": 1
              }
            ]
          }
        ],
        "childFamilies": [
          {
            "name": "FAM4",
            "cyclePoint": "20170101T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [
              {
                "id": "sutherlander/baz/20170101T0000+12/qaz",
                "state": "held",
                "latestMessage": "",
                "depth": 2,
                "jobs": []
              },
              {
                "id": "sutherlander/baz/20170101T0000+12/qux",
                "state": "succeeded",
                "latestMessage": "succeeded",
                "depth": 2,
                "jobs": [
                  {
                    "id": "sutherlander/baz/20170101T0000+12/qux/01",
                    "host": "localhost",
                    "batchSysName": "background",
                    "batchSysJobId": "3586",
                    "submittedTime": "2019-05-28T20:43:57+12:00",
                    "startedTime": "2019-05-28T20:43:58+12:00",
                    "finishedTime": "2019-05-28T20:44:18+12:00",
                    "submitNum": 1
                  }
                ]
              }
            ],
            "childFamilies": []
          },
          {
            "name": "FAM",
            "cyclePoint": "20170101T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [],
            "childFamilies": [
              {
                "name": "FAM2",
                "cyclePoint": "20170101T0000+12",
                "state": "held",
                "depth": 2,
                "childTasks": [],
                "childFamilies": [
                  {
                    "name": "FAM3",
                    "cyclePoint": "20170101T0000+12",
                    "state": "held",
                    "depth": 3,
                    "childTasks": [
                      {
                        "id": "sutherlander/baz/20170101T0000+12/bar",
                        "state": "held",
                        "latestMessage": "",
                        "depth": 4,
                        "jobs": []
                      },
                      {
                        "id": "sutherlander/baz/20170101T0000+12/foo",
                        "state": "succeeded",
                        "latestMessage": "succeeded",
                        "depth": 4,
                        "jobs": [
                          {
                            "id": "sutherlander/baz/20170101T0000+12/foo/01",
                            "host": "localhost",
                            "batchSysName": "background",
                            "batchSysJobId": "3583",
                            "submittedTime": "2019-05-28T20:43:57+12:00",
                            "startedTime": "2019-05-28T20:43:58+12:00",
                            "finishedTime": "2019-05-28T20:44:28+12:00",
                            "submitNum": 1
                          }
                        ]
                      }
                    ],
                    "childFamilies": []
                  }
                ]
              }
            ]
          }
        ]
      },
      {
        "name": "root",
        "cyclePoint": "20170201T0000+12",
        "state": "held",
        "depth": 0,
        "childTasks": [
          {
            "id": "sutherlander/baz/20170201T0000+12/baa",
            "state": "held",
            "latestMessage": "",
            "depth": 1,
            "jobs": []
          }
        ],
        "childFamilies": [
          {
            "name": "FAM4",
            "cyclePoint": "20170201T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [
              {
                "id": "sutherlander/baz/20170201T0000+12/qux",
                "state": "held",
                "latestMessage": "",
                "depth": 2,
                "jobs": []
              }
            ],
            "childFamilies": []
          },
          {
            "name": "FAM",
            "cyclePoint": "20170201T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [],
            "childFamilies": [
              {
                "name": "FAM2",
                "cyclePoint": "20170201T0000+12",
                "state": "held",
                "depth": 2,
                "childTasks": [],
                "childFamilies": [
                  {
                    "name": "FAM3",
                    "cyclePoint": "20170201T0000+12",
                    "state": "held",
                    "depth": 3,
                    "childTasks": [
                      {
                        "id": "sutherlander/baz/20170201T0000+12/foo",
                        "state": "held",
                        "latestMessage": "",
                        "depth": 4,
                        "jobs": []
                      }
                    ],
                    "childFamilies": []
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

Note: The reason why I used the family_proxies as the entry point was to split the result by cycle. (I need to put it under the workflows also)

[hjoliver]

GraphQL is nice :+1:

[kinow]

:eyes: I am trying to imagine how long it would take for me to craft a similar query. Thanks a lot @dwsutherland !!! Will give it a try and hopefully come back with a working UI, created with this data! :tada:

[hjoliver]

(@kinow - I recall you commented on the state filtering checkboxes at the bottom of the screenshot above; in fact that is a very old screenshot and filtering is now controlled via a popup window ... and maybe we'll do something completely different in the web UI).

[dwsutherland]

👀 I am trying to imagine how long it would take for me to craft a similar query. Thanks a lot @dwsutherland !!! Will give it a try and hopefully come back with a working UI, created with this data! 🎉

No probs.. Thankfully that one will work with any workflow, only the depth will need iterated out (obviously you could add/remove fields/variables at your pleasure)..

[kinow]

The Vue components for Trees are amazing, but it would require some further work to allow to display the remaining information, as currently we have actually a Table, with a Tree structure.

This component has a table that supports trees, as well as this other one. They are quicker to produce a working prototype for the tree view IMO, but can't say whether they would be the final solution - as we can decide to completely change the tree view, and then other components would make more sense.

[kinow]

Queries worked with no issues. Had a bit of struggle to update the Python code as I thought the variables were passed as a string value, but they are actually passed as a JS dictionary. The Python code is simply using the Python requests library to do exactly the same as GraphiQL does (I actually copied the requests as curl as reference). Only extra layer is that the vcrpy is then recording whatever traffic requests get and producing the cassettes which I use for Vue.js.

New cassettes rewound and ready to play in Vue now as mocked data :musical_score: https://github.com/kinow/cylc-cassettes/tree/master/cassettes/kinow/five

[kinow]

Got the mocked data in the SuiteService, and created some initial scaffold to use when implementing this view.

The first entry, is the family root of the first moment of the mocked data, when my suite five was held. It has three children also in held. Next will be to put the hierarchy as in the old GUI, but that will be for tomorrow, when I will run 7.8.2 to get a more up to date screenshot.

With this new screenshot as reference, next step will be to use a Vue.js component to create the table with the hierarchy, and finally test with and without the mocked data.

[kinow]

On task filtering, @dwsutherland normally for a a REST application, we would have an endpoint returning the list of possible task states. Do we have anywhere in GraphQL that I can query for that?

[dwsutherland]

@kinow - I don't have a lot of workflow code meta as a query (yet!) neither did the REST one I think, and the list of possible states has been static (hard coded in cylc-flow)... The closest thing at the moment are the fields under `stateTotals' (feild of workflow), hence you could introspec it from the graphql schema...

[kinow]

The closest thing at the moment are the fields under `stateTotals' (feild of workflow), hence you could introspec it from the graphql schema...

Perfect! Will try that for now. With the separation client / server, we may need this and a few more things that weren't necessary with cylc+gtk (essentially same app, as they shared the Python project), e.g. colors, themes, user settings (though there is an issue somewhere about storing user settings client vs. server side I think).

Thanks!

[kinow]

Still looking into components for creating the view. Webix has nice commercial widgets that are free for GPL open source projects. Looks like what we need is a TreeGrid, or DataGrid, or TreeTable - in UI/JS terms I guess.

But what I am looking for now, is actually a component that I can update only specific nodes. This way, in the future, when we get the incremental updates, we won't have to re-render a tree, but instead just update one specific entry.

EDIT: Webix also supports Vue.js and data binding, so we can still use Vue and take advantage of its reactivity.

[kinow]

Ok, going with https://github.com/arnedesmedt/vue-ads-table-tree for now. As it's Vue.js based, it should work with no issues. Its license is OK too. Development is one-man's job, but appears to be still active.

Worst case we can use another component later or use https://github.com/arnedesmedt/vue-ads-table-tree as reference for building our own - not super complex, but takes a long time to be fully complete.

It should work editing the data when elements change, as well as collapse/expand. There's a global filtering option. And the pagination feature probably won't play nice with the tree view of tasks in the pool, so will leave it off for now.

[kinow]

The work for tree view depends on some pending PR's, so for the time being I am working on a branch from mode-offline PR: https://github.com/kinow/cylc-ui/tree/mode-offline-plus-treeview

Issues related:

• https://github.com/arnedesmedt/vue-ads-table-tree/pull/25
• https://github.com/arnedesmedt/vue-ads-table-tree/pull/26
• https://github.com/arnedesmedt/vue-ads-table-tree/pull/27

[kinow]

Component integrated, rendering some simple JS objects/dicts for now.

Next step is to display both the old table and the new table (good for comparison for now I think), and plug in the mocked service data. This involves transforming/massaging some data first.

With that others should be able to review/opinionate, while I will test:

• data syncing: what happens when I add or remove a child? What happens when I change a property in a child node (e.g. change the status field)
• how well does filtering works?
• can we change filtering to support special cases, like status with some sort of check boxes?
• how to make the style look as much Material UI as possible?

[kinow]

We have the mocked data in the format

"familyProxies":[
          {
            "name":"root",
            "cyclePoint":"20130808T0000Z",
            "state":"ready",
            "depth":0,
            "childTasks":[
              {
                "id":"kinow/five/20130808T0000Z/prep",
                "state":"succeeded",
                "latestMessage":"succeeded",
                "depth":1,
                "jobs":[
                  {
                    "id":"kinow/five/20130808T0000Z/prep/01",
                    "host":"localhost",
                    "batchSysName":"background",
                    "batchSysJobId":"11841",
                    "submittedTime":"2019-05-29T23:21:37Z",
                    "startedTime":"2019-05-29T23:21:37Z",
                    "finishedTime":"2019-05-29T23:21:37Z",
                    "submitNum":1
                  }
                ]
              },
              {
                "id":"kinow/five/20130808T0000Z/foo",
                "state":"ready",
                "latestMessage":"",
                "depth":1,
                "jobs":[
                  {
                    "id":"kinow/five/20130808T0000Z/foo/01",
                    "host":"localhost",
                    "batchSysName":"background",
                    "batchSysJobId":"",
                    "submittedTime":"",
                    "startedTime":"",
                    "finishedTime":"",
                    "submitNum":1
                  }
                ]
              },
              {
                "id":"kinow/five/20130808T0000Z/bar",
                "state":"waiting",
                "latestMessage":"",
                "depth":1,
                "jobs":[

                ]
              }
            ],
            "childFamilies":[

            ]
          }
        ]

And here's what the Vue component vue-ads-table-tree expects as parameter:

rows: [
        {
          task: 'foo',
          state: 'Failed',
          host: 'localhost',
          jobId: '12121',
          latestMessage: 'Job failed!',
          depth: 0,
          _showChildren: true,
          _children: [
            {
              task: 'bar',
              state: 'Success',
              host: 'twitwi',
              jobId: '1994',
              latestMessage: 'Job succeeded!',
              depth: 1
            }
          ]
        }
      ],

The rows object is the rows parameter in the table. It must contain an object {} with the attributes to be displayed. These attributes are related to the data in another field, columns.

The _children is a component specific entry, that represents the hierarchy in the table. You are free to add as many children as you would like. But the component only displays the children if _showChildren is set to true.

It appears the component gives the first row a left pad of 1rem. And for each child node it simply adds a normal row, but with a left pad of 1.5rem. There are still more features to test in the plugin.

But one good thing is that I could confirm that the data kept by the Vue application for the rows, once updated, automatically reflects in the component.

So if we keep this data in Vuex and any other part of the application updates the structure used by the Suite Tree View screen, which I believe is going to give a good user experience (plus the app will be ligher/performant with less data duplication/move) :tada:

Still pending collapsing/expanding, filtering, and plug in the real mocked data :grimacing:

[hjoliver]

Sounds great @kinow :+1:

[kinow]

Using mocked data:

Lessons learned so far:

• The current model classes are useful, and can be monkey-patched by transformers. I've created a VueAdsTableTreeTransformer, which gets the GraphQL data and produces the data necessary for the VueAdsTableTree component. In case we experiment with other components, I plan to leave the transformers even if not used (just in case we decide to use the component again). Or at least until we have settled on one component.
• Still needs some theming as it's hard to tell the status, what is a family or task.
• Both Family and Task are displayed as rows in the Suite Tree View. But when you have a Family, only the name and state are actually important - from what I could tell. The other columns are blank as a Family doesn't have a host/jobId/etc.
• If you have a Family called "root", then instead of displaying the family name, what is actually displayed is the cycle point (again, this is not done in our Model classes, it is done in the transformer for the Suite Tree View, which means it doesn't affect other views/screens)
• Developing with offline data, and npm run serve is really nice! If you open a tab in Firefox, and you have the Vue addon (not sure if this is necessary, but nice for developing too), then when you alter code, the npm server reloads the code, and sends a websocket message to your browser tab to refresh :) so one monitor with the browser being automatically refreshed, another one with the IDE :+1:
• Will have to record the messages again as I forgot to include the attribute task from the PbTaskProxy... or at least I hope that's the field I need... I have only the id which gives me something like: "kinow/five/20130808T0000Z/foo", but I want to display only "foo" (and don't want to write code for parsing that if possible)
• Oh, and there is no mention of the suite/workflow name anywhere except for the URL... in the screen shot it's not possible - AFAICT - to tell that that's the suite number five.

EDIT: I cannot add task as that appears to be an object and not string (interesting that I was looking at the diagram only, but that's misleading. Actually had to add the following in the childTasks part of the FamilyProxy fragment:

task {
      name
    }

[kinow]

(posted to chat, copying here)

I've merged all the pending pull requests of cylc-ui (as a way to test it and check for any issues), then worked on top of that to experiment with the first component that could be used for the Suite Tree View, the vue-ads-tree-table. The developer is responsive, but the project is still one-man-job. But still, I found it quite good.

Then used the query suggested by @dwsutherland to retrieve the data for the suite tree view (added only task { name } to the childTasks). With this query, recorded its responses from the graphql endpoint while running the suite five every 5 seconds.

And added the response in a mocked service. The service behaves similar to the normal service, and its signature is the same. But it has an instance value currentTaskIndex that defaults to 0.

Then added a little timer after the Suite Tree View is initialized, to fetch the data again every 2 seconds (so the cassette is playing the suite in fast-forward 😬).

Finally, as one of the pending pull requests is for the mode-offline, which enables the mocked services, I've simply done a NODE_ENV=offline npm run build.... this produced the distribution files for the Cylc UI, with the mocked data/services... and uploaded it to GitHub: https://kinow.github.io/index.html

Now anybody should be able to take a look, and send some feedback.

[kinow]

@hjoliver I think this can be closed as superseded by #145 . WDYT?