GUI for AutoSubmit v1 #30

Closed kinow closed 1 month ago

In GitLab by @dbeltran on Jan 3, 2019, 16:46

@mcastril

Hi there,

I am developing a GUI for autosubmit as you can see in autosubmit@3d4163eea9dc3aff5208b32ab541986d34ca9401.

It will be a separated executable which will be called autosubmit-GUI.

It will be a slow process because it is the first time that I develop any GUI application related, for now, I just committed a version (in an isolated branch) that looks awful but at least I made it work (except grouping/expand) with autosubmit create.

As related to the development strategy:

Create a functional ugly GUI that tries to be intuitive.
Make compatible all the actual console autosubmit functionalities.
Once everything works as well as console autosubmit, renew the ugly GUI for a beautiful one as much as I can.

As for the libraries, for now, I am using PySide2(free for commercial use) <= 5.11 because I have trouble making it work with 5.12 due to having an old version of GCC (4.8).

Of course, all of this is open to discussion.

Edit 1: The code will be a bit mess up until I learn more PySide(QT) since I'm testing different approaches to archive what I want

offtopic: Git buttons for list, bold, etc does not work with chrome.

To do:

[x] Detailed analysis of the grouping function of Autosubmit.
[x] Detailed analysis of wrapping in Autosubmit.
[x] Improve wrapping documentation in Autosubmit.
[x] Improve grouping function in Autosubmit (and in Autosubmit GUI as a consequence)
[x] Implement Clustering (a vis-network feature) as a way to represent grouping in the graph.
[x] Improve justification for wide workflows, so parents are in the middle of their children and dependency lines are not so horizontal.
[x] Add ungroup button in the group menu.
[x] Implement graph positioning algorithm for large experiments.
[x] Implement Live Monitor for Tree View.
[x] Implement Live Monitor wrapper updates for Graph visualization.

To do (Beta Priority):

[x] Show the start date and the date (start date + crunch * chunk size)
[x] Put the description of the experiment next to the name.
[x] The navigation per status buttons should point to the first job with that status, and not to the last one as it is happening now.
[x] In the description panel (right box): Show the list of currently waiting jobs.
[x] In the description panel (right box): Show chunk number.
[x] In the description panel (right box): Show member name.
[x] Remove model and pkl from the top description.
[x] The name of the Branch and HPC should be bigger.
[x] The log should be scrollable.
[x] When switching to the Tree View node, the Tree should be rendered automatically. If the Tree is already rendered and the experiment is running, the Tree View data should be refreshed automatically.
[x] Gather information about the statistics fields in the result and display that information in the statistics tab.
[x] Show average queuing time for failed jobs in the experiment.
[x] Show average running time for jobs (types) in the experiment.
[x] For big experiments, the Graph visualization should start zooming into a relevant section of the graph instead of showing the whole render.

Priority:

[x] Show performance metrics
[x] Provide set status commands through the GUI
[x] Add specific URL for GUI functionality

Architecture discussion on autosubmit#522 for future reference.

In GitLab by @mcastril on Jan 4, 2019, 16:40

Thanks @dbeltran for the initiative, this is a long term task but it is good if we can with a tentative approach.

I suggest you to check what libraries other workflow managers are using. I think they are also based in QT:

ECFLOW: https://confluence.ecmwf.int/display/ECFLOW

Cylc (Rose): https://cylc.github.io/cylc/

PS: About problems with the new GitLab, please report them to https://earth.bsc.es/gitlab/es/requests/issues/513

In GitLab by @dbeltran on Jan 4, 2019, 17:00

mentioned in commit autosubmit@e0491a6ebd6ddd34ce0065ffd246bcdabe050e9c

In GitLab by @dbeltran on Jan 4, 2019, 17:00

mentioned in commit autosubmit@15d0fa9656c80216d0d18fe4ee9a53b18f02b317

In GitLab by @dbeltran on Jan 9, 2019, 13:20

mentioned in commit autosubmit@df821aa7b2c59dba08440a79f5d09c80010ce105

In GitLab by @dbeltran on Jan 9, 2019, 13:20

mentioned in commit autosubmit@e927e4735e68c713b0b62236702132aa724400f7

In GitLab by @dbeltran on Jan 15, 2019, 10:16

Hi there,

ECEFLOW is based in QT5 (pyqt (GPL)) , for now I am using pyside2 which is also based in QT5(LGPL). Cylc: Uses GTK+.

Nowadays, there is not much difference from using one or the other. However, the internet most popular opinion is that QT is the future since it is more active than GTK+.

Since we're aiming also to have a web interface, I saw that qt has webengine which allows to communicate with javascript and html. Also it has a mechanism to combine javascript,html with QT and also a way to expose QT api for be accesible by remote clients.

With that in mind , I think whatever library based in QT5 is complete enough.

In GitLab by @mcastril on Jan 15, 2019, 11:00

Thanks @dbeltran , from what I've read I also thought everything is moving to QT.

A Web interface would be more versatile because obviously you could access your experiments without installing a client from everywhere.

In that sense, it seems promising what you say about the web capabilities. With the WebEngine it seems feasible to create a Desktop application that only visualizes the Web application, so we only would need to develop a WebApp that can be visualized both in the browser or the desktop QT app.

The Qt WebEngine module provides a web browser engine that makes it easy to embed content from the World Wide Web into your Qt application on platforms that do not have a native web engine.

For the communication between QT and HTML/JavaScript I think we would need the QtWebChannel (although maybe not, if you operate directly with the HTML content inside the WebView.

The Qt WebChannel module enables HTML or JavaScript clients to access Qt APIs such as QObject.

In any case, if we focus on a Web Application we would obviously have to develop also a server, which is not a requirement for a Desktop-only application. For an experiment viewer we maybe would only need a DataBase viewer, but we want a GUI that interacts with Autosubmit and the workflow. So in that case a proper server is needed.

So I see two options here:

Desktop app (QT5)
WebApp (HTML+JavaScript, could be AngularJS, ReactJS...) + Server (could be Python) [ + Desktop QTWebEngine ]

In GitLab by @dbeltran on Jan 24, 2019, 10:40

removed milestone

In GitLab by @dbeltran on Jul 12, 2019, 12:13

unassigned @wuruchi

In GitLab by @mcastril on Jul 12, 2019, 13:15

So to recap, the idea would be to create a WebApp that could maybe later be visualized in a separate Desktop application.

An important aspect to discuss will be how to perform the communication between the GUI and Autosubmit. I was initially thinking on a server, but we can maybe keep it simple:

To display the workflow maybe it's enough to read the updated pkl file.
To interact with the workflow and change job status we can maybe use the updated_list ( https://autosubmit.readthedocs.io/en/latest/troubleshoot.html#how-to-change-the-job-status-without-stopping-autosubmit ).

In GitLab by @wuruchi on Jul 12, 2019, 14:50

While I am still looking at the general processes of Autosubmit, I have an idea that you can consider in the meantime until further meetings are held and something more concrete is developed:

The communication between Autosubmit and our WebApp will be handled with web services.

These web services (RESTful resource) will be in charge of executing the shell scripts necessary to retrieve information from Autosubmit (e.g autosubmit monitor a22l), to capture the output, and return it. In the case of monitor, a simplified monitor should be implemented that does not return a pdf, but a plain text file with enough information to rebuild the graph in the front end. The RESTful resource exposes the necessary operations.

Our WebApp will consume web services and show relevant information. Bootstrap + Any web framework you prefer (ReactJS seems to be quite effective nowadays), should do the job in the front end.

The main objective is to avoid changing Autosubmit as much as possible.

In GitLab by @mcastril on Jul 12, 2019, 15:57

Hi Wilmer,

Please note that the Autosubmit status could be retrieved directly from the pkl file, that is being updated in Autosubmit run.

The only "disadvantage" is that you have to generate yourself the graph, but this was one of the things to improve in the WebApp, because Autosubmit is not so efficient doing this in large workflows. Additionally, take into account that in the WebApp you would only need to generate the plot on start, and then update the jobs that are changing their status, which is much more efficient that performing multiple autosubmit monitor.

I am not saying that we will discard the web services (or microservices) option but in the beginning we can live without a server and only develop it if we really need it.

So maybe the first task is to figure out how to generate a plot from the pkl structure in Web environment and in an efficient way.

Of course we will need access from the Web Server to /esarchive, but we also would need it for the web services. This is something that I will try to manage.

In GitLab by @mcastril on Jul 22, 2019, 12:21

In the plans for XIOS I/O server (used by NEMO to output data asynchronously) I saw they are considering to use this library as visual backend. Maybe it's an option to consider:

https://visjs.org/

In GitLab by @wuruchi on Jul 22, 2019, 13:07

That's a cool library. I am sure it will be useful at some point in the presentation layer. Thank you.

I will start to build the REST endpoints from which our GUI will get its information. These endpoints will live in the autosubmit project, since it will make use of its classes and the functionality that will be added in the future. Or perhaps it should be in another project, I am analyzing these possibilities.

In GitLab by @wuruchi on Jul 24, 2019, 11:36

A somewhat important update:

Currently, there is one endpoint running at http://84.88.185.94:5002/monitor/a239 where you can replace a239 for the expid of the experiment you want to monitor (some expid can fail due to reading permissions for my user on some directories). This endpoint returns a JSON representation of the job dependency hierarchy tree, it is running on my PC and published in the local network. Try to access it and test it, also tell me what other important information about jobs we can include.

Now I will start working on the front-end (using ReactJS) that will consume this endpoint and render a user-friendly representation of the JSON data.

In GitLab by @dbeltran on Jul 24, 2019, 11:51

Hi @wuruchi ,

I'm working from home and it seems that I can't access to that IP via the VPN, So I'll check it tomorrow.

In GitLab by @wuruchi on Jul 24, 2019, 12:27

No problem.

Anyway, my computer was dead for almost one hour, I will try to bring the endpoint back online ASAP.

In GitLab by @wuruchi on Jul 24, 2019, 13:00

Should be working again.

In GitLab by @mcastril on Jul 24, 2019, 13:10

Very exciting to read that @wuruchi

I am trying to test it but it keeps on trying to connect

In GitLab by @wuruchi on Jul 24, 2019, 13:15

Should be working now, I was making a test with a huge experiment but it killed the development server. In a real life scenario only the instance that is being served would be killed.

In GitLab by @mcastril on Jul 24, 2019, 13:18

Great! it works now

In GitLab by @wuruchi on Jul 30, 2019, 12:15

Hi @dbeltran @mcastril

A somewhat important milestone for the GUI project. I have implemented an interface using ReactJS and vis.js that renders the dependency hierarchy of an experiment. You can test it here:

http://84.88.185.94:3000/

The expected input is the expid

The next step is to make it also render the color according to the current status of the job. Then, that it refreshes automatically.

@pechevar I think you mentioned that you wanted to test any progress.

UPDATE: Now the graph nodes have their color according to their current status.

In GitLab by @wuruchi on Aug 2, 2019, 16:20

Hi team,

REST Endpoints for Autosubmit have been created:

http://84.88.185.94:8888/expinfo/{expid} where {expid} is a variable. It returns a JSON object with data from the experiment.
http://84.88.185.94:8888/search/{var} where {var} can be a expid or part of the description of the experiment. It returns a list of JSON objects with data from the experiments that coincides with the value sent.
http://84.88.185.94:8888/graph/{expid} Returns a graph representation of the experiment as a JSON object.
http://84.88.185.94:8888/monitor/{expid} Same as the last one but as a tree representation. With these endpoints, it was possible to create a graph representation of the experiment and the current status of the jobs using the network component of the library https://visjs.org/. However, it needs a little bit more polishing. http://84.88.185.94:3000/ still serves the first version of the graph interface.

An Autosubmit experiment searcher has been implemented for testing: http://84.88.185.94:3001/ The graph representation of the experiment’s jobs and status will be included in this WebApp in the short term future as part of the first iteration of the GUI for Autosubmit.

@pechevar

In GitLab by @wuruchi on Aug 12, 2019, 15:16

Hello @mcastril @dbeltran

Some progress has been achieved on the GUI: http://84.88.185.94:3001/

The GUI now also includes the Graph Visualization of the experiment using vis.js. Seems to be working fine with experiments up to 4000 jobs. The toughest one I have tested it is a1de that yielded a result after about 10 - 15 min.
The Graph shows colors according to the state.
The experiment data include the contents of the folder in esarchive. However, only a top 10 is shown to avoid saturation of the interface.

The next step is to update the graph's nodes according to the changes of status of the jobs, and also tell the user what has been changed as text output on the interface.

In GitLab by @mcastril on Aug 12, 2019, 15:24

Nice job @wuruchi ! And I see you can click on the nodes and they are marked, so I guess you will be able to capture the node id to perform an action.

About the performance we will find out a way to improve it for large experiments. Maybe in the future you can do something similar to the grouping done in the monitor (and group-ungroup nodes using clicks). But it is not a priority right now.

In GitLab by @dbeltran on Aug 12, 2019, 15:38

Hi @wuruchi ,

First great work you're progressing so fast,

I noticed that the graph is a bit weird in one of my experiments:

It seems that there is a bug in your graph representation since it is not acknowledging the 000_2_SIM job in it correspondent level, maybe it is due to the way you're traveling for the graph since your graph seem that is traveling with DFS(deep) instead of BFS(Breadth) comparation

In GitLab by @wuruchi on Aug 12, 2019, 17:11

Thank you @mcastril, @dbeltran

As for capturing the data from the graph: Yes, there is a select event that captures the node that is selected and the edge. I have added multi-select (using CTRL-click), and navigation buttons at the bottom of the Graph. Also, I will be working on a panel that shows the names of the nodes currently selected.

Regarding the display of the hierarchical view: After going through the documentation in more detail I found some options that should have improved the sequence of jobs. Please take a look @dbeltran

All these updates are reflected in http://84.88.185.94:3001/

This one generates a cool graph: http://84.88.185.94:3001/experiment/a21g

In GitLab by @wuruchi on Aug 14, 2019, 16:52

Hi @mcastril @dbeltran

Some refactoring in the code has been performed to allow for better performance.

Navigating through the experiments and generating the graphs should be faster now, for example http://84.88.185.94:3001/experiment/a1de (4812 nodes, 6006 edges) took around 10 minutes in the previous version. Now it takes about 20 seconds, around 10 for data retrieval and around 10 more for rendering. This is the heaviest experiment I know, so feel free to try with other bigger experiments if there is any·

Furthermore, now you will see at the bottom of the page that the app now captures the IDs of the nodes you select in a controlled way, so we can use this data to implement further functionality, for example: showing job details.

The project now lives in https://earth.bsc.es/gitlab/wuruchi/autosubmitreact.

PD: Also, since graph rendering has been improved, seamlessly and automatically updating the colors of the nodes seems more feasible now.

@pechevar

In GitLab by @mcastril on Aug 14, 2019, 18:22

Very good news @wuruchi ! Seems really promising. I am trying it through the VPN.

If you want a massive experiment to test, this is it: http://84.88.185.94:3001/experiment/a251

A strategy to do the update in a seamless way could be to generate another canvas in the background and play with the visibility of the layers, but this is an optimization task. I think we can at this stage focus on 90% of the experiments, that will have reasonable size.

In GitLab by @wuruchi on Aug 21, 2019, 17:04

Hello @mcastril @dbeltran

The development server has moved to http://84.88.185.30:3000/.

Press on "Start Job Monitor" in the Experiment page to start the automatic update of Jobs in the Graph. This option will only be visible if the experiment is currently running.

@pabretonniere

In GitLab by @wuruchi on Sep 16, 2019, 12:11

Hello @mcastril @dbeltran @pabretonniere

Autosubmit GUI is back to its old home, al least for now:

http://84.88.185.94:3000/

In GitLab by @mcastril on Sep 16, 2019, 12:29

Hi @wuruchi . It is still not accessible from the new host? Did you answer Kim about the 8081 port?

In GitLab by @wuruchi on Sep 16, 2019, 12:35

On Friday Kim said that the request had been sent, no updates since then.

In GitLab by @wuruchi on Sep 16, 2019, 15:59

For some reason, it was down. It is back up now.

In GitLab by @pabretonniere on Sep 16, 2019, 16:00

Thanks. I can now load the page.

In GitLab by @wuruchi on Sep 19, 2019, 09:41

Hello team,

@dbeltran @mcastril @kserrade @pabretonniere

A somewhat important milestone:

Autosubmit API, the one that does the heavy lifting, is now running on bscesweb04, it is running on 4 threads, meaning that there are actually 4 instances of Autosubmit API that are listening for requests at all times and this will substantially increase performance. As of now, the web app is sending its requests to this API. That should reflect into increased response time, we will see.

I have verified that all the available requests are working as intended but report any bug here, please.

Next step is to also have the GUI running on the server, we will let you know when that happens. URL hasn't changed, yet.

In GitLab by @mcastril on Sep 19, 2019, 09:57

Congratulations @wuruchi and thanks for the update.

In GitLab by @wuruchi on Sep 23, 2019, 15:02

Hello team,

@mcastril @dbeltran

Please let me know if http://84.88.185.94:3000/ is working correctly.

In other news, I will start with the grouping implementation in my local project. I think group by "date, section, chunk, and (perhaps) automatic" is a good functionality to start working on. I hope there is something to show on Friday.

Oh, by the way, seems like there is something wrong with experiment a1wo. Please let me know if you know anything about that experiment.

In GitLab by @dbeltran on Sep 23, 2019, 15:06

@wuruchi , it is working for me.

Great news

I tried it with a257, what is wrong with a1wo?

In GitLab by @wuruchi on Sep 23, 2019, 15:10

The plain text graph representation that is generated by the API triggers a recursion loop in the vis-network algorithm, which could indicate that there is some inappropriate value in the job list of this experiment (a1wo), so far I haven't found anything unusual, and I haven't looked into the vis-network cod. Also, I haven't tried the monitor function of autosubmit since this experiment has more than 12k jobs.

I will keep looking into it.

In GitLab by @wuruchi on Sep 26, 2019, 16:57

Hello team:

@dbeltran @mcastril

The "group by date, member, chunk, automatic" functionality has been implemented in http://84.88.185.94:3000/

However, I deleted from the original code all those parts that tried to modify the packages database, so Autosubmit GUI won't interfere or modify anything. It seems to be working fine with most experiments, although there are a couple that said something along: Package table does not exist.

Please test the functionality.

Now I will work on publishing the App on bscesweb04.

In GitLab by @mcastril on Sep 26, 2019, 19:23

Thank you @wuruchi , this is a really nice improvement. It makes possible to pack the jobs in more understandable workflows.

However I think there's a bug with the chunk grouping. I am trying with http://84.88.185.94:3000/experiment/a204

There should also be a button to disable the grouping (I mean, without redoing the plot by using show graph).

And I have one more suggestion (that you surely have thought of). It would help a lot if you can unfold/fold groups by clicking on the elements. I understand this may not be easy and that we still did not implement contextual boxes when clicking on the jobs, so this is something for the 'to do' list.

In GitLab by @wuruchi on Sep 27, 2019, 09:12

Hello @mcastril

Yes, I see that the output from monitor is different than the output from the app. Grouping works in mysterious ways sometimes, but I think I know what could be the issue there, and is related with to the cyclic nature of the chunk grouping.

As for the functionality of spreading the graph on click on a group, well yes, actually there is a way to do it in vis network and I think it might not be too difficult to implement but after today's meeting, we will have a better idea of a roadmap of priorities, since this thing might take more than a couple of days.

Also, switching back to the complete graph representation without having to make the request again should be implemented in the near future.

@dbeltran I haven't really delved into the grouping code yet, I mean, I know enough to encapsulate it and make it work, but there are details that are beyond me, I will send you an email or post here a list of the code snippets that really put me in awe.

In GitLab by @mcastril on Sep 27, 2019, 09:18

Yes, I see that the output from monitor is different than the output from the app. Grouping works in mysterious ways sometimes, but I think I know what could be the issue there, and is related with to the cyclic nature of the chunk grouping.

It should not have to be necessarily equal but at least to show N groups, where N = NCHUNKS.

As for the functionality of spreading the graph on click on a group, well yes, actually there is a way to do it in vis network and I think it might not be too difficult to implement but after today's meeting, we will have a better idea of a roadmap of priorities, since this thing might take more than a couple of days.

Yes of course, as I said this is a to-do task, given that it will not be the only action to perform on nodes, we have others (show logs, change status, etc). So we will need a context box or something similar.

In GitLab by @dbeltran on Sep 27, 2019, 09:31

@dbeltran I haven't really delved into the grouping code yet, I mean, I know enough to encapsulate it and make it work, but there are details that are beyond me, I will send you an email or post here a list of the code snippets that really put me in awe.

Hi @wuruchi , put them in a new git issue so it doesn't add off-topic in this issue, since it would be information/documentation about a feature, maybe it will be useful in the future for someone else so better keep a track on git and in dedicated issue instead of a mail where it is probable that the information will be gone in a few months

In GitLab by @wuruchi on Sep 27, 2019, 15:34

Noted.

Hello team.

@dbeltran @mcastril @pabretonniere @pechevar @kserrade

AutosubmitGUI is finally in its permanent home http://bscesweb04.bsc.es/autosubmitapp/.

It should be working just fine, but report any 404 error or anything you feel out of place.

Grouping functionality has been added in this release, but it's still in early alpha, anyhow, try to test it and compare to the results from monitor of your experiments, I am aware there is one bug, but also I am sure there some more. This function needs more work.

The immediate task is to add wrappers to the Graph visualization and allow for smooth switching between Grouped representation and Full representation.

Graph node recolor does not work with Grouped Representation when executing Job Monitor, as you will see.

Also, suggestions about the layout and color palette are mostly welcomed.

In GitLab by @dbeltran on Sep 27, 2019, 15:59

Hi @wuruchi ,

Great news, the new home is working for me in 2 experiments no problems so far.

I tested a bit the grouping, and is working on 2/3 experiments fine (same output than in master branch) a28k gives an error with package_table like you mentioned in previous comments however without the options of expand or expand_status I can't really test much so I'll be waiting to the next version.

One thing that I noticed is while the searching of experiment I can barely see the previous searches that I did (attaching photo)

Screenshot_from_2019-09-27_15-56-09

In GitLab by @wuruchi on Sep 27, 2019, 16:05

Thank you @dbeltran

I was already forgetting about those options. They are next on the list.

About the dark color, might it be because of your google chrome theme? I am using the default theme. Anyway, I will look into some simple fix for it.

In GitLab by @dbeltran on Sep 27, 2019, 16:13

I think you're right, it is the theme because it makes the background black. However, so far, I only had issues with this page due it is maintaining the letter color in black instead of swap to white.

About the expand options, you could do it via buttons for now.

Edit: (the expand_status option, the other would be too clunky via buttons )

And later , as @mcastril mentioned:

It would help a lot if you can unfold/fold groups by clicking on the elements. I understand this may not be easy and that we still did not implement contextual boxes when clicking on the jobs, so this is something for the 'to do' list.

Implement also this.

In GitLab by @mcastril on Sep 27, 2019, 18:25

Congratulations @wuruchi . This deployment was not easy but finaly it's there. No you can focus on the development tasks.

I have not seen any bug besides the one commented by you and Daniel (package_table) but I will tell in case.

One thing that I commented and that still could be improved is to increment the separation between the nodes in the graphic. When an experiment has n parent jobs, each one having m children, if n is big and m is >2 , then a lot of jobs (n*m) are accumulated in the second level , being much more than n, and all the dependency lines are creating a big black stripe. I think this could be improved by incrementing the space in n , but probably needs also that the parent is in the center of the children. For some experiments this is respected but for some others they are justified on the left producing this issue.

http://bscesweb04.bsc.es/autosubmitapp/experiment/a27v

BSC-ES / autosubmit-gui

GUI for AutoSubmit v1 #30