GUI for AutoSubmit v1

[kinow]

In GitLab by @dbeltran on Jan 3, 2019, 16:46

@mcastril

Hi there,

I am developing a GUI for autosubmit as you can see in autosubmit@3d4163eea9dc3aff5208b32ab541986d34ca9401.

It will be a separated executable which will be called autosubmit-GUI.

It will be a slow process because it is the first time that I develop any GUI application related, for now, I just committed a version (in an isolated branch) that looks awful but at least I made it work (except grouping/expand) with autosubmit create.

As related to the development strategy:

• Create a functional ugly GUI that tries to be intuitive.
• Make compatible all the actual console autosubmit functionalities.
• Once everything works as well as console autosubmit, renew the ugly GUI for a beautiful one as much as I can.

As for the libraries, for now, I am using PySide2(free for commercial use) <= 5.11 because I have trouble making it work with 5.12 due to having an old version of GCC (4.8).

Of course, all of this is open to discussion.

Edit 1: The code will be a bit mess up until I learn more PySide(QT) since I'm testing different approaches to archive what I want

offtopic: Git buttons for list, bold, etc does not work with chrome.

To do:

• [x] Detailed analysis of the grouping function of Autosubmit.
• [x] Detailed analysis of wrapping in Autosubmit.
• [x] Improve wrapping documentation in Autosubmit.
• [x] Improve grouping function in Autosubmit (and in Autosubmit GUI as a consequence)
• [x] Implement Clustering (a vis-network feature) as a way to represent grouping in the graph.
• [x] Improve justification for wide workflows, so parents are in the middle of their children and dependency lines are not so horizontal.
• [x] Add ungroup button in the group menu.
• [x] Implement graph positioning algorithm for large experiments.
• [x] Implement Live Monitor for Tree View.
• [x] Implement Live Monitor wrapper updates for Graph visualization.

To do (Beta Priority):

• [x] Show the start date and the date (start date + crunch * chunk size)
• [x] Put the description of the experiment next to the name.
• [x] The navigation per status buttons should point to the first job with that status, and not to the last one as it is happening now.
• [x] In the description panel (right box): Show the list of currently waiting jobs.
• [x] In the description panel (right box): Show chunk number.
• [x] In the description panel (right box): Show member name.
• [x] Remove model and pkl from the top description.
• [x] The name of the Branch and HPC should be bigger.
• [x] The log should be scrollable.
• [x] When switching to the Tree View node, the Tree should be rendered automatically. If the Tree is already rendered and the experiment is running, the Tree View data should be refreshed automatically.
• [x] Gather information about the statistics fields in the result and display that information in the statistics tab.
• [x] Show average queuing time for failed jobs in the experiment.
• [x] Show average running time for jobs (types) in the experiment.
• [x] For big experiments, the Graph visualization should start zooming into a relevant section of the graph instead of showing the whole render.

Priority:

• [x] Show performance metrics
• [x] Provide set status commands through the GUI
• [x] Add specific URL for GUI functionality

Architecture discussion on autosubmit#522 for future reference.

[kinow]

In GitLab by @mcastril on Jan 27, 2021, 15:17

What I meant in my last message was that I think something is wrong at the AS level. It is like AS set all job logs to the same time at some point (and curiously with today's date).

[kinow]

In GitLab by @mcastril on Jan 27, 2021, 15:24

I think this is a separate issue that should be treated separately. I opened autosubmit#649

[kinow]

In GitLab by @etourign on Jan 27, 2021, 15:57

sorry @mcastril I should have opened an issue myself. Thanks for your help all!

[kinow]

In GitLab by @etourign on Jan 27, 2021, 15:57

I will be working on a workaround extension for this feature that will look for the latest existing job log (.out or .err depending on the case), if the log listed in the pkl does not exists and the job has been finished.

I will post here when it is done.

that sounds like a great idea, thanks a lot!

[kinow]

In GitLab by @mcastril on Jan 27, 2021, 16:03

No worries @etourign .

I will be working on a workaround extension for this feature that will look for the latest existing job log (.out or .err depending on the case), if the log listed in the pkl does not exists and the job has been finished.

Would this mean that you cannot see logs from a previous experiment iteration? I understand if it's not possible in case we are not saving this information in the database.

[kinow]

In GitLab by @wuruchi on Jan 27, 2021, 16:49

Would this mean that you cannot see logs from a previous experiment iteration? I understand if it's not possible in case we are not saving this information in the database.

Currently, the out and err log names are not stored in the job historical database. Do you think we should do it?

[kinow]

In GitLab by @mcastril on Jan 27, 2021, 17:09

Do you think that it would entail a lot of overhead?

Given that we already store and show historical job information, I think it would be really powerful to associate logs with step iterations.

Many times jobs succeed after a retrial, so I think we can make good use of this information.

[kinow]

In GitLab by @wuruchi on Jan 27, 2021, 18:43

Hello @mcastril

I had seen the need to add some fields (e.g. MaxRSS and AveRSS) but I was waiting until we finish more important issues. I will include these 2 (out and err) new fields in the upcoming changes.

[kinow]

In GitLab by @wuruchi on Jan 28, 2021, 13:36

Hello @etourign @mcastril

The out and err fix has been implemented in Autosubmit GUI. Please examine if the logs correspond to the latest run of the job.

[kinow]

In GitLab by @etourign on Jan 28, 2021, 13:42

seems to work for a3e8, thanks a lot! Will let you know if I run into other issues.

I guess Showing last 150 lines. is to make sure that long logs are not transferred? I think it is reasonable, but with time we might find we need a few more lines.

[kinow]

In GitLab by @mcastril on Jan 28, 2021, 17:12

Thanks @wuruchi !

I guess Showing last 150 lines. is to make sure that long logs are not transferred? I think it is reasonable, but with time we might find we need a few more lines.

I think it's enough for 90% of the cases. Otherwise we always can look the real files, but no need to add more load if it's not generalised. Anyway we will look with time.

@etourign, @dbeltran is still fixing the pkl issue so it should stop happening in the future once the fix is applied. @wuruchi's fix tries to be resilient against "bad pkls".

[kinow]

In GitLab by @wuruchi on Mar 3, 2021, 15:27

marked the checklist item In the description panel (right box): Show the list of currently waiting jobs. as completed

[kinow]

In GitLab by @wuruchi on Apr 12, 2021, 12:24

marked the checklist item Gather information about the statistics fields in the result and display that information in the statistics tab. as completed

[kinow]

In GitLab by @wuruchi on Apr 12, 2021, 12:26

I am marking

Gather information about the statistics fields in the result and display that information in the statistics tab.

As completed, since the implementation of the historical database, and the recently implemented summary view provide related information that we can expand.

[kinow]

In GitLab by @wuruchi on Apr 12, 2021, 12:28

To do (Beta):

• [ X ] Try adding an authentication method.
• [ X ] Add contextual actions in the nodes that allow the user to see the log of the node, change status, re-run job or change platform.

I think we can try these in the next iteration of Autosubmit GUI.

[kinow]

In GitLab by @mcastril on Jan 3, 2019, 16:46

moved from autosubmit#363