GUI for AutoSubmit v1 #30

Closed kinow closed 1 month ago

In GitLab by @dbeltran on Jan 3, 2019, 16:46

@mcastril

Hi there,

I am developing a GUI for autosubmit as you can see in autosubmit@3d4163eea9dc3aff5208b32ab541986d34ca9401.

It will be a separated executable which will be called autosubmit-GUI.

It will be a slow process because it is the first time that I develop any GUI application related, for now, I just committed a version (in an isolated branch) that looks awful but at least I made it work (except grouping/expand) with autosubmit create.

As related to the development strategy:

Create a functional ugly GUI that tries to be intuitive.
Make compatible all the actual console autosubmit functionalities.
Once everything works as well as console autosubmit, renew the ugly GUI for a beautiful one as much as I can.

As for the libraries, for now, I am using PySide2(free for commercial use) <= 5.11 because I have trouble making it work with 5.12 due to having an old version of GCC (4.8).

Of course, all of this is open to discussion.

Edit 1: The code will be a bit mess up until I learn more PySide(QT) since I'm testing different approaches to archive what I want

offtopic: Git buttons for list, bold, etc does not work with chrome.

To do:

[x] Detailed analysis of the grouping function of Autosubmit.
[x] Detailed analysis of wrapping in Autosubmit.
[x] Improve wrapping documentation in Autosubmit.
[x] Improve grouping function in Autosubmit (and in Autosubmit GUI as a consequence)
[x] Implement Clustering (a vis-network feature) as a way to represent grouping in the graph.
[x] Improve justification for wide workflows, so parents are in the middle of their children and dependency lines are not so horizontal.
[x] Add ungroup button in the group menu.
[x] Implement graph positioning algorithm for large experiments.
[x] Implement Live Monitor for Tree View.
[x] Implement Live Monitor wrapper updates for Graph visualization.

To do (Beta Priority):

[x] Show the start date and the date (start date + crunch * chunk size)
[x] Put the description of the experiment next to the name.
[x] The navigation per status buttons should point to the first job with that status, and not to the last one as it is happening now.
[x] In the description panel (right box): Show the list of currently waiting jobs.
[x] In the description panel (right box): Show chunk number.
[x] In the description panel (right box): Show member name.
[x] Remove model and pkl from the top description.
[x] The name of the Branch and HPC should be bigger.
[x] The log should be scrollable.
[x] When switching to the Tree View node, the Tree should be rendered automatically. If the Tree is already rendered and the experiment is running, the Tree View data should be refreshed automatically.
[x] Gather information about the statistics fields in the result and display that information in the statistics tab.
[x] Show average queuing time for failed jobs in the experiment.
[x] Show average running time for jobs (types) in the experiment.
[x] For big experiments, the Graph visualization should start zooming into a relevant section of the graph instead of showing the whole render.

Priority:

[x] Show performance metrics
[x] Provide set status commands through the GUI
[x] Add specific URL for GUI functionality

Architecture discussion on autosubmit#522 for future reference.

In GitLab by @wuruchi on Sep 30, 2019, 09:20

Hello @mcastril

As you mention, there is still some more work to do regarding the graph representation. Some of the problems you mentioned are related to the limitations of the library we are using since it tries to be a general tool for any graph; meanwhile, our experiments are a particular graph instance of this universe. There is so much configuration can do as a general approach, meaning that we can configure only that much the options provided by the vis-network API; however, there are still some approaches to try, and I think we can extend the functionality of this library to a satisfactory state.

In GitLab by @mcastril on Sep 30, 2019, 09:39

Thanks @wuruchi . I understand. I think we must keep in mind the limitations of this library and try to stick to changing only configuration parameters before doing more complex things as adding new functions and so on. Then, when we know all the limitations and things that can't be satisfied, we can decide whether extending the functionality of this library or evaluating the usage of another one.

We will add then suggestions or pending fixes as checkboxes at the top (in the description of the issue). We will also add there other eventual actions from the meeting with Paco. I would ask you then to mark the things that are done, and for the undone comment if it's straight forward or not to do it (and estimated effort).

In GitLab by @wuruchi on Sep 30, 2019, 12:06

Hi @mcastril

Speaking of that, this library has a nice clustering function that we can use for grouping. However, to implement it effectively I will have to analyze in detail the rules that apply in the grouping function of Autosubmit. This is related to what I told @dbeltran before about making a list of the code snippets related to this function that are too difficult.

So, I think it is best to start with the complete analysis of this function, which also includes wrappers and synchronization.

The current implementation of grouping seems to work well enough for most cases so perhaps it is time to tell our beta users to try the app and give us feedback. What do you think?

In GitLab by @mcastril on Sep 30, 2019, 17:34

Hi, I think it is important to fully understand what Autosubmit group does in order to use it in the GUI. I think this functionality is underused in Autosubmit, partially because lack of knowledge about its potential, partially because of the difficulty of applying it through commands. A GUI always facilitates this work.

About the Beta, we should meet Paco first. As we gonna meet tomorrow, after the meeting we can decide if doing the publication then or do some changes in advance. I think that, even if it's a Beta, we have to provide user's a good experience so they encourage the others using the tool.Hi, I think it is important to fully understand what Autosubmit group does in order to use it in the GUI. I think this functionality is underused in Autosubmit, partially because lack of knowledge about its potential, partially because of the difficulty of applying it through commands. A GUI always facilitates this work.

About the Beta, we should meet Paco first. As we gonna meet tomorrow, after the meeting we can decide if doing the publication then or do some changes in advance. I think that, even if it's a Beta, we have to provide user's a good experience so they encourage the others using the tool.

In GitLab by @wuruchi on Oct 1, 2019, 09:02

marked the checklist item Detailed analysis of the grouping function of Autosubmit. as completed

In GitLab by @wuruchi on Oct 1, 2019, 09:02

marked the checklist item Detailed analysis of the grouping function of Autosubmit. as incomplete

In GitLab by @wuruchi on Oct 7, 2019, 12:02

marked the checklist item Detailed analysis of the grouping function of Autosubmit. as completed

In GitLab by @wuruchi on Oct 7, 2019, 15:57

marked the checklist item Implement Clustering (a vis-network feature) as a way to represent grouping in the graph. as completed

In GitLab by @wuruchi on Oct 7, 2019, 15:57

marked the checklist item Implement Clustering (a vis-network feature) as a way to represent grouping in the graph. as incomplete

In GitLab by @wuruchi on Oct 7, 2019, 16:19

Hello team.

@dbeltran @mcastril

Now that the mysteries of grouping have been unraveled, it is time to think about how to improve this useful function so it is better understood by our users, and more widely used as a consequence. Perhaps we should open a new issue for this purpose or use the one that I already opened? Anyway, the bottom line is that the way grouping is handled now is not very efficient, I mean, the graph is made smaller but the pre-processing is still the same as if you were calling the whole experiment. Also, some relations are lost or oversimplified in the process given the complexity of the dependencies of some experiments. I think a better approach could be clustering, where jobs that belong to the same chunk, member, or date, are clustered together but jobs with many dependencies, e.g. asynchronous jobs, are left out of these clusters, so the workflow of the experiment is preserved. For that purpose, automatic grouping should disappear and some general rules established when grouping by date, member, or chunk; so there is no necessity to include further filters.

Another task is to perform a similar analysis of the wrapping function and improve its documentation. Should I open a new issue for this purpose as I did with "Grouping"?

The experiment tree view is currently under development, a first implementation should be up this week.

In GitLab by @mcastril on Oct 7, 2019, 18:09

Thanks @wuruchi

We can discuss about this in general in the meeting tomorrow.

I think that rather of an issue you can create a milestone (with an issue inside about the discussion). I think that it's better to split the process in small tasks.

In GitLab by @wuruchi on Oct 9, 2019, 15:44

Hello team,

@dbeltran @mcastril

After reviewing and testing some procedures for grouping (clustering) and wrapping identification (draw boxes around jobs that belong to the same wrapper), it seems that it will be necessary, and the best approach, to implement graphviz graph positioning algorithm:

A Technique for Drawing Directed Graphs

After this, I believe, not only will be the graph representation in Autosubmit GUI the same as the monitor output, but also faster will allow for clustering.

In GitLab by @mcastril on Oct 9, 2019, 17:56

Hi @wuruchi . Is it implemented in graphviz library? In that case I understand you mean it should be implemented into visjs.

In GitLab by @wuruchi on Oct 10, 2019, 09:08

Hi @mcastril

It is currently implemented in graphviz. The objective is that the API sens the (x,y) coordinates for each node according to the graphviz algorithm. If this is possible using the current implementation, that would be the best approach, but if not, then the algorithm should be implemented to produce the coordinates for each node.

In GitLab by @wuruchi on Oct 11, 2019, 17:32

Hello team

@dbeltran @mcastril

After thoroughly testing the monitor function and how it works with graphviz to produce the coordinates necessary to produce the same layout in the GUI as it appears in the monitor output (pdf, svg, etc.), I have arrived at some findings:

The pydotplus graphviz implementation is capable of returning the coordinates (x,y) for each node of the graph. However, the bigger the graph, the longer it takes to produce this output.
In some cases it will work less slow than usual, and that is because there are packages (wrappers) in the experiment, these packages are defined as clusters during the monitor process, which act as subgraphs in the construction process of the graph, so it simplifies the coordinate calculation and the whole graph generation process becomes a divide and conquer situation. Now the problem are the experiments that do not have clusters (wrappers), or that are too big, in this case, the monitor function (if not sent with grouping) will take forever. Try a1zm without grouping, notice that its package table is empty.
Also, that is the reason a1de monitor didn't work and just got stuck there in the terminal. Maybe after some hours, it would have produced something.
In conclusion, any experiment without packages (wrappers) defined, with 500+ nodes, will just take forever to produce a monitor output.

In GitLab by @wuruchi on Oct 11, 2019, 17:55

@dbeltran @mcastril

So, those were the bad news.

On the bright side, there are some things that can be done about this:

If an experiment does not have packages (wrappers) defined, then in the monitor process, and just for that purpose, some abstract packages could be created and used to simplify the graphviz workload, as many as necessary depending on the size of the experiment.
Another way is to disregard graph positioning according to graphviz and keep the positioning generated by vis-network. This option implies that wrappers will not be shown in the GUI Graph with a dashed box, and certainly they will not be similar to the monitor output, but the nodes belonging to a wrapper will be highlighted when the user selects the name of a wrapper. Same situation in the Tree View representation of the experiment. Of course, here the main drawback is that the position of nodes will not be very familiar, but at least vis-network clustering can help the user make sense of it.

I think that implementing abstract clustering to help graphviz calculate positioning is the most reliable approach right now. Now, not every experiment that includes wrappers has enough wrappers to make the positioning process fast enough, but for GUI purposes this is not a problem, we will just treat any experiment as having no wrappers, and include those real wrappers later. Now there is the objective of making the autosubmit monitor function also benefit from these discoveries and general improvements, and for that purpose, I think that any experiment without wrappers should include abstract clustering.

Suggestions?

In GitLab by @wuruchi on Oct 11, 2019, 18:06

mentioned in issue autosubmit#353

In GitLab by @mcastril on Oct 17, 2019, 19:15

Thanks for the clear explanation @wuruchi , it is really useful to know this kind of things and start to see how to attack them.

In GitLab by @mcastril on Oct 17, 2019, 19:31

If an experiment does not have packages (wrappers) defined, then in the monitor process, and just for that purpose, some abstract packages could be created and used to simplify the graphviz workload, as many as necessary depending on the size of the experiment.

Clever. I guess you mean for performance purposes right?

Of course, here the main drawback is that the position of nodes will not be very familiar, but at least vis-network clustering can help the user make sense of it.

So would be the job wrappers visually identified by some manner? Or you mean they only will be put in another colour after the user performs some action.

I think that implementing abstract clustering to help graphviz calculate positioning is the most reliable approach right now.

It seems so. So if I understood well, we would generate fake wrappers to increase Graphviz performance. Then Visjs will use this information to display wrappers. Then there should be a property to distinguish real and fake clusters.

So is there the possibility to send the coordinates from Graphviz to Visjs? You mentioned this in this comment (https://earth.bsc.es/gitlab/es/autosubmit/issues/363#note_56606) but it is not clear to me if it is definitely possible.

Maybe you are suggesting that the main constraint here is the performance and the GUI cannot wait forever for the Graphviz to generate the clusters?

In GitLab by @wuruchi on Oct 18, 2019, 09:42

Clever. I guess you mean for performance purposes right?

Yes, only for performance.

So would be the job wrappers visually identified by some manner? Or you mean they only will be put in another colour after the user performs some action.

Yes, in the situation where we don't have the graphviz layour, wrappers can be identified by listing the graphs on the side of the graph, and when the user clicks on an item of that list, the jobs that belong to that wrapper are highlighted.

It seems so. So if I understood well, we would generate fake wrappers to increase Graphviz performance. Then Visjs will use this information to display wrappers. Then there should be a property to distinguish real and fake clusters.

Yes, but Visjs will use this positioning not only to display wrappers but also to work with clustering, which is a function that will replace grouping and will allow for cluster disaggregation on user click (when a user clicks on a cluster, the cluster opens and shows the nodes that were inside).

So is there the possibility to send the coordinates from Graphviz to Visjs? You mentioned this in this comment (https://earth.bsc.es/gitlab/es/autosubmit/issues/363#note_56606) but it is not clear to me if it is definitely possible.

Yes, right now that works (in my development implementation), but for some experiments like the ones that do not have wrappers it is VERY slow. I mention this on https://earth.bsc.es/gitlab/es/autosubmit/issues/353#note_56812.

So the problem right now is to find a way to make this process faster for the average experiment. That's when "abstract wrappers" the method I suggest in my previous comment might help. Basically, we partition the experiment tree into subtrees so graphviz works faster, and in consequence, we get the coordinates faster, which translates into a faster rendering of the graph in the GUI.

In GitLab by @wuruchi on Nov 27, 2019, 13:41

Hello @mcastril @dbeltran

You can see that some changes have been implemented in the Autosubmit GUI. An example:

http://bscesweb04.bsc.es/autosubmitapp/experiment/a2dn

Now there are two options for the user in the Graph Tab: "Classic" and "Grouped". Classic shows the usual graph with all its nodes, Grouped will group the nodes by date and member, where when the user double clicks on a group, it opens to reveal all the nodes contained inside. However, there are some instances for which the grouped option will not work, namely, those that cannot be computed by graphviz in a reasonable time.

As an added feature, each tab has its own command bar, just to make it clear which options correspond to each tab.

Continuing with the Graph Visualization, right next to the graph there are two tabs: the first is the usual selection function that already existed, and the other is a list of Wrappers where you can click on any of the wrappers and the nodes that belong to that wrapper will be highlighted in the graph. Currently, this list is not updated while executing the Job Monitor, but that will be implemented along with wrapper visualization in the Graph (as noted by @dbeltran).

There is some not negligible fraction of experiment that cannot be handled in a reasonable time by graphviz, and when we do not have the x and y coordinates generated by graphviz and resort to the positions calculated by vis network, some options are not properly shown, i.e. "Reasonable wrapper clustering, edge colors for wrappers, grouping". For that reason, I am now focusing my efforts on developing an algorithm that generates the x and y coordinates for these, usually, large experiments in efficient time. Once that is completed, a third option can be added to the Graph tab, so the user can choose which algorithm they want to try for the visualization.

In GitLab by @wuruchi on Nov 27, 2019, 13:41

marked the checklist item Implement Clustering (a vis-network feature) as a way to represent grouping in the graph. as completed

In GitLab by @wuruchi on Nov 27, 2019, 13:41

marked the checklist item Improve justification for wide workflows, so parents are in the middle of their children and dependency lines are not so horizontal. as completed

In GitLab by @wuruchi on Nov 27, 2019, 13:42

marked the checklist item Add ungroup button in the group menu. as completed

In GitLab by @wuruchi on Nov 28, 2019, 10:06

Hi @mcastril @dbeltran

Just to give some details on the current implementation of the Grouping function of Autosubmit GUI:

For current purposes, clusters refers to the graphical representation, and groups to the logical.

Autosubmit GUI implements an Automatic Grouping option that groups jobs into date-member clusters. These clusters have a color according to the following Status hierarchy:

FAILED
RUNNING
SUSPENDED
SUBMITTED
QUEUING
COMPLETED

Meaning that if all jobs have the status COMPLETED in the group, the cluster will have the COMPLETED color, which is yellow in the current implementation. Furthermore, if all are completed but there is at least one job QUEUING, the color will change to the QUEUING color; and so on. FAILED rules all colors.

Not clustering for a date-member group given certain conditions is not yet implemented. For example, if in a group a job has the status FAILED, then the jobs belonging to that group will not be merged into a cluster. Suggestions for rules for this feature are welcome. A trivial one could be that groups with FAILED jobs should not be clustered.

In GitLab by @mcastril on Dec 2, 2019, 16:40

Now there are two options for the user in the Graph Tab: "Classic" and "Grouped". Classic shows the usual graph with all its nodes, Grouped will group the nodes by date and member, where when the user double clicks on a group, it opens to reveal all the nodes contained inside. However, there are some instances for which the grouped option will not work, namely, those that cannot be computed by graphviz in a reasonable time.

I couldn't see the groups in that particular experiment but I saw the feature in others that you told me.

As an added feature, each tab has its own command bar, just to make it clear which options correspond to each tab.

Much better now. Much more clean interface.

Continuing with the Graph Visualization, right next to the graph there are two tabs: the first is the usual selection function that already existed, and the other is a list of Wrappers where you can click on any of the wrappers and the nodes that belong to that wrapper will be highlighted in the graph.

Nice!

Unless we change the visualization I think it should be clear somewhere that squared boxes are wrappers.

In GitLab by @wuruchi on Dec 17, 2019, 14:13

Hello @mcastril

Looks like something has happened to the autosubmit experiment database /esarchive/autsubmit/ecearth.db. The details table is empty. Can you take a look at the cron job?

In GitLab by @pechevar on Dec 17, 2019, 14:17

Hi!

I'm trying to fix it but looks like there are a duplicate id

@mcastril did you fix it some months ago ?

...

User: dbeltran
Created: 2019-12-16 09:15:45.722666
Model: https://earth.bsc.es/gitlab/es/auto-ecearth3.git
Branch: trunk
HPC: marenostrum4
Inserting a2g9 details into db
a2g9 details inserted successfully!
Looking for a2g9 db id
Id on db is 4591
Looking for a2g9 details
Describing a2g9

Checking configuration files...
autosubmit_a2g9.conf OK
platforms_a2g9.conf OK
jobs_a2g9.conf OK
expdef_a2g9.conf  OK
Configuration files OK

User: dbeltran
Created: 2019-12-16 09:15:45.722666
Model: https://earth.bsc.es/gitlab/es/auto-ecearth3.git
Branch: trunk
HPC: marenostrum4
Inserting a2g9 details into db
Traceback (most recent call last):
  File "populate_db.py", line 138, in <module>
    main()
  File "populate_db.py", line 12, in main
    populate_db(args.db_file)
  File "populate_db.py", line 66, in populate_db
    create_details(conn, details)
  File "populate_db.py", line 108, in create_details
    cur.execute(sql, details)
sqlite3.IntegrityError: UNIQUE constraint failed: details.exp_id

In GitLab by @pechevar on Dec 17, 2019, 15:02

Looks like some @dbeltran experiments don't have properly set the EXPID. He is fixint it now.

In GitLab by @pechevar on Dec 17, 2019, 15:08

It was fixed

In GitLab by @mcastril on Dec 17, 2019, 15:19

[x] Developing refreshing of the tree view.
[x] Information on QUEING, RUNNING jobs, besides COMPLETED ones in the collapsed tree.
[x] Queued and running time for COMPLETED jobs. Database to store the times.
[x] Re-arrange the view and try putting experiment info on a column at the left side.
[x] Grouping: By default group by status.
[x] Show Running experiments on Home view. Cron job to search for running exps every 10 mins?

In GitLab by @wuruchi on Jan 3, 2020, 12:42

marked the checklist item Detailed analysis of wrapping in Autosubmit. as completed

In GitLab by @wuruchi on Jan 3, 2020, 12:42

marked the checklist item Implement Live Monitor wrapper updates for Graph visualization. as completed

In GitLab by @wuruchi on Jan 3, 2020, 12:42

marked the checklist item Implement graph positioning algorithm for large experiments. as completed

In GitLab by @wuruchi on Jan 3, 2020, 12:42

marked the checklist item Improve grouping function in Autosubmit (and in Autosubmit GUI as a consequence) as completed

In GitLab by @wuruchi on Jan 3, 2020, 13:20

marked the checklist item Improve wrapping documentation in Autosubmit. as completed

In GitLab by @wuruchi on Jan 3, 2020, 13:20

marked the checklist item Implement Live Monitor for Tree View. as completed

In GitLab by @wuruchi on Jan 3, 2020, 14:41

Hello @mcastril @dbeltran

As you can see, I have been putting a check in some of the pending tasks that are operational enough to be considered complete. Anyway, you can check on your own, since I have uploaded all the changes to the server: http://bscesweb04.bsc.es/autosubmitapp/.

Graph representation is completely functional. It now updates wrappers (this was pending) and shows them accordingly while Job Monitor is active, it also shows HELD jobs. We have these options for visualization:

Classic: It will attempt to use Graphviz but if the experiment is too large for that, it will use an ad-hoc algorithm.
Laplacian: Uses the principles of Spectral Graph Layout.
Grouped by D-M: Groups the result by Date and Member
Grouped by Status: Groups all the WAITING jobs.

The "Grouped by" options are more or less a first draft of what they can do. So, it is open to suggestions.

The Tree View now has a "Refresh" button that will update all the data of the experiment (if it is running). I decided to include it as a button that should be clicked each time the user wants to update the experiment, as opposed to the "Job Monitor" option of the Graph that is clicked once to activate it and once to deactivate it, to see which one is preferred by the user. However, that is just a first draft; in the next meeting, we can discuss which way is best for our purposes.

The next week I should finish the big pending tasks of the GUI:

Queued and Running times for COMPLETED jobs.
Show Running Experiments on Home View.
Show .out and .err logs through the GUI.
Re-arrange layout of components of the GUI.

Tasks 1 and 2 require the development of an independent program that will query the autosubmit files in regular intervals to gather information.

In GitLab by @wuruchi on Jan 7, 2020, 09:54

Hello @mcastril

I remember that some time ago you sent a link to some kind of wiki that explained how to setup "populate_db", the python project that populates the autosubmit database. Can you post it here? If it exists, perhaps I am confusing it for something else.

In GitLab by @kserrade on Jan 7, 2020, 10:08

@wuruchi I think you're looking this information

https://earth.bsc.es/gitlab/es/auto-ecearth3/wikis/web_experiments

In GitLab by @wuruchi on Jan 8, 2020, 16:15

Hello @mcastril @dbeltran

There is a new table in /esarchive/autosubmit/ecearth.db called "experiment_status", this table has the following columns:

exp_id: Id of the experiment, same as the exp_id in 'details' table.
name: expid
status: "RUNNING" if the experiment is running, "NOT RUNNING" otherwise. If the experiment is not in the table, that means that is definitely NOT RUNNING and has not been running since the updater started working.
seconds_diff: Difference in seconds between the current time and the last time the experiment was detected as running. If negative, just consider it as 0.
modified: Date and time the experiment was modified, either changed status or inserted into the table.

The table is updated every 5 minutes if an updater task, which is part of the "autosubmitAPIwu" package and runs under crontab in bscesweb04, detects it as running. The process has been optimized, so the 5 minutes interval should not result in excessive traffic. Basically, the task looks at the pkl files of each experiment to detect changes according to the "Last Modified" date, this implies that at the beginning, the task might overlook some experiments that are running big jobs and do not update their pkl too often; however, it will catch up in sufficiently short time. On the other hand, if an experiment is identified as NOT RUNNING anymore, the task will perform an exhaustive check and look for the "Last Modified" date of its "run.log" file. Thus, asserting that the experiment is truly not running. The 5 minutes interval might leave some room for misreporting, but I guess that is acceptable.

I will proceed to implement the "Currently Running Experiments" button on the main page of Autosubmit GUI, which will show the experiments that are registered in the previously mentioned table. Furthermore, the "View" that provides the list of experiments after a "Search" will be updated to also include the current detected status of the experiment.

On the next step, a similar approach will be used to gather the total running time for COMPLETED jobs of all the experiments and save that data in a database for easy querying.

Also mentioning @pechevar.

In GitLab by @wuruchi on Jan 10, 2020, 13:56

Hello @mcastril @dbeltran

There are 2 new tables in "/esarchive/autosubmit/ecearth.db".

experiment_times. COLUMNS:
1. exp_id: Id of exp, from the experiment table.
2. name: expid.
3. created: Timestamp of the date this register was created.
4. modified: Timestamp of the date this register was last modified.
5. total_jobs: The number of jobs the experiment has.
6. completed_jobs: The number of current completed jobs.
job_times. COLUMNS:
1. exp_id: Id of exp, from the experiment table.
2. exp_name: expid.
3. job_name: The unique name of a job.
4. created: Timestamp of the date this register was created.
5. modified: Timestamp of the date this register was last modified.
6. minutes: The number of minutes the experiment spent running until it was completed.
7. status_code: Status code of the experiment.

These tables are currently being filled incrementally with the information of all the experiments in "/esarchive/autosubmit/" by a task running on my computer. This is a first massive insert. After this first task is completed, a "cron" job will be in charge of updating these tables periodically with current information.

Needless to say, this will allow access to useful information in an easy and efficient way.

In GitLab by @mcastril on Jan 10, 2020, 17:52

Hi @wuruchi . Very nice to see that we can start to collect this important information that for sure we will exploit, besides enhancing the GUI experience and workload.

I think that looking first at the pkl and then at the log is a good idea.

On the other hand I think that for Autosubmit4, we should make the tool upload the tables itself, instead of running a cron job. We can talk about it in our next meeting. Of course we will need the cron anyway for the time we still are using Autosubmit 3, but I find it more efficient that the workflow manager does it itself.

Another point is the growth of the database. In the case of "experiment_status" and "experiment_times" it is not problematic as I see it will be one row per experiment. In the other case it is one row per job, and having experiments with thousands of jobs we have to look more about the complexity.

In GitLab by @wuruchi on Jan 13, 2020, 15:50

Hello @mcastril

As you say, saving one row per job is quite expensive, but it also simplifies querying. We will look into this exchange of complexity and see what works best.

There is a somewhat important and useful update on Autosubmit GUI. It now has the button "RUNNING" in the home page that shows all the experiments that are currently RUNNING:

http://bscesweb04.bsc.es/autosubmitapp/

In GitLab by @pechevar on Jan 15, 2020, 11:07

Hi,

Suggestions from yesterday meeting:

@yruprich @vsicardi @etourign @jberlin @fmacchia @gmontane

Give the start date and the date (start date + crunch * chunk size)
Move the name of the job in the center of the circle (is not clear if the name is from the top or the bottom node)
To have the description next to the expid will be nice, e.g.: t01q weekly test case: T255L91-ORCA1L75 cold start
The buttons in the classic graph should go to the first job (now is going to the last one)
To have in the right box:
- The list of waiting jobs will be useful
- The chunk
- The member
See the model and PKL is not useful
Branch and HPC name could be bigger
The Log should be shown the last line, but the user can be scroll
In the tree view, since is the only one, should be generated automatically
What is failed_run, failed_jobs and failed_queues?
About the time, Is useful have the average of queue time / failed jobs
Average of time to run any job SIM, CMOR, etc

I think is all, please feel free to correct or change whatever you want

In GitLab by @mcastril on Jan 15, 2020, 11:10

Thank you Pablo!

I will study all the suggestions and will create a new merged checkbox (with the one above that had a couple of tasks pending). We will discuss it after Wilmer's comeback at the first weekly Autosumit meeting that we had.

In GitLab by @jberlin on Jan 15, 2020, 11:17

Hi here some other remarks:

add authentication mechanism
remove things that are not useful ( this creates confusion to the users)
add contextual actions in the nodes ( right click on nodes ):
- allow see the log of the job
- allow change status of the job
- allow re-run the job or change the platform where the user wants it to run
for big experiments, is not very useful to draw the whole thing , instead make a partial zoom of the root nodes or the last failed job.

I think that's what I had in mind after the demo we had, looking good @wuruchi !

In GitLab by @mcastril on Jan 15, 2020, 11:21

Thank you @jberlin . The authentication (and interactive actions) are planned for the second stage, but it is very useful to see what the user's think in advance.

Thanks for the tips, we will take them also into account.

In GitLab by @pechevar on Jan 15, 2020, 12:18

I didn't mention it: ASOME JOB GUYS!

I feel this will made a big improvement in the AS usability.

In GitLab by @wuruchi on Jan 15, 2020, 12:39

Thank you Pablo.

We will definitely look into these suggestions!

BSC-ES / autosubmit-gui

GUI for AutoSubmit v1 #30