Autosubmit GUI new release preparation #44

Closed kinow closed 2 months ago

In GitLab by @jberlin on Oct 13, 2022, 13:02

Working path for the deployment of the new release of the Autosubmit GUI:

Pre-Staging

[x] Update documentation for GUI and API in the wiki or readthedocs.io (if needed).
[x] Update the news file located in src/components/pages/News.js with the latest changes done/ fixed bugs

Deployment VM for Validation

[x] Generate the API new version and upload to Pypi.
[x] Update GUI and deploy API in development environment for the final validation
[x] Final validation of the release, Close all issues in the milestone, both in the GUI and API projects.

Deployment in Production

[x] if validation in VM was OK, deploy in production, before that send email to distributions lists to warn the users of the update/possible disruptions
[x] Validation in production, checking of the solved the issues & new features.

Announce

[x] Prepare a message to send to the BSC-ES mailing list, with a brief summary of the new features (and also reminding abut the best advantages of using the GUI)

@mcastril @cgutierr @mgimenez

In GitLab by @cgutierr on Oct 17, 2022, 08:37

Regarding the new changelog that will go in the news section, I've drafted this from different changes.

There are things that should be added such as @jberlin new wrapper ordering, the favicon, a mention to the new dev env, etc...

Feel free to modify and delete any of the parts that you feel like don't belong here, or make any suggestions!

xx-10-2022

This new release has been dedicated to improving some capabilities towards a more performant and better experience for our users. The main goal is to make the website more reactive and dynamic.

In the Dashboard, we increased the retrieval speed for the experiments' Detailed Data. Now only experiments on the current page will be refreshed.

A new endpoint for stopping current background API jobs has been implemented (on the API side). Alongside this, we have a clean-up hook to use in all the required situations.

The new endpoint allows discarding expensive, non-relevant petitions and dispatching new user demands on time. Thus, proving a better user experience.

Graph, Tree and QuickView start loading automatically, and the transition between them is improved due to the mentioned task canceling mechanism.

An early initial load of the Active experiments at the homepage has been implemented without requiring the user to click on the button each time.

Tasks without dependencies are displayed correctly in the Graph View in a single row.

Added redirection support after logging in.

Fixed a bug in the tree generation that was making the load more tedious.

Fixed a visual bug regarding the close button in the pop-up windows.

Mouse square selection for multiple graph nodes on Right Click. (Pending)

In GitLab by @cgutierr on Oct 17, 2022, 09:12

Also, is this repo Autosubmit GUI private?

We should link the user here for issues rather than autosubmit repo.

In GitLab by @mgimenez on Oct 17, 2022, 09:18

Yes, this repo is private. That is why I had to create another repo just with the documentation at add-documentation branch. Maybe we can use that other repo as issue tracker while we can't make this repo public.

In GitLab by @jberlin on Oct 17, 2022, 09:22

Hi ! which is the repo that is private @cgutierr ? am I wrong or it seems the link is missed ?

In GitLab by @jberlin on Oct 17, 2022, 09:25

thanks @cgutierr for preparing the news part. I think for this week we can start with the preparations for the new release. from my side I need to re-check the label thing that we decided to add in addition to the fix of the ordering in the treeview

In GitLab by @mcastril on Oct 17, 2022, 13:23

Sorry I am lost. It's private but open to any Earth Sciences user. As the GUI is only used internally, what else would we need?

In GitLab by @cgutierr on Oct 17, 2022, 13:31

In the About page, at the last section at the bottom, the link named "Autosubmit GUI Issue page" redirects to the Autosubmit issues page. Shouldn't it redirect to Autosubmit GUI issues page?

In GitLab by @cgutierr on Oct 17, 2022, 13:32

Yes, this is the only solution if this repo stays in private mode.

In GitLab by @mcastril on Oct 17, 2022, 13:32

Thanks @cgutierr. I did minor modifications on the text

In GitLab by @mcastril on Oct 17, 2022, 13:36

Yes, please change it.

This is legacy. Before, all issues were reported in Autosubmit.

In GitLab by @mcastril on Oct 18, 2022, 17:57

Can you start with the Pre-staging @jberlin ? Nothing prevents us from moving it right?

In GitLab by @jberlin on Oct 18, 2022, 18:16

Hi @mcastril, I need to validate my latest changes for the treview and then I think we are OK for the pre-staging. I copied the a550 for this.we will need to clean up the development environment and install the release over there to test the involved issues without making any change.

In GitLab by @mcastril on Oct 19, 2022, 10:13

I need to validate my latest changes for the review

True, the labels in startdate level were missing. Please report the progress in the corresponding issue so that we can comment it tomorrow.

In GitLab by @jberlin on Oct 24, 2022, 18:14

mentioned in commit ad221660bec80c6f0dc76b8260f7fae3dae51375

In GitLab by @jberlin on Oct 24, 2022, 18:14

marked the checklist item Update the news file located in src/components/pages/News.js with the latest changes done/ fixed bugs as completed

In GitLab by @jberlin on Oct 24, 2022, 18:15

I updated the News.JS tomorrow I will check the UI readthedoc.io to see if we are missing something

In GitLab by @jberlin on Oct 25, 2022, 10:39

Hi @mgimenez , I was checking the readthedocs.io corresponding to the GUI, is it possible that some imgs are missed or the links broken ?

https://autosubmit-gui.readthedocs.io/en/latest/experiment/index.html

I think we can add in the section of the tree, the improvements done concerning the UI/UX such as the ordering of the wrappers, the new folders and functionalities to expand/collapse all items.

In GitLab by @jberlin on Oct 25, 2022, 12:22

marked the checklist item Update documentation for GUI and API in the wiki or readthedocs.io (if needed). as completed

In GitLab by @mgimenez on Oct 26, 2022, 08:40

Yeah, images were not uploaded to the repository because that is bad practice. I was supposed to find another way of making readthedocs fetch the images without having to upload them to the repo. Any ideas, @bdepaula?

In GitLab by @bdepaula on Oct 26, 2022, 09:29

If the images are static, I think they can go into the Sphinx html_static_path location.

But if the images are created dynamically, and depend on the project code, then +1 to not including the images directly. We can instead use Python to generate the images during the build (in readthedocs, their build system would generate the images for us).

There are a few different ways of achieving that, depending on the image type (png/jpg generated by the tool, a dot/graphviz graph, etc.). If this is blocking a release, maybe we can add the images in static for now and open a follow-up issue to generate the images dynamically.

Cheers

In GitLab by @mgimenez on Oct 26, 2022, 09:34

So we upload the images onto another service, like i dunno, drive, flicker, etc and put the folder path with images directly to the html_static_path? This could be very much useful for Autosubmit's Docs aswell.

In GitLab by @mcastril on Oct 26, 2022, 09:49

If you use static images you could explore uploading them to GitLab (by creating a wiki and upload them, they are stored in the server I think)

If not, you can always add them into the Earth wiki:

https://earth.bsc.es/wiki/doku.php?id=tools:autosubmit

Media Manager...

In GitLab by @bdepaula on Oct 26, 2022, 09:51

Assuming the images are not really big (>4MB or so I guess), we can upload them directly in the git repository, in the html_static_path or in any other location of the source directory.

The html_static_path is normally _static in Sphinx projects, e.g. in CWL:https://github.com/common-workflow-language/user_guide/blob/0b55dc767ecd8b288d3ffa3e6f24afbf59864f5f/src/conf.py#L158 and in Cylc: https://github.com/cylc/cylc-doc/blob/9304c0837d75bfcfcd6e0b09bc0a52f12c9f0920/src/conf.py#L183

In the CWL User Guide the logo is rendered from a subfolder of html_static_path: https://github.com/common-workflow-language/user_guide/blame/0b55dc767ecd8b288d3ffa3e6f24afbf59864f5f/src/introduction/basic-concepts.md#L15

But images can reside elsewhere too. In the case of Cylc, for instance, the images are stored in subdirectories of the sections. Their tutorial folder has an img folder for images: https://github.com/cylc/cylc-doc/tree/master/src/tutorial/img

Then a directive is accessing that location to load images, e.g. https://github.com/cylc/cylc-doc/blob/master/src/tutorial/scheduling/integer-cycling.rst#recurrence-sections

That's one way of doing it... we can store images anywhere with the source code. Some images might go to static (e.g. logo, favicons), while others may be stored near the sections... or they could be stored elsewhere too if we prefer I think.

IMHO, the disadvantage of using an external service is that users wouldn't be able to build and run the documentation offline (e.g. if someone cloned the project in some area of the HPC, or they are in a flight working offline, etc.).

In GitLab by @bdepaula on Oct 26, 2022, 09:55

I mentioned dynamic images… here's an example, also from the CWL User Guide: https://github.com/common-workflow-language/user_guide/blame/0b55dc767ecd8b288d3ffa3e6f24afbf59864f5f/src/introduction/basic-concepts.md#L51-L103

This is used to generate an SVG graph during build time in ReadTheDocs that is added to the documentation:

In GitLab by @mgimenez on Oct 26, 2022, 11:14

Yeah, it worked fine if we uploaded to the repo all images (Autosubmit's repo is like this). But as far as I know (not much) it is not advised to upload to git images. But if both you and @mcastril are in favor of uploading images to the repo, I'll do it now and solve this issue.

In GitLab by @jberlin on Oct 26, 2022, 16:28

Hi @mgimenez , maybe you can fork the main branch and add the images there temporarily and use that new branch to compile and generate the documentation. then we we finally decide the best approach we take the right approach and delete this branch and use the master again

In GitLab by @mgimenez on Oct 27, 2022, 09:17

Done, @jberlin. Created new branch with-images at the docs repo and activated the branch at readthedocs. You can check it out.

In GitLab by @jberlin on Oct 27, 2022, 10:11

Thanks @mgimenez

In GitLab by @cgutierr on Nov 2, 2022, 07:58

Hi @jberlin ,

Regarding the deployment situation we could go the easy way (once I deploy the latest changes regarding Issue #47 ) and just push to master the current active versions of the API and the GUI that are on the Rocky env dev and delete the other branches such as the one I created to work while the API user log in was not available.

What do you think?

In GitLab by @jberlin on Nov 2, 2022, 10:05

Hi @cgutierr , is not bad idea but we will run into additional merges and some parts of the code may also be dirty (debugging alerts messages, commented/dead code, etc) and merges in console are usually error-prone .

Some of the changes need code-review before actually putting them to master so please after #47 is completed submit the branch for CR and assign it to me, thanks!. After we merge these changes I will do a clean checkout of the master branch and deploy both the API and the GUI in dev.

In GitLab by @cgutierr on Nov 3, 2022, 08:42

Sorry I thought I had to assign you as reviewer only and not also as assignee. It's done now.

In GitLab by @mgimenez on Nov 3, 2022, 16:32

Merged with-images branch to the master.

In GitLab by @mcastril on Nov 3, 2022, 18:59

Yes, I think it's better to be conservative and do the usual merges and reviews, although it's more work, it's safer.

After all the merges I agree that all the other non-merged and legacy branches should be removed if they are not aiming to be merged in master.

If you find the develop/commit/deploy process slow, we can talk about alternatives.

In GitLab by @mcastril on Nov 3, 2022, 19:02

Perfect, I see that latest contains the images. I guess that with-images has to be removed from RTD.

In GitLab by @mgimenez on Nov 5, 2022, 13:24

I thought that the branch would be deleted. I disabled the branch.

In GitLab by @cgutierr on Nov 9, 2022, 07:53

It's fine for me, I'll try to be more declarative and to make more independent commits (one for each functionality for example) so that the review process gets easier :smile:

In GitLab by @jberlin on Nov 9, 2022, 17:18

marked the checklist item Generate the API new version and upload to Pypi. as completed

In GitLab by @jberlin on Nov 9, 2022, 17:19

new version of the API generated and uploaded to pypi.org:

https://pypi.org/project/autosubmit-api/1.0.26/

In GitLab by @jberlin on Nov 9, 2022, 17:19

marked the checklist item Update GUI and deploy API in development environment for the final validation as completed

In GitLab by @jberlin on Nov 9, 2022, 17:25

I generated TAGs in both projects to mark the end of this release:

AS_GUI_SP1
AS_API_SP1

I deployed both TAGs in the development environment, until we close all the issues in the milestone, we should not change anything. we can validate the issues tomorrow or in the upcoming days

In GitLab by @jberlin on Nov 9, 2022, 18:17

I also updated deployment wiki for AS API, to make it more clear

In GitLab by @mcastril on Nov 9, 2022, 18:37

Yes, this is important. They will be squashed anyway at the time of merging them.

In GitLab by @mcastril on Nov 9, 2022, 18:55

Good idea to create those tags.

I think it would be useful to add the list of new features in the tag description. It will be very helpful in the future for many reasons: to justify your work; to do a recap in dissemination sessions, workshops etc; to have a track of the work done at the end of the year...

In GitLab by @jberlin on Nov 10, 2022, 13:23

mentioned in commit 5bd89f74b25dacb87d00f9518a0a8f90bac51459

In GitLab by @jberlin on Nov 10, 2022, 13:28

mentioned in commit 3bbd9f56ced78d5d0d9d889b97f0e16fffa103bd

In GitLab by @mcastril on Nov 16, 2022, 12:59

I reviewed the wiki and I think it lack's info about the development and production environments.

In the GUI wiki there's no mention about it.

In the API wiki there is some mention to rocky but then I am not sure if the paths refer to the development or production environment.

To add this information, I think the best is that you create a section in the API wiki explaining the production and development environment (hosts, users, directories, etc).

Then, in the deployment and update sections' for both wikis, explain the depoyment-testing-deployment process involving the two machines, referring to the new wiki section and ensuring that the paths are consistent. If the paths are already explained and abstracted in the new wiki section, they can be just referred with their abstract name or variable.

In GitLab by @jberlin on Nov 16, 2022, 13:33

Hi @mcastril , yes that is true, we can enrich the current documentation with more details of each environment, could be very useful for deploy

In GitLab by @mcastril on Nov 16, 2022, 13:38

The development VM is only explained in the CAS issue, I think, so it's difficult to access if you don't have bookmarked it.

In GitLab by @jberlin on Nov 16, 2022, 17:13

Hi @mcastril, I added in both wikis a new section called environments where there is a table with the most important things to know and some other useful information

In GitLab by @jberlin on Nov 18, 2022, 12:53

marked the checklist item Final validation of the release, Close all issues in the milestone, both in the GUI and API projects. as completed

BSC-ES / autosubmit-gui

Autosubmit GUI new release preparation #44