Create docker container (all users) and a helm chart (EDITO)

[LuiggiTenorioK]

In GitLab by @kinow on Dec 1, 2023, 10:11

Now that we have a Docker image for Autosubmit, it is time to create one for Autosubmit API too. That image can be used by any site using AS (e.g. ClimateDT, Kit, EDITO, BSC-ES, etc.).

We will also need to create a k8s helm for EDITO (not sure if that will be generic enough to be shared with others).

[LuiggiTenorioK]

In GitLab by @kinow on Dec 1, 2023, 10:12

marked this issue as related to autosubmitreact#94

[LuiggiTenorioK]

Agree! This will be really useful. But, I have my doubts about what is the best way to link the API container with the persistence layer it will need (DDBBs, config files, autosubmitrc).

I'm not sure if this docker container should pull the autosubmit one and use its docker area filesystem, or interact directly with the user filesystem (by the docker mount volume option). :thinking:

[LuiggiTenorioK]

created branch create-docker to address this issue

[LuiggiTenorioK]

mentioned in commit d4b3c857dc36b7751e3bcce9cdd37e0d4f83e0d3

[LuiggiTenorioK]

mentioned in commit b5543e7873de6efb3f3057a318a8588d9c6ed7e0

[LuiggiTenorioK]

mentioned in commit fb90c73a1fc180bcad7396a3eb5c24b942402039

[LuiggiTenorioK]

I have already included a docker image in the repo :raised_hands:

Still, I'll need more information about the EDITO environment to understand how the helm chart will interact with the DDBBs and file system of autosubmit inside that environment. Also, it's relevant to figure out how it should be exposed to other users of EDITO or other instances of the GUI.

[LuiggiTenorioK]

mentioned in commit 894ffda3221ad865dd1010a67d7bc36f2253a786

[LuiggiTenorioK]

mentioned in commit 311e143ad47c677eef36b1b09a518fb27c78ce1c

[LuiggiTenorioK]

mentioned in commit fdb422fd2ad779058a836b8508bb365a2351c7b1

[LuiggiTenorioK]

Our whiteboard from today's meeting:

@kinow @edgano

[LuiggiTenorioK]

In GitLab by @kinow on Jul 16, 2024, 13:13

I think now I have enough to continue working during your vacation, and even update the API if necessary :slight_smile:

Summarizing what was discussed:

• 1 Database
  • This is nearly done, just pending the history module on autosubmit (Bruno)
• 2 Pickle file
  • We will have to handle this differently, possibly with PostgreSQL serializing it (Bruno/Luiggi, new code & tests needed)
• 3 Lock file
  • When PostgreSQL is used that can be an alternative to portalocker files (Bruno/Luiggi, new code & tests needed)
• 4 Logs
  • This is the biggest hurdle, and we think we might be able to have daemon processes on the pods with AS. These processes would sync the logs to S3 periodically.
  • The API would then need to also be modified to read logs from S3 when configured to do so.
  • The It depends on whether we can share a bucket on EDITO (I've asked them about it) or not.

For September, considering we also need to modify the Kubernetes deployment, this is a quite tight schedule... one possible scenario is we implement steps 1, 2, 3, leaving 4 logs out. But that depends on whether the GUI and API would be usable after that.

Thank you @LuiggiTenorioK !

[LuiggiTenorioK]

mentioned in issue autosubmit#1391

[LuiggiTenorioK]

mentioned in issue autosubmit#1392