Packages git-cloned or in virtualenv instead of links

[AlexandreMary]

At the moment the python packages required by Davai (ial_build, ial_expertise, vortex, epygram, ecbundle + davai_taskutil) are links to pre-installed versions on given user accounts.

It could be better to git-clone them or to set up a virtualenv for each experiment ? (and in this case to publish these on PyPI).

Status of sub-packages on PyPI:

• [ ] IAL-build: ACCORD-NWP/IAL-build#1
• [ ] IAL-expertise: ACCORD-NWP/IAL-expertise#1
• [ ] Vortex: (-> Vortex 1.11)
• [x] epygram
• [ ] ecbundle: OK ?
• [ ] davai_taskutil
• [ ] DAVAI-env: ACCORD-NWP/DAVAI-env#11 (not mandatory for this current Issue but would be better)

(This issue is a duplicate of ACCORD-NWP/DAVAI-env#9)

[tlestang]

Summary of a discussion with @AlexandreMary during the Davai WW 2024.

Currently it is expected that the davai-run_tests command is ran with the current working directory being a "experiment" directory as created by the davai-new_xp command.

This directory looks like this:

conf/
davai_taskutil -> DAVAI-tests/src/davai_taskutil
DAVAI-tests/
ecbundle -> /home/acrd/public/ecbundle/2.0.0/ecbundle
epygram -> /home/acrd/public/EPyGrAM/1.4.x
ial_build -> /home/acrd/public/IAL-build/1.2.6/src/ial_build
ial_expertise -> /home/acrd/public/IAL-expertise/1.1.6/src/ial_expertise
logs -> /ec/res4/scratch/sos/davai/logs/dv-0002-atos_bologna@sos
tasks -> DAVAI-tests/src/tasks
vortex -> /home/acrd/public/vortex

The DAVAI-tests directory is a clone of the repository, performed on the fly by the davai-new_xp command.

One option is to work with one python virtual environment per experiment, instead of a directory containing a clone of DAVAI-tests and links to existing installations of the various dependencies. This virtual environment would be created by the davai-new_xp command.

davai-new_xp

$ ls
dv-0002-atos_bologna@sos
$ ls dv-0002-atos_bologna@sos
bin include lib lib64 pyvenv.cfg

If we assume the dependencies (epygram`, `ial_build`, `ial_expertise and mkjob) can be installed via pip, they would installed within this virtual environment.

Now, instead of cloning the DAVAI-tests repo, we could instead pip install the tasks` and `davai_taskutil packages it contains, potentially merging them into one. With this package installed, a job generated by mkjob would be able to import the task module from the tasks package, e.g.

import tasks.<module> as todo

Running a davai experiment is now a matter of using the davai-run_exp providing a experiment id, or possibly the path to a experiment virtualenv:

davai-run_exp dv-0002-atos_bologna@sos

This would run the mkjob entry point located in the virtualenv for the dv-0002-atos_bologna@sos experiment.

The above assumes that both mkjob and vortex are available as regular pip installable package, which is not yet the case.