search

LightForm-group / non-repo-issues

A repo to hold any project tasks that don't have their own repo, or aren't relate to code.
0 stars 0 forks source link

Maintain shared matflow version on CSF? #25

Open gcapes opened 1 week ago

gcapes commented 1 week ago

Yuchen recently asked on slack for this to be updated. I just wanted a discussion around what the best policy would be for which version people should use, and whether a central install is the best idea?

Adam has previously told me that you need to have matflow installed in the environment you're using for your task schema (if you want all the functionality). As such, you (sometimes) have to install it yourself anyway, even if you're running your workflow using a different installation (which has to be the same version number as in your task schema environment). It seems to me that if you have to install it anyway, there's not much advantage to having a central install. But maybe I've missed something?

I don't think it's changed recently. It depends on the script_data_in and script_data_out action fields. If they are direct, this means the script will interact with matflow directly, so matflow must be installed. I will be adding a new format for script_data_in/out soon which will become the default, and this won't require matflow to be installed.

With that quoted text in mind, maybe it will make more sense to have a central/shared install, but perhaps they should have version names instead of matflow-dev or similar?

aplowman commented 1 week ago

A shared installation/configuration is in general a good idea I think, and should be kept and ideally updated semi-regularly. Updating the shared installation has two steps:

  1. Adding the new (single-file) matflow executable (e.g. the matflow-v0.3.0a128-linux artifact from the GH release https://github.com/hpcflow/matflow-new/releases/tag/v0.3.0a128) to the software/matflow_exes directory in our shared RDS space on CSF3; and updating the matflow-dev symbolic link in this directory to point to the new version.
  2. Triggering a new run of the matflow-environments action that generates an updated matflow_full_env packed conda environment; then uploading the resulting artifact matflow_full_env-linux.tar.gz to the software/matflow_conda_envs directory in our CSF3 shared RDS space, and following the instructions here for unpacking the conda environment.

With the shared installation, the users just need to have the software/matflow_exes directory on their path. They can also use previous shared versions by referencing the versioned files (e.g. matflow-v0.3.0a110-linux) since they also exist in that directory.