January 2021 Release

[agstephens]

We will document the process of creating the 2021 release here:

Server (internal): 192.168.3.20

This machine has only a private IP, so must be accessed via another machine in the ceda dev tenancy as ssh gateway (not posting their IPs publicly here, check in the cloud portal).

[agstephens]

Clone repos

cd ~root/
git clone git@github.com:cedadev/jaspy-manager.git
cd jaspy-manager/

export JASPY_BASE_DIR=/appx/jasmin

Add latest miniconda (at 3.7)

Edit ./etc/minicondas.json file to include the latest miniconda binary and checksum. (Then commit and push this change.)

Run the miniconda installer:

./bin/install-miniconda.sh py3.7

...which should pick up the latest python 3.7.x

• _What this does: downloads installer to $JASPY_BASE_DIR/jaspy/src/ and installs new miniconda at /appx/contrib/jaspy/miniconda_envs/jaspy3.7/m3-<version> (containing root env only)_

Clone and link: ceda-jaspy-envs

./bin/add-envs-repo.sh https://github.com/cedadev/ceda-jaspy-envs

• What this does: adds alongside the jaspy-manager directory a ceda-jaspy-envs directory which is a clone the above mentioned repo, and creates a symlink environments-ceda-jaspy-envs inside the jaspy-manager clone (current directory)

Create a recipe for a new env

Copy the an old initial environment recipe to a new one:

./bin/clone-initial-env.sh <env_spec> <new_name>

where <env_spec> can be any of: the name of the existing environment, or path to the environment directory, or path to the initial.yml file inside that directory

• If you not using the same miniconda version as the previous environment, then for <new_name> give the path rather than just the name (anything containing / will be treated as a path, even if a relative rather than absolute path). Example:

  ./bin/clone-initial-env.sh \
  environments-ceda-jaspy-envs/py3.7/m3-4.7.10/jaspy3.7-m3-4.7.10-r20191126 \
  environments-ceda-jaspy-envs/py3.7/m3-4.9.2/jaspy3.7-m3-4.9.2-r20210105

  • What this does: creating directories as required, creates an initial.yml file which is a clone of the existing one but with the environment name inside it changed, and puts this in the correct location (alongside the existing environment directory if just the name given, otherwise at the path specified)

• NOTE: check the new directory structure matches old format

Create environment recipe (which builds environment from initial.yml file)

./bin/create-env-recipe.sh <path_to_initial_yaml_file>

If it resolves, test that it worked

py_version=$(echo $ENV_NAME | cut -d\- -f1)
miniconda_version=$(echo $ENV_NAME | cut -d\- -f2-3)
export PATH=$JASPY_BASE_DIR/jaspy/miniconda_envs/${py_version}/${miniconda_version}/bin:$PATH

source activate $ENV_NAME
python -c 'import sys; print(sys.version)'

[agstephens]

Next step

Changing the environment:

1. Unpin packages
2. Update pinned packages (to new versions if/as required) - and comment the reason in the initial.yml file
3. Add new packages
4. Move pip installs to conda installs (if possible)
5. Resolving conflicts?

[agstephens]

Done