ucgmsim / slurm_gm_workflow

Porting the GM workflow to run on new NeSI HPC (Maintainer: Jonney)
MIT License
0 stars 2 forks source link

Update to Maui/Mahuika environment creation #501

Closed sungeunbae closed 1 year ago

sungeunbae commented 1 year ago

Noticed a few issues (or confusion) during Maui/Mahuika env creation and updated manual and scripts accordingly.

  1. We need a copy of workflow somewhere (eg. home) to create a new environment. This means, sometimes you need to git clone this repo just to utilize scripts in environments directory, which will create a new environment and git clone another copy of this repo there. Maybe nice to have a separate "create_env" repo, but it is what it is for now.

  2. When we create a new environment due to Python upgrade, things can be a little confusing We have to make sure that the correct module of python is loaded before attempting to create a new environment. I made it more explicit in the README

  3. When a new virtual env is created, it doesn't install a new copy of pip. (https://stackoverflow.com/questions/35079438/how-do-i-can-install-pip-inside-virtual-environment) As a result, when a script executes "pip", this will be coming from the system python. The system pip will try to install a package under system's site-packages (which you have no permission).

It may not be always the case, but on one occasion, I noticed a latest copy of pip installed under .local after pip install--upgrade pip. It seems it found system python's site-packages are not writable, and silently created a .local and installed pip there. When you use this pip to install other packages, I am afraid things can get pretty messy when we want all new packages to be installed under new virt_envs lib/python3.x/site-packages. Doing python -m pip install --upgrade pip instead of pip install --upgrade pip seems to fix this issue.