cambiotraining / hpc-intro

Practical course on running jobs on a HPC
https://cambiotraining.github.io/hpc-intro/
Other
14 stars 13 forks source link

Not enough space for Conda environments #20

Closed tavareshugo closed 1 year ago

tavareshugo commented 2 years ago

In the course iteration 2022-07-14, when we all created the scipy environment (here), we started getting errors because /home ran out of space.

Some solutions:

Our quick-and-dirty solution was:

mv ~/miniconda3/ /scratch/$USER/
ln -s /scratch/$USER/miniconda3/ ~/miniconda3
tavareshugo commented 2 years ago

We will increase the space to avoid this problem again.

Another thing that should do is cache the packages needed for numpy and matplotlib to make installation quicker.

This should work:

conda install -y -c conda-forge numpy=1.23.1 matplotlib=3.5.2
conda remove -y --force numpy matplotlib

This should remove those two packages, without removing the dependencies. So the installation process later on is quicker.

Also, update the materials to install those specific versions of the packages, otherwise newer versions (with different dependencies) might end up being downloaded.

tavareshugo commented 2 years ago

Add TODOs to this comment:

tavareshugo commented 1 year ago

conda environment now lives on /scratch