Open ocaisa opened 3 years ago
It is a basic configuration but works for backends not connected to the internet. A more complex configuration is documented at https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#example
Note that populating the cache is a bit flaky so you may need to run it a few times before things are fully populated.
Also note that we can easily restrict what is cached using a dirtab file, see https://cvmfs.readthedocs.io/en/stable/cpt-hpc.html#preloading-the-cernvm-fs-cache so we can restrict ourselves to specific releases/arch (or even more fine-grained control)
Public keys should be served via HTTPS (for production use anyway).
FYI This script allows running MPI applications on workers that are not connected to the internet:
[ocais1@juwels01 ~]$ SLURM_MPI_TYPE=pspmix OMP_NUM_THREADS=2 srun --time=00:05:00 --nodes=1 --ntasks-per-node=24 --cpus-per-task=2 singularity exec --fusemount "$EESSI_CONFIG" --fusemount "$EESSI_PILOT" ~/client-pilot_centos7-2020.08.sif /cvmfs/pilot.eessi-hpc.org/2020.12/software/x86_64/intel/haswell/software/GROMACS/2020.1-foss-2020a-Python-3.8.2/bin/gmx_mpi mdrun -s ion_channel.tpr -maxh 0.50 -resethway -noconfout -nsteps 10 -g logfile
Script updated to allow us to query what arch directory we should be pre-populating our alien cache with (rather than pulling in every arch)
Just to also log this somewhere: your script was aimed at using an alien cache on a system where the batch nodes don't have internet access. Another use case for the alien cache is if you want to use the EESSI
container to run multiprocessing.
E.g. simply bind mounting a local dir and then starting a parallel run
...
export SINGULARITY_BIND="/tmp/$USER/var-run-cvmfs:/var/run/cvmfs,/tmp/$USER/var-lib-cvmfs:/var/lib/cvmfs
srun singularity exec --fusemount "$EESSI_CONFIG" --fusemount "$EESSI_PILOT" docker://eessi/client-pilot:centos7-$(uname -m)-2020.12 <some_mpi_program>
Typically fails with the error Failed to initialize loader socket
(see e.g. https://github.com/EESSI/docs/pull/40), because multiple instances of the container are launched on an individual node, that all try to use the same /tmp/$USER/var-run-cvmfs
.
I've adapted the above script somewhat to remove the preloading, make the lower cache writable, and set a the proxy to DIRECT
:
# shared_alien_cache.sh
# Set group (as required, useful if you would like to share the cache with others)
MYGROUP=$GROUPS
# Set user
MYUSER=$USER
# Set path to shared space
SHAREDSPACE="/path/to/shared/space"
# Set path to (node) local space to store a local alien cache (e.g., /tmp or /dev/shm)
# WARNING: This directory needs to exist on the nodes where you will mount or you will
# get a binding error from Singularity!
LOCALSPACE="/tmp"
# Chose the Singularity image to use
STACK="2020.12"
SINGULARITY_REMOTE="client-pilot:centos7-$(uname -m)"
#########################################################################
# Variables below this point can be changed (but they don't need to be) #
#########################################################################
SINGULARITY_IMAGE="$SHAREDSPACE/$MYGROUP/$MYUSER/${SINGULARITY_REMOTE/:/_}.sif"
# Set text colours for info on commands being run
YELLOW='\033[0;33m'
NC='\033[0m' # No Color
# Make the directory structures
SINGULARITY_CVMFS_ALIEN="$SHAREDSPACE/$MYGROUP/alien_$STACK"
mkdir -p $SINGULARITY_CVMFS_ALIEN
SINGULARITY_HOMEDIR="$SHAREDSPACE/$MYGROUP/$MYUSER/home"
mkdir -p $SINGULARITY_HOMEDIR
##################################################
# No more variable definitions beyond this point #
##################################################
if [ ! -f $SINGULARITY_HOMEDIR/default.local ]; then
echo -e "${YELLOW}\nCreating CVMFS configuration for shared and local alien caches\n${NC}"
echo "# Custom settings" > $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_WORKSPACE=/var/lib/cvmfs" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_PRIMARY=hpc" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_hpc_TYPE=tiered" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_hpc_UPPER=local" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_hpc_LOWER=alien" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_LOWER_READONLY=no" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_local_TYPE=posix" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_local_SHARED=no" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_local_QUOTA_LIMIT=-1" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_local_ALIEN=\"/local_alien\"" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_alien_TYPE=posix" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_alien_SHARED=no" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_alien_QUOTA_LIMIT=-1" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_CACHE_alien_ALIEN=\"/shared_alien\"" >> $SINGULARITY_HOMEDIR/default.local
echo "CVMFS_HTTP_PROXY=\"DIRECT\"" >> $SINGULARITY_HOMEDIR/default.local
fi
# Environment variables
export EESSI_CONFIG="container:cvmfs2 cvmfs-config.eessi-hpc.org /cvmfs/cvmfs-config.eessi-hpc.org"
export EESSI_PILOT="container:cvmfs2 pilot.eessi-hpc.org /cvmfs/pilot.eessi-hpc.org"
export SINGULARITY_HOME="$SINGULARITY_HOMEDIR:/home/$MYUSER"
export SINGULARITY_SCRATCH="/var/lib/cvmfs,/var/run/cvmfs"
# export SINGULARITY_BIND="$SINGULARITY_CVMFS_ALIEN:/shared_alien,$LOCALSPACE:/local_alien"
export SINGULARITY_BIND="$SINGULARITY_CVMFS_ALIEN:/shared_alien,$LOCALSPACE:/local_alien,$SINGULARITY_HOMEDIR/default.local:/etc/cvmfs/default.local"
# Get a shell
echo -e "${YELLOW}\nTo get a shell inside a singularity container (for example), use:\n${NC}"
echo -e " export EESSI_CONFIG=\"$EESSI_CONFIG\""
echo -e " export EESSI_PILOT=\"$EESSI_PILOT\""
echo -e " export SINGULARITY_HOME=\"$SINGULARITY_HOME\""
echo -e " export SINGULARITY_BIND=\"$SINGULARITY_BIND\""
echo -e " export SINGULARITY_SCRATCH=\"/var/lib/cvmfs,/var/run/cvmfs\""
echo -e " singularity shell --fusemount \"\$EESSI_CONFIG\" --fusemount \"\$EESSI_PILOT\" $SINGULARITY_IMAGE"
I then create a script for the commands I want to run in the container:
#!/bin/bash
# run_gromacs.sh
source /cvmfs/pilot.eessi-hpc.org/2020.12/init/bash
module load GROMACS
gmx_mpi mdrun -s ion_channel.tpr -maxh 0.50 -resethway -noconfout -nsteps 10000 -g logfile
And finally create a job script:
#!/bin/bash
# gromacs_job.sh
#SBATCH ...
source shared_alien_cache.sh
# Copy gromacs runscript plus input to the directory that singularity exec will start in:
cp run_gromacs.sh $SINGULARITY_HOMEDIR/
cp ion_channel.tpr $SINGULARITY_HOMEDIR/
srun --mpi=pmix singularity exec --fusemount "$EESSI_CONFIG" --fusemount "$EESSI_PILOT" docker://eessi/client-pilot:centos7-$(uname -m)-$STACK ./run_gromacs.sh
This works fine for me :)
Note that for 'laptop users' we could make it even simpler: we don't need the tiered caches, and probably just configuring a single, alien cache is sufficient to make it shareable between multiple MPI processes.
@casparvl Yeah, there are a few different configurations that should be documented here. I wonder is it worth creating something that will generate the right setup for you based on a few settings? Is it complicated enough to consider moving from bash to python?
I wonder is it worth creating something that will generate the right setup for you based on a few settings?
Yeah, this would be an option. Just as an idea, I think something with arguments like
--local_tmpdir <some_local_dir>
--shared_tmpdir <some_shared_dir> [optional]
--dirtab [optional]
would be sufficient to cover most scenarios I think.
--local_tmpdir
, which would configure a single level alien cache (needs to be an alien cache to be shareable between MPI processes).--local_tmpdir
and --shared_tmpdir
, and would get a two-tiered alien cache, where the lower level is writable.
This is a (basic) script to create and populate a CVMFS alien cache for use on systems that do not have access to the internet (but you do require internet access to perform the initial run of the script).