Closed wdconinc closed 1 month ago
Hi @wdconinc
thank you for your contribution. I'll have to double check if this works on all clusters and on both login nodes and compute nodes though: the way /localscratch
is configured differs a little, and sometimes the apptainer module is loaded on a login node but apptainer executed with those bind settings on the compute node.
That said, there's another alternative: /var/tmp
. This directory dynamically bind mounts $SLURM_TMPDIR
on all GP clusters (Niagara computes have no local disk), and is automatically mounted by apptainer already. So if your workflow can simply use /var/tmp
instead of $SLURM_TMPDIR
the extra bind mount would not be needed.
I think we've now found a way around this. We are submitting jobs to cedar through an OSG htcondor glideins that runs as a slurm job (or plural). The slurm environment is available to our jobs and was preferentially used over the htcondor environment, in particular what concerns SLURM_TMPDIR and _CONDOR_SCRATCH. We have now modified our jobs to use the htcondor tempdirs over the slurm tempdirs and that gets us unstuck.
We have been encountering issues with
/localscratch
(i.e.$SLURM_TMPDIR
) not getting mounted on startup of jobs that are landing oncedar
from OSG htcondor. Because htcondor's+SingularityImage
causes the job to immediately start a container, we don't have a way to injectAPPTAINER_BIND
modifications.This PR adds
/localscratch
to the paths that are injected inAPPTAINER_BIND
for bind mounting.