Set the Apptainer cache variables to point to folders within the Glidein tree

[mambelli]

Is your feature request related to a problem? Please describe. Here the report from Jeff

Hi all,

While debugging issues at Purdue Anvil which we realize in the end are
hitting disk quota limits, we discovered largish files of what I'm
assuming are cached apptainer images, being written to $HOME/.apptainer

This is bad because there's only at 25GB quota on $HOME for Purdue, and
on occasion files from a single glidein blow up the .apptainer dir to
6GB and on occasion 11GB.

I'm assuming this is a default location for apptainer, but is there any
way to tell the glideins to put the .apptainer cache in the current
working directory, since the glidein generally changes dir to a scratch
area on the WN? This would at least prevent us from hitting the quota
limit from Purdue.

Also, I'm more concerned about the larger issue, that we could be
leaking large files outside of the glidein_NNN dirs and thus they may
not be cleaning up when the glidein terminates.

Thanks,
Jeff

Describe the solution you'd like Setting APPTAINER_CACHEDIR and APPTAINER_TMPDIR to a proper temp space within the glidein_NNN directory See https://apptainer.org/docs/user/main/build_env.html

Describe alternatives you've considered N/A

Info (please complete the following information):

• Priority: high
• Stakeholders: FactoryOps, OSG
• Components: glidein, apptainer

Additional context Add any other context or supporting files about the feature request here.

[mambelli]

PR 404 will set the default and fix this. As a workaround you can manually set APPTAINER_CACHEDIR and APPTAINER_TMPDIR in the Factory or Frontend (attrs). Note that if APPTAINER_TMPDIR, Apptainer is using $TEMP which is already in the Glidein tree ($glidein_XXX/tmp)

[mambelli]

To patch GWMS v3.10.6 you can replace /var/lib/gwms-factory/web-base/singularity_lib.sh with https://raw.githubusercontent.com/glideinWMS/glideinwms/master/creation/web_base/singularity_lib.sh and issue a factory upgrade