Sometimes singularity uses /home as cache dir

[kelly-sovacool]

Occasionally, singularity will inexplicably use my home dir as the cache dir:

    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: while creating SIF: while unloading container: close /home/sovacoolkl/.singularity/cache/oci-tmp/tmp_1615581310: disk quota exceeded

while running from nextflow:

nextflow run main.nf -profile ci_stub,slurmint -stub

even though it's set to the shared location in the config file:

nextflow config main.nf -profile ci_stub,slurmint

singularity {
    enabled = true
    autoMounts = true
    cacheDir = "/data/CCBR_Pipeliner/SIFS"
    envWhitelist='https_proxy,http_proxy,ftp_proxy,DISPLAY,SLURM_JOBID,SINGULARITY_BINDPATH'
}

Confusingly, some of the time it does correctly use /data/CCBR_Pipeliner/SIFS as the cache dir.

[kelly-sovacool]

This might be related to biowulf. Perhaps this problem is why the hpc admins set a seemingly redundant env variable in their nextflow config?

https://github.com/nf-core/configs/blob/361358571fe90f2f133b96add128339a926a6374/conf/biowulf.config#L32-L33

env {
    SINGULARITY_CACHEDIR="/data/$USER/singularity"
}

[kelly-sovacool]

I think setting env.SINGULARITY_CACHEDIR in the config as above fixed it on biowulf

[kelly-sovacool]

This still happens sometimes

FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: initializing source oci:/home/sovacoolkl/.singularity/cache/blob:sha256.5d5fe68932653b173e58f452733144c9485dc62f5324efc4eccebd4f4ace1384: writing blob: happened during read: unexpected EOF

even though I'm using the biowulf profile which has:

singularity {
    enabled = true
    autoMounts = true
    cacheDir = "/data/CCBR_Pipeliner/SIFS"
    envWhitelist='https_proxy,http_proxy,ftp_proxy,DISPLAY,SLURM_JOBID,SINGULARITY_BINDPATH'
}

env {
    SINGULARITY_CACHEDIR = "/data/CCBR_Pipeliner/SIFS"
}

[kelly-sovacool]

possibly related error...

    INFO:    Creating SIF file...
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: while creating squashfs: create command failed: signal: killed: