Problem while pulling the singularity image on a cluster

[Enorya]

Dear,

I'm trying to use your pipeline on a cluster (slurm) but each time I try to launch it, it terminates really quickly because there is a problem with the pull of the container (I'm working with singularity giving that I'm on a cluster). Each time I end up with the following error message:

Pulling Singularity image docker://maestsi/metontiime:latest [cache /scratch/leuven/340/vsc34088/maestsi-metontiime-latest.img]
Error executing process > 'concatenateFastq'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name maestsi-metontiime-latest.img.pulling.1696410374382 docker://maestsi/metontiime:latest > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:43d4fc75415a6b2c35bace7dae5b9bbcb41d8c972cabb5fff50b0110500a4356
    Copying blob sha256:94d07c8690a4ebb0c3efd27c229864e07d2047a60fad23472f4a5a9bc9cb4b9e
    Copying blob sha256:aac8deeeec9f25370156d9d1a1fcab24f3b2e212c98a02c564d45f194ef98f59
    Copying blob sha256:b11f2fa60ecd11aec202de8a71a3483e8b1a79c5082343802bb2935899e9e621
    Copying blob sha256:26c5c85e47da3022f1bdb9a112103646c5c29517d757e95426f16e4bd9533405
    Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
    Copying blob sha256:ad2d4ff3088a2c724663c7b356b48b8d0271776658685ca2727d73bc6797cb0f
    Copying blob sha256:dc05e78ef388bf65a5f3e30b8131665c7480c833e7c30947ea433264f5c8ca4e
    Copying blob sha256:e88e1201cafe26ff9e2bb4cb241ac25948b0ab3d33123b4fe2e7ab00736254c5
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: while fetching image: initializing sour
ce oci:/user/leuven/340/vsc34088/.apptainer/cache/blob:17be4dcd10b7da7924be96e47695652dca32cc28fb39ea7ba83aff814360275f: writing blob: write /user/leuven/340/vsc34088/.apptainer
/cache/blob/oci-put-blob3745024075: input/output error

Do you have any idea how to fix this? I think it's linked to a space limit in the cachedir directory so I already tried changing the singularity cachedir to a location where there is more space

singularity.cacheDir = "/scratch/leuven/340/vsc34088" # in the config file

and I changed the container option

containerOptions = '--bind /scratch/leuven/340/vsc34088/:/home'

but for a weird reason it doesn't work, it always go back to my user directory where the space is really limited. Do you know how I can change the path of the cachedir?

Thank you in advance for your help.

[MaestSi]

Hi @Enorya , my advice would be to pull the docker image from dockerHub on a computer where you have Docker installed, copy it on the cluster where you would like to run the pipeline with Singularity and convert it to a Singularity image. Then, in the config file, you can specify the path of the image on your cluster. You could try something like this:

//pull image from dockerHub
docker pull maestsi/metontiime:latest
//export image in img format
docker save -o metontiime.img maestsi/metontiime:latest
//copy metontiime.img file on the cluster
//scp metontiime.img user@cluster:/path/to/singularityCacheDir
//convert docker image to singularity image
singularity build metontiime.sif docker-archive://metontiime.img
//edit metontiime2.conf file
//container = 'maestsi/metontiime:latest' -> container = "/path/to/metontiime.sif"

Changing the singularity cache dir should also work, but there are also environment variables which may have different priority compared to singularity.cacheDir. I remember I had this issue too, and I had to do some troubleshooting to fix it. Moreover, I would advise not to change the path from the guest to the host system, i. e. use something like: containerOptions = '--bind /scratch/leuven/:/scratch/leuven' I hope the docker -> singularity solution works, let me know if you succeed! SM

[Enorya]

Hi @MaestSi ,

The solution you proposed solved my issue, thanks for that! However, I have another problem now. The path I gave for the working directory is not recognized. I gave the full path but all the commands that use this path are not working (like cp or find). I always end up with the following error message while running the nextflow pipeline:

Error executing process > 'concatenateFastq'

Caused by:
  Process `concatenateFastq` terminated with an error exit status (1)

Command executed:

  # mkdir -p /lustre1/project/stg_00026/enora/EMBRC-MinION/MetONTIIME/16S_combined_8c_100M_new-pip
  # mkdir -p /lustre1/project/stg_00026/enora/EMBRC-MinION/MetONTIIME/16S_combined_8c_100M_new-pip/concatenateFastq
  cp /lustre1/project/stg_00026/enora/EMBRC-MinION/raw_data/16S-COI_fastq/*fastq.gz /lustre1/project/stg_00026/enora/EMBRC-MinION/MetONTIIME/16S_combined_8c_100M_new-pip/concatenateFastq

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  cp: cannot stat '/lustre1/project/stg_00026/enora/EMBRC-MinION/raw_data/16S-COI_fastq/*fastq.gz': No such file or directory

I checked by trying the commands myself in the terminal and it works when I use this path.

Do you have any idea why the path is not recognized? Am I doing something wrong?

[MaestSi]

Hi @Enorya , I think this is due to the fact that you forgot to mount the /lustre1 disk. To do that, at line 74 of metontiime2.conf file, please edit:

containerOptions = '--bind /home/:/home --bind /lustre1:/lustre1'

In this way, also /lustre1 disk should be accessible to Singularity. Best, SM

[Enorya]

Hi @MaestSi ,

It worked, thanks for your help!

Best,

[MaestSi]

Perfect! Best, SM