search

EBI-Metagenomics / emg-viral-pipeline

VIRify: detection of phages and eukaryotic viruses from metagenomic and metatranscriptomic assemblies
Apache License 2.0
119 stars 16 forks source link

Singularity on Slurm #68

Closed indrikwijaya closed 2 years ago

indrikwijaya commented 2 years ago

Hi All,

I'm quite new to Nextflow and encountered an issue which maybe related to Singularity. I tried to run the basic test for the pipeline

module load nextflow
module load singularity/latest

HOME_DIR="/home/users/astar/gis/wijayai"
OUTPUT_DIR="${HOME_DIR}/scratch/virify"

nextflow run EBI-Metagenomics/emg-viral-pipeline -r v0.4.0 \
--fasta ${HOME_DIR}/.nextflow/assets/EBI-Metagenomics/emg-viral-pipeline/nextflow/test/assembly.fasta \
--cores 4 \
--outout ${OUTPUT_DIR} \
--workdir ${OUTPUT_DIR}/work \
--databases ${OUTPUT_DIR}/virifyDB \
--cachedir ${HOME_DIR}/.singularity \
-profile slurm,singularity

and got this error

Error executing process > 'preprocess:rename (1)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name microbiomeinformatics-emg-viral-pipeline-python3-v1.img.pulling.1654852796106 docker://microbiomeinformatics/emg-viral-pipeline-python3:v1 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:8559a31e96f442f2c7b6da49d6c84705f98a39d8be10b3f5f14821d0ee8417df
    Copying blob sha256:62e60f3ef11eb77464958fc8fed447414fe652a21fc69aea87038a02e05e6000
    Copying blob sha256:93c8ae15378255764d0f4929a03e87b1a2497ca34b479bc3d66a53c6f99d1333
    Copying blob sha256:ea222f757df7f8b1f7f73b74a335497a015e3a254ce99cc27fe2c591132ad4ce
    Copying blob sha256:e97d3933bbbe5af2dcfa3af027e5fbf95562d4bd37ed7b85bef9f5c5e0dc2862
    Copying blob sha256:adafb644e4de15d419df2c1ebcbdec9666d4d1b0336b1910088a04743f6f387f
    Copying blob sha256:5c7900f7bd30eec8a4e096ce021092672609b00170d570524e96ba914311c1f0
    Copying config sha256:b9f1fd37db9a9e00804a77f504cbb80ff3c8ff3bbac09d0df07e40853b5d554a
    Writing manifest to image destination
    Storing signatures
    2022/06/10 17:20:40  info unpack layer: sha256:8559a31e96f442f2c7b6da49d6c84705f98a39d8be10b3f5f14821d0ee8417df
    2022/06/10 17:20:42  info unpack layer: sha256:62e60f3ef11eb77464958fc8fed447414fe652a21fc69aea87038a02e05e6000
    2022/06/10 17:20:43  info unpack layer: sha256:93c8ae15378255764d0f4929a03e87b1a2497ca34b479bc3d66a53c6f99d1333
    2022/06/10 17:20:43  info unpack layer: sha256:ea222f757df7f8b1f7f73b74a335497a015e3a254ce99cc27fe2c591132ad4ce
    2022/06/10 17:20:43  info unpack layer: sha256:e97d3933bbbe5af2dcfa3af027e5fbf95562d4bd37ed7b85bef9f5c5e0dc2862
    2022/06/10 17:20:44  info unpack layer: sha256:adafb644e4de15d419df2c1ebcbdec9666d4d1b0336b1910088a04743f6f387f
    2022/06/10 17:20:44  info unpack layer: sha256:5c7900f7bd30eec8a4e096ce021092672609b00170d570524e96ba914311c1f0
    INFO:    Creating SIF file...
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /home/users/astar/gis/wijayai/cache/oci-tmp/tmp_693074224: copy_file_range: resource temporarily unavailable

I googled and thought that this maybe because insufficient of space for the cache. And then, I changed the --cachedir to my scratch folder which has more space. I have also changed NXF_SINGULARITY_CACHEDIR and SINGULARITY_CACHEDIR to my scratch. But, I still got similar error. Do you know what might cause this error? Thank you :)

hoelzer commented 2 years ago

Hey @indrikwijaya, thanks for your interest in the pipeline and so sorry for the late reply.

As a first check: can you in general pull Singularity images to your machine? Can you please try smt like:

HOME_DIR="/home/users/astar/gis/wijayai"
singularity pull  --name ${HOME_DIR}/scratch/virify/microbiomeinformatics-emg-viral-pipeline-python3-v1.img docker://microbiomeinformatics/emg-viral-pipeline-python3:v1

Does this work and generate an image file in the defined --name path?

If that works, can you try to run the nextflow pipeline again but define --cores 1 --max_cores 1 - sometimes on HPC there are issues w/ the initial pull and conversion of Docker --> Singularity images when done in parallel. Once you have the images downloaded, you can just re-use them also w/ more cores to run the pipeline.

indrikwijaya commented 2 years ago

hi @hoelzer, thanks for the reply! It's indeed an issue w/ the initial pull and conversion of the Docker image. I've managed to resolve this by building the singularity image separately. Let me close this issue too, thanks!