Download of sourmash and vibrant databases always fail

[pentamorfico]

Command used:

nextflow run replikation/What_the_Phage -r v1.2.1 --setup -profile local,singularity -work-dir ./work --cachedir ./singularity_images

I have tried more than 10 times, deleting folders, and caching dirs, and it always seems to fail, either in the Sourmash or the Vibrant download step. I have tried all the solutions proposed in the topics, but none of them are working.

The error that I get:

Error executing process > 'setup_wf:sourmash_download_DB'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name nanozoo-sourmash-3.4.1--16a8db7.img.pulling.1688470482137 docker://nanozoo/sourmash:3.4.1--16a8db7 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b
    Copying blob sha256:9c388eb6d33c40662539172f0d9a357287bd1cd171692ca5c08db2886bc527c3
    Copying blob sha256:96cf53b3a9dd496f4c91ab86eeadca2c8a31210c2e5c2a82badbb0dcf8c8f76b
    Copying blob sha256:ef57804bb8d7bd43551c7578105df7bd6883f6195b5da718e3d7bf0380136d8c
    Copying blob sha256:7c508a59c9fa531bc7b5b4dc85bd11f2ae0b8b411a0abcae3e66d14f5ff52ebd
    Copying blob sha256:fc62efe8c22929d7ef3c510cba16533039db9858387f78b4aeeb0b1ecfa6d55f
    Copying config sha256:54d385a016d39a69f08d9bcb24717ac334414d6fa9a9f79b1e0a455668d5b177
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: no descriptor found for reference "5a0bb1f5021f4cc68f5553771606701c75c55beca34d769a2116b5bbcf9774c2"

If I manually run the following line:

singularity pull  --name nanozoo-sourmash-3.4.1--16a8db7.img.pulling.1688470482137 docker://nanozoo/sourmash:3.4.1--16a8db7 > /dev/null

It works

so It doesn't seem to be a problem of the image pulling?

[mult1fractal]

Hey,

you could download the remaining docker/singularity files manually and store them at --cachedir ./singularity_images

I know that's not the convenient solution but the fastest for now.

In a quick google search I found the following:

At the beginning of the pipeline execution, I get an error message such as FATAL: While making image from oci registry: while building SIF from layers: conveyor failed to get: no descriptor found for reference or any other mentioning OCI, SIF or Singularity.

Ensure you have an up-to-date version of Singularity. Otherwise you may need to clean some Singularity directories, the singularity one (where pipeline images are stored) in FA-nf base directory and .singularity in your $HOME directory.

I will install singularity on my workstation and replicate your command and check if I get the same error

Best Mike

[pentamorfico]

Hi Mike!

Thanks you very much for your suppafast reply! Doing it that way, It worked! :smiley:

I noticed now that I was following the instructions for the install in the documentation:

conda create -n wtp nextflow==20.07.01 singularity==3.6.1
conda activate wtp
(wtp) $ nextflow run replikation/What_the_Phage -r v1.2.0 --setup ...

And I needed to change the nexftlow version from 20.07.01 to 21.04. Perhaps singularity also needs to be updated to a more recent version? Could that be the cause of the issue? I can try to replicate the error in a new environment if that helps

Sorry if my previous message sounded a bit rude, I was in a rush :sweat:

[mult1fractal]

thats what this issue section is for :D, also I want to escape thesis writing

so I installed singularity-3.5.3.tar.gz and I got the same issue like you. I replaced the "old sourmash 3.4.1 docker' with a new one and this worked

waiting now for Vibrant

[mult1fractal]

when you have more time (tomorrow or so) you could try the latest singularty environment? and run the setup command and see if it will work then..

if yes I will update the documentation 👍🏽