Unhelpful error message in nf-core download when container registry is not defined or is wrong

remiolsen commented 6 months ago

Description of the bug

Hi! I was trying to download a non nf-core nextflow pipeline using nf-core download, which I understand could (or should?) work. However when I tried this for epi2me-labs/wf-single-cell version 1.1.0 it would fail in downloading the singularity images. I could gather after carefully reading the commandline output that it tried to download the images from quay.io when in reality the images are hosted on docker.io. The error message was not helpful in immediately identifying this as it only stated that it "failed for unclear reasons.".

This might be caused by the authors not specifying docker.io anywhere e.g. in docker.registry. I was able to get around this problem by using the option -l docker.io (thanks @ewels). This could be less troublesome by a more informative error message and/or not defaulting to quay.io if no registry is given - I'll leave that discussion to others.

Thanks.

Command used and terminal output

$ nf-core download epi2me-labs/wf-single-cell
[...]
INFO     Saving 'epi2me-labs/wf-single-cell'                                                                                                                                                                                                  
          Pipeline revision: 'v1.1.0'                                                                                                                                                                                                         
          Use containers: 'singularity'                                                                                                                                                                                                       
          Container library: 'quay.io'                                                                                                                                                                                                        
          Output directory: 'epi2me-labs-wf-single-cell_v1.1.0'                                                                                                                                                                               
          Include default institutional configuration: 'True'                                                                                                                                                                                 
INFO     Downloading centralised configs from GitHub                                                                                                                                                                                          
INFO     Downloading workflow files from GitHub                                                                                                                                                                                               
INFO     Processing workflow revision v1.1.0, found 2 container images in total.                                                                                                                                                              
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% • 0/2 completed
ontresearch/wf-common:sha2816439fd5aa81902836ea5794447b54947fa056  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ERROR    "ontresearch/wf-common:sha2816439fd5aa81902836ea5794447b54947fa056" failed for unclear reasons.                                                                                                                                      

INFO     Pulling of "ontresearch/wf-common:sha2816439fd5aa81902836ea5794447b54947fa056" failed.                                                                                                                                               
         Please troubleshoot the command                                                                                                                                                                                                      
         "singularity pull --name /crex/proj/sllstore2017079/private/remi/tmp/epi2me-labs-wf-single-cell_v1.1.0/singularity-images/ontresearch-wf-common-sha2816439fd5aa81902836ea5794447b54947fa056.img                                      
         docker://quay.io/ontresearch/wf-common:sha2816439fd5aa81902836ea5794447b54947fa056" manually.                                                                                                                                        

ERROR    "ontresearch/wf-common:sha2816439fd5aa81902836ea5794447b54947fa056" failed for unclear reasons.  
[...]

System information

nextflow version 23.04.3.5875
CentOS Linux 7
nf-core/tools 2.11.1
Python 3.10.6

remiolsen commented 6 months ago

In case anyone else is playing along with what I'm attempting to do, i.e. deploying non nf-core pipelines using download. I had to cut-and-paste the following nf-core stuff into the config to make it work:

params {
    // Config options
    custom_config_version      = 'master'
    custom_config_base         = "${projectDir}/../configs/"
    config_profile_description = null
    config_profile_contact     = null
    config_profile_url         = null
    config_profile_name        = null
}
// Load nf-core custom profiles from different Institutions
try {
    includeConfig "${params.custom_config_base}/nfcore_custom.config"
} catch (Exception e) {
    System.err.println("WARNING: Could not load nf-core/config profiles: ${params.custom_config_base}/nfcore_custom.config")
}
singularity.cacheDir = "${projectDir}/../singularity-images/"

MatthiasZepper commented 5 months ago

I could gather after carefully reading the commandline output that it tried to download the images from quay.io when in reality the images are hosted on docker.io.

Since nf-core images are primarily hosted on quay.io, I believe it is a reasonable default, if no explicit -l / --container-library parameter has been specified. The only alternative would be to make this argument mandatory without a default?

The error message was not helpful in immediately identifying this as it only stated that it "failed for unclear reasons."

Admittedly, this is not ideal. Since nf-core download only operates on the stderroutput of singularity pull to assess the cause of an error, it is somewhat limited. The registries have different error messages in the response bodies of the request, and it is difficult to keep the regexes/keywords appropriately in sync despite a corresponding test. Please feel free to submit a PR suggesting an appropriate change to the ContainerError class to make it work in that case.

The better solution would be to make a separate request to the URI prior to trying singularity pull and utilize the HTTP status codes rather than bits of the response body text. Again, if you feel like implementing this, a PR is more than welcome.

nf-core / tools