search

nf-core / tools

Python package with helper tools for the nf-core community.
https://nf-co.re
MIT License
242 stars 191 forks source link

`nf-core download` does not save some singularity images with the correct file name #2644

Closed Aratz closed 8 months ago

Aratz commented 10 months ago

Description of the bug

This is a follow up of https://github.com/nf-core/demultiplex/issues/164. I think I've nailed down the problem to nf-core/tools not saving singularity images with the right name when using nf-core download:

If the container is specified with e.g. container "nf-core/bclconvert:4.2.4", tools will download it as nf-core-bclconvert-4.2.4.img but nextflow expects quay.io-nf-core-bclconvert-4.2.4.img.

Command used and terminal output

$ nf-core download demultiplex

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.11.1 - https://nf-co.re

WARNING  Could not find GitHub authentication token. Some API requests may fail.                                                                                                                                   
? Select release / branch: 1.4.0  [release]
? Include the nf-core's default institutional configuration files into the download? Yes

In addition to the pipeline code, this tool can download software containers.
? Download software container images: singularity

Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity images are stored. This allows downloaded images to be cached in a 
central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: n

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little space.
? Choose compression type: none
INFO     Saving 'nf-core/demultiplex'                                                                                                                                                                              
          Pipeline revision: '1.4.0'                                                                                                                                                                               
          Use containers: 'singularity'                                                                                                                                                                            
          Container library: 'quay.io'                                                                                                                                                                             
          Output directory: 'nf-core-demultiplex_1.4.0'                                                                                                                                                            
          Include default institutional configuration: 'True'                                                                                                                                                      
INFO     Downloading centralised configs from GitHub                                                                                                                                                               
INFO     Downloading workflow files from GitHub                                                                                                                                                                    
INFO     Processing workflow revision 1.4.0, found 10 container images in total.                                                                                                                                   
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 10/10 completed
$ cd nf-core-demultiplex_1.4.0
$ ls singularity-images
depot.galaxyproject.org-singularity-falco-1.2.1--h867801b_3.img
depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img
depot.galaxyproject.org-singularity-fqtk-0.2.1--h9f5acd7_0.img
depot.galaxyproject.org-singularity-multiqc-1.17--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-multiqc-1.18--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-sgdemux-1.1.1--ha982bd6_0.img
depot.galaxyproject.org-singularity-ubuntu-20.04.img
nf-core-bases2fastq-1.1.0.img
nf-core-bcl2fastq-2.20.0.422.img
nf-core-bclconvert-4.2.4.img
$ NXF_ANSI_LOG=false NXF_SINGULARITY_CACHEDIR=singularity-images nextflow run 1_4_0/main.nf -profile singularity,test --outdir /tmp/test_demux
N E X T F L O W  ~  version 23.04.3
Launching `1_4_0/main.nf` [special_neumann] DSL2 - revision: a8c81121bd

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/demultiplex v1.4.0
------------------------------------------------------
Core Nextflow options
  runName                   : special_neumann
  containerEngine           : singularity
  launchDir                 : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0
  workDir                   : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/work
  projectDir                : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0
  userName                  : adrien
  profile                   : singularity,test
  configFiles               : 

Workflow options
  skip_tools                : []

Input/output options
  input                     : https://raw.githubusercontent.com/nf-core/test-datasets/demultiplex/samplesheet/1.3.0/flowcell_input.csv
  outdir                    : /tmp/test_demux

Demultiplexing options
  demultiplexer             : bclconvert

Institutional config options
  custom_config_base        : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0/../configs/
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/demultiplex for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.7153103

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/demultiplex/blob/master/CITATIONS.md

Pulling Singularity image docker://quay.io/nf-core/bclconvert:4.2.4 [cache /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/singularity-images/quay.io-nf-core-bclconvert-4.2.4.img]
[28/c0e23c] Submitted process > NFCORE_DEMULTIPLEX:DEMULTIPLEX:BCL_DEMULTIPLEX:BCLCONVERT (220422_M11111_0222_000000000-K9H97.1)
...

System information

nf-core, version 2.11.1 nextflow, version 23.04.3 build 5875 nf-core/demultixplex, version 1.4.0

MatthiasZepper commented 9 months ago

Also see #2751 and the lengthy discussion in Slack.

Bottom line: It is a feature, not a bug, but went out of sync with updates to Nextflow. As of now, we can only reintroduce the library or one needs to manually rename some images :-\

MatthiasZepper commented 8 months ago

Should be fixed with the nf-core tools release 2.13.1 today! 👌🎉

(I would still appreciate seeing an update to Nextflow as well at some point, but for now symlinks are created so that the images are found nonetheless.)