Some singularity images are missing when using `nf-core download demultiplex`

[Aratz]

Description of the bug

After downloading the singularity images with nf-core download demultiplex, nextflow still attempts to download the bclconvert singularity image (similar problems occur with bcl2fastq and bases2fastq).

I suspect this has to do with how the images are named, i.e. the singularity_images folder still contains a file named nf-core-bclconvert-4.2.4.img but nextflow insists on downloading quay.io-nf-core-bclconvert-4.2.4.img.

Another (smaller) issue is that multiQC is downloaded twice (v1.17 and v1.18).

Command used and terminal output

$ nf-core download demultiplex

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.11.1 - https://nf-co.re

WARNING  Could not find GitHub authentication token. Some API requests may fail.                                                                                                                                   
? Select release / branch: 1.4.0  [release]
? Include the nf-core's default institutional configuration files into the download? Yes

In addition to the pipeline code, this tool can download software containers.
? Download software container images: singularity

Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity images are stored. This allows downloaded images to be cached in a 
central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: n

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little space.
? Choose compression type: none
INFO     Saving 'nf-core/demultiplex'                                                                                                                                                                              
          Pipeline revision: '1.4.0'                                                                                                                                                                               
          Use containers: 'singularity'                                                                                                                                                                            
          Container library: 'quay.io'                                                                                                                                                                             
          Output directory: 'nf-core-demultiplex_1.4.0'                                                                                                                                                            
          Include default institutional configuration: 'True'                                                                                                                                                      
INFO     Downloading centralised configs from GitHub                                                                                                                                                               
INFO     Downloading workflow files from GitHub                                                                                                                                                                    
INFO     Processing workflow revision 1.4.0, found 10 container images in total.                                                                                                                                   
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 10/10 completed
$ cd nf-core-demultiplex_1.4.0
$ ls singularity-images
depot.galaxyproject.org-singularity-falco-1.2.1--h867801b_3.img
depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img
depot.galaxyproject.org-singularity-fqtk-0.2.1--h9f5acd7_0.img
depot.galaxyproject.org-singularity-multiqc-1.17--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-multiqc-1.18--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-sgdemux-1.1.1--ha982bd6_0.img
depot.galaxyproject.org-singularity-ubuntu-20.04.img
nf-core-bases2fastq-1.1.0.img
nf-core-bcl2fastq-2.20.0.422.img
nf-core-bclconvert-4.2.4.img
$ NXF_ANSI_LOG=false NXF_SINGULARITY_CACHEDIR=singularity-images nextflow run 1_4_0/main.nf -profile singularity,test --outdir /tmp/test_demux
N E X T F L O W  ~  version 23.04.3
Launching `1_4_0/main.nf` [special_neumann] DSL2 - revision: a8c81121bd

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/demultiplex v1.4.0
------------------------------------------------------
Core Nextflow options
  runName                   : special_neumann
  containerEngine           : singularity
  launchDir                 : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0
  workDir                   : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/work
  projectDir                : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0
  userName                  : adrien
  profile                   : singularity,test
  configFiles               : 

Workflow options
  skip_tools                : []

Input/output options
  input                     : https://raw.githubusercontent.com/nf-core/test-datasets/demultiplex/samplesheet/1.3.0/flowcell_input.csv
  outdir                    : /tmp/test_demux

Demultiplexing options
  demultiplexer             : bclconvert

Institutional config options
  custom_config_base        : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0/../configs/
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/demultiplex for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.7153103

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/demultiplex/blob/master/CITATIONS.md

Pulling Singularity image docker://quay.io/nf-core/bclconvert:4.2.4 [cache /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/singularity-images/quay.io-nf-core-bclconvert-4.2.4.img]
[28/c0e23c] Submitted process > NFCORE_DEMULTIPLEX:DEMULTIPLEX:BCL_DEMULTIPLEX:BCLCONVERT (220422_M11111_0222_000000000-K9H97.1)
...

Relevant files

No response

System information

nf-core, version 2.11.1 nextflow, version 23.04.3 build 5875 nf-core/demultixplex, version 1.4.0

[maxulysse]

I'd say you're right on point with your issue, but I feel like the first part is more of a tools issue than demultiplex.

I'm guessing multiqc issues are due to custom/dumpsoftwareversions, and this is a demultiplex issue, but hopefully we're getting rid of this module soon

[Aratz]

Right, it seems like tools is getting confused with this line https://github.com/nf-core/modules/blob/3fb69a931aa3c968ac87bc17961f30da4c9f17b5/modules/nf-core/bclconvert/main.nf#L5 (compare to the url for falco for instance: https://github.com/nf-core/modules/blob/3fb69a931aa3c968ac87bc17961f30da4c9f17b5/modules/nf-core/falco/main.nf#L7-L9)

[Aratz]

Just created a follow-up issue in nf-core/tools, we can leave this one open for awareness.

[MatthiasZepper]

The main issue should be fixed with the nf-core tools release 2.13.1 today! 👌🎉

(I would still appreciate seeing an update to Nextflow in the future, but for now symlinks ensure that the images are found nonetheless.)

Leaving this open as a reminder of the dissociative identity disorder of MultiQC.

[grst]

I think we can close this. Modules subworkflows have been updated in the meanwhile. There's also no guarantee that all modules use the same version of the multiqc container (the multiqc container is sometimes used not for multiqc, but just as a python container that happens to have a couple of useful packages installed).