search

nf-core / funcscan

(Meta-)genome screening for functional and natural product gene sequences
https://nf-co.re/funcscan
MIT License
69 stars 20 forks source link

Incorrect docker container uris #281

Closed sunitj closed 1 year ago

sunitj commented 1 year ago

Description of the bug

I seem to be getting errors like this with v1.1.1 with the docker profile:

nextflow.exception.ProcessUnrecoverableException: Task failed to start - CannotPullContainerError: Error response from daemon: pull access denied for nf-core/ubuntu, repository does not exist or may require 'docker login': denied: requested access to the resource is denied;

I tested by downloading nf-core/ubuntu manually, but I get the same error:

$ docker pull nf-core/ubuntu:20.04
Using default tag: latest
Error response from daemon: pull access denied for nf-core/ubuntu, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

To ensure that this wasn't an actual issue with my user. I also tried downloading the hello-world image, and that worked.

$ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
719385e32844: Pull complete
Digest: sha256:fc6cf906cbfa013e80938cdf0bb199fbdbb86d6e3e013783e5a766f50f5dbce0
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

Further investigation, seem to also indicate that URIs should start with quay.io/biocontainers/ start with biocontainers instead.

Command used and terminal output

No response

Relevant files

Here is a summary of all the incorrect URIs I could find and what I think they should be.

file current docker URI correct docker URI
nf-core/gunzip/main.nf 'nf-core/ubuntu:20.04' 'ubuntu:20.04'
nf-core/untar/main.nf 'nf-core/ubuntu:20.04' 'ubuntu:20.04'
nf-core/ampcombi/main.nf 'biocontainers/ampcombi:0.1.7--pyhdfd78af_0' 'quay.io/biocontainers/ampcombi:0.1.7--pyhdfd78af_0'
nf-core/ampir/main.nf 'biocontainers/r-ampir:1.1.0' 'quay.io/biocontainers/r-ampir:1.1.0'
nf-core/bioawk/main.nf 'biocontainers/bioawk:1.0--h5bf99c6_6' 'quay.io/biocontainers/bioawk:1.0--h5bf99c6_6'
nf-core/fargene/main.nf 'biocontainers/fargene:0.1--py27h21c881e_4' 'quay.io/biocontainers/fargene:0.1--py27h21c881e_4'
nf-core/fastqc/main.nf 'biocontainers/fastqc:0.11.9--0' 'quay.io/biocontainers/fastqc:0.11.9--0'
nf-core/multiqc/main.nf 'biocontainers/multiqc:1.14--pyhdfd78af_0' 'quay.io/biocontainers/multiqc:1.14--pyhdfd78af_0'
nf-core/prodigal/main.nf 'biocontainers/mulled-v2-2e442ba7b07bfa102b9cf8fac6221263cd746ab8:57f05cfa73f769d6ed6d54144cb3aa2a6a6b17e0-0' 'quay.io/biocontainers/mulled-v2-2e442ba7b07bfa102b9cf8fac6221263cd746ab8:57f05cfa73f769d6ed6d54144cb3aa2a6a6b17e0-0'
nf-core/prokka/main.nf 'biocontainers/prokka:1.14.6--pl5321hdfd78af_4' 'quay.io/biocontainers/prokka:1.14.6--pl5321hdfd78af_4'
nf-core/pyrodigal/main.nf 'biocontainers/mulled-v2-2fe9a8ce513c91df34b43a6610df94c3a2eb3bd0:697b3838b186fac6a9ceec198b09d4032162a079-0' 'quay.io/biocontainers/mulled-v2-2fe9a8ce513c91df34b43a6610df94c3a2eb3bd0:697b3838b186fac6a9ceec198b09d4032162a079-0'

System information

Nextflow: 23.04.1 Hardware: EC2 instance Executor: local and AWS Batch Container engine: Docker OS: Amazon Linux 2 nf-core/funcscan version: 1.1.1

jfy133 commented 1 year ago

I think this might be related to https://github.com/nf-core/funcscan/pull/279

jfy133 commented 1 year ago

And by related, I mean solved

jfy133 commented 1 year ago

@sunitj could you see if running with -r dev solves the problem? (x-post from slack)

sunitj commented 1 year ago

@jfy133 I'm not sure if this is related, but I'm getting this error:

pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fa is a symbolic link

More context:

Jun-01 17:53:54.392 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL (SPAdes-MetaBAT2-group-H1167_CA004_2060.45); work-dir=/mnt/efs/scratch/sunit/funcscan/TEST/20230601_dev/work/d1/3773d8c81bf52d679490e68c0ed00d
  error [nextflow.exception.ProcessFailedException]: Process `NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL (SPAdes-MetaBAT2-group-H1167_CA004_2060.45)` terminated with an error exit status (1)
Jun-01 17:53:54.408 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL (SPAdes-MetaBAT2-group-H1167_CA004_2060.45)'

Caused by:
  Process `NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL (SPAdes-MetaBAT2-group-H1167_CA004_2060.45)` terminated with an error exit status (1)

Command executed:

  pigz -cdf SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fa > pigz_fasta.fna

  pyrodigal \
      -p meta   -g 11 \
      -i pigz_fasta.fna \
      -o SPAdes-MetaBAT2-group-H1167_CA004_2060.45.gff \
      -d SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fna \
      -a SPAdes-MetaBAT2-group-H1167_CA004_2060.45.faa \
      -s SPAdes-MetaBAT2-group-H1167_CA004_2060.45.score

  pigz -nm SPAdes-MetaBAT2-group-H1167_CA004_2060.45*

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL":
      pyrodigal: $(echo $(pyrodigal --version 2>&1 | sed 's/pyrodigal v//'))
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fa is a symbolic link

Work dir:
  /mnt/efs/scratch/sunit/funcscan/TEST/20230601_dev/work/d1/3773d8c81bf52d679490e68c0ed00d

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

Running the .command.run separately:

$ cd /mnt/efs/scratch/sunit/funcscan/TEST/20230601_dev/work/d1/3773d8c81bf52d679490e68c0ed00d
$ bash .command.run
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fa is a symbolic link
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.faa.gz exists
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.faa.gz ends with .gz
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fna.gz exists
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.fna.gz ends with .gz
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.gff.gz exists
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.gff.gz ends with .gz
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.score.gz exists
pigz: skipping: SPAdes-MetaBAT2-group-H1167_CA004_2060.45.score.gz ends with .gz

Command used:

nextflow run nf-core/funcscan \
    -r dev \
    --input test_samplesheet.csv \
    --outdir CA004 \
    -profile docker \
    --run_bgc_screening \
    --save_databases \
    --bgc_skip_hmmsearch
sunitj commented 1 year ago

For a little more context, I was able to run the same test_samplesheet.csv using the same command (except the -r dev) without any issues on v1.1.0.

Command:

nextflow run nf-core/funcscan \
    -r 1.1.0 \
    --input test_samplesheet.csv \
    --outdir CA004 \
    -profile docker \
    --run_bgc_screening \
    --save_databases \
    --bgc_skip_hmmsearch

Output:

executor >  local (92)
[1e/dc357f] process > NFCORE_FUNCSCAN:FUNCSCAN:INPUT_CHECK:SAMPLESHEET_CHECK (test_samplesheet.csv)               [100%] 1 of 1 ✔
[b4/9080ff] process > NFCORE_FUNCSCAN:FUNCSCAN:GUNZIP_FASTA_PREP (SPAdes-MetaBAT2-group-H621_WT_Male.63.fa.gz)    [100%] 10 of 10 ✔
[56/43f7c8] process > NFCORE_FUNCSCAN:FUNCSCAN:BIOAWK (SPAdes-MetaBAT2-group-H621_CA004_SHAM.45)                  [100%] 10 of 10 ✔
[7c/7e04ca] process > NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL (SPAdes-MetaBAT2-group-H621_CA004_SHAM.45)               [100%] 10 of 10 ✔
[3a/d7f8cd] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:UNTAR_CSS (css.tar.gz)                                         [100%] 1 of 1 ✔
[3e/389697] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:UNTAR_DETECTION (detection.tar.gz)                             [100%] 1 of 1 ✔
[a5/dda0cf] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:UNTAR_MODULES (modules.tar.gz)                                 [100%] 1 of 1 ✔
[33/a624c4] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:ANTISMASH_ANTISMASHLITEDOWNLOADDATABASES                       [100%] 1 of 1 ✔
[88/98c20c] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:ANTISMASH_ANTISMASHLITE (SPAdes-MetaBAT2-group-H1651_WT_IC.54) [100%] 10 of 10 ✔
[de/2fd151] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:DEEPBGC_DOWNLOAD                                               [100%] 1 of 1 ✔
[02/56a33c] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:DEEPBGC_PIPELINE (SPAdes-MetaBAT2-group-H621_CA004_SHAM.45)    [100%] 10 of 10 ✔
[0b/2b45d0] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:GECCO_RUN (MEGAHIT-MetaBAT2-group-H1074_CA004_2058.25)         [100%] 10 of 10 ✔
[c2/4ad49a] process > NFCORE_FUNCSCAN:FUNCSCAN:BGC:COMBGC (SPAdes-MetaBAT2-group-H621_CA004_SHAM.45)              [100%] 24 of 24 ✔
[0f/826419] process > NFCORE_FUNCSCAN:FUNCSCAN:CUSTOM_DUMPSOFTWAREVERSIONS (1)                                    [100%] 1 of 1 ✔
[60/ffbdab] process > NFCORE_FUNCSCAN:FUNCSCAN:MULTIQC                                                            [100%] 1 of 1 ✔
-[nf-core/funcscan] Pipeline completed successfully-
Completed at: 01-Jun-2023 19:27:50
Duration    : 37m 16s
CPU hours   : 6.8
Succeeded   : 92
jfy133 commented 1 year ago

Should be fixed in: https://github.com/nf-core/funcscan/pull/279

1.1.2 coming imminentnyl!