Error when run pipeline with docker file

[Ritzttk]

Hi, when I ran this command :

nextflow run nf-core/vipr -params-file vi.yaml -profile docker

I met this error:

N E X T F L O W  ~  version 0.31.1
Launching `nf-core/vipr` [gloomy_almeida] - revision: 1ca3412b9d [master]
==================================================
 nf-core/vipr : Viral amplicon/enrichment analysis and intrahost variant calling: v1.0-dev
==================================================
List of samples: PDH203_GTGGCC
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$trim_and_combine = <value>` with a process selector
[warm up] executor > local
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$decont = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$kraken = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$tadpole = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$gap_fill_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$polish_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$final_mapping = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$var_calling = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$genomecov = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$vipr_tools = <value>` with a process selector
[c6/04f746] Submitted process > trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)
ERROR ~ Error executing process > 'trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)'

Caused by:
  Process `trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)` terminated with an error exit status (125)

Command executed:

  # loop over readunits in pairs per sample
  pairno=0
  echo 425-N706-S517_S6_L001_R1_001.fastq 425-N706-S517_S6_L001_R2_001.fastq | xargs -n2 | while read fq1 fq2; do
      let pairno=pairno+1
      # note: don't make reads smaller than assembler kmer length
      skewer --quiet -t 4 -m pe -q 3 -n -l 31 -z -o pair${pairno}-skewer-out $fq1 $fq2;
      cat *-trimmed-pair1.fastq.gz >> PDH203_GTGGCC_R1-trimmed.fastq.gz;
      cat *-trimmed-pair2.fastq.gz >> PDH203_GTGGCC_R2-trimmed.fastq.gz;
      rm *-trimmed-pair[12].fastq.gz;
  done
  fastqc -t {task.cpus} PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz;

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'nfcore/vipr:1.0-dev' locally
  docker: Error response from daemon: manifest for nfcore/vipr:1.0-dev not found.
  See 'docker run --help'.

Work dir:
  /home/vi/work/c6/04f746274f2e2baec8dcb1b457ba47

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details
Oops... Pipeline execution stopped with the following message: Unable to find image 'nfcore/vipr:1.0-dev' locally
docker: Error response from daemon: manifest for nfcore/vipr:1.0-dev not found.
See 'docker run --help'.
Pipeline execution summary
    ---------------------------
    Completed at : Mon Sep 24 03:55:39 UTC 2018
    Duration     : 5.6s
    Success      : false
    Work Dir     : /home/vi/work
    Exit status  : 125
    Error report : Error executing process > 'trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)'

Caused by:
  Process `trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)` terminated with an error exit status (125)

Command executed:

  # loop over readunits in pairs per sample
  pairno=0
  echo 425-N706-S517_S6_L001_R1_001.fastq 425-N706-S517_S6_L001_R2_001.fastq | xargs -n2 | while read fq1 fq2; do
      let pairno=pairno+1
      # note: don't make reads smaller than assembler kmer length
      skewer --quiet -t 4 -m pe -q 3 -n -l 31 -z -o pair${pairno}-skewer-out $fq1 $fq2;
      cat *-trimmed-pair1.fastq.gz >> PDH203_GTGGCC_R1-trimmed.fastq.gz;
      cat *-trimmed-pair2.fastq.gz >> PDH203_GTGGCC_R2-trimmed.fastq.gz;
      rm *-trimmed-pair[12].fastq.gz;
  done
  fastqc -t {task.cpus} PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz;

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'nfcore/vipr:1.0-dev' locally
  docker: Error response from daemon: manifest for nfcore/vipr:1.0-dev not found.
  See 'docker run --help'.

Work dir:
  /home/vi/work/c6/04f746274f2e2baec8dcb1b457ba47

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.

After that, I think I have prolem with the url of docker, So I edited the nextflow.config from container = "nfcore/vipr:${version}" // for GIS translated into conda env name! to container = "nfcore/vipr So the pipeline worked on first process. But it error on the second process

>N E X T F L O W  ~  version 0.31.1
Launching `nf-core/vipr` [ridiculous_boyd] - revision: 1ca3412b9d [master]
==================================================
 nf-core/vipr : Viral amplicon/enrichment analysis and intrahost variant calling: v1.0-dev
==================================================
List of samples: PDH203_GTGGCC
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$trim_and_combine = <value>` with a process selector
[warm up] executor > local
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$decont = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$kraken = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$tadpole = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$gap_fill_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$polish_assembly = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$final_mapping = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$var_calling = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$genomecov = <value>` with a process selector
WARN: Process configuration syntax $processName has been deprecated -- Replace `process.$vipr_tools = <value>` with a process selector
[f8/88c7c1] Submitted process > trim_and_combine (Preprocessing of 1  read pairs for PDH203_GTGGCC)
[42/27852b] Submitted process > decont (Decontaminating PDH203_GTGGCC)
ERROR ~ Error executing process > 'decont (Decontaminating PDH203_GTGGCC)'

Caused by:
  Process `decont (Decontaminating PDH203_GTGGCC)` terminated with an error exit status (1)

Command executed:

  decont.py -i PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz -t 4 -c 0.5 -r human_g1k_v37.fasta -o PDH203_GTGGCC_trimmed_decont;
  # since this is the last fastqc processing step, let's run fastqc here
  fastqc -t {task.cpus} PDH203_GTGGCC_trimmed_decont_1.fastq.gz PDH203_GTGGCC_trimmed_decont_2.fastq.gz;

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  Traceback (most recent call last):
    File "/opt/conda/bin/decont.py", line 449, in <module>
      assert shutil.which('bwa') and shutil.which('samtools')
  AttributeError: 'module' object has no attribute 'which'

Work dir:
  /home/vi/.nextflow/assets/nf-core/vipr/work/42/27852be75b8444b072ea29c2bbaa61

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
Oops... Pipeline execution stopped with the following message: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/opt/conda/bin/decont.py", line 449, in <module>
    assert shutil.which('bwa') and shutil.which('samtools')
AttributeError: 'module' object has no attribute 'which'
Pipeline execution summary
    ---------------------------
    Completed at : Mon Sep 24 04:09:36 UTC 2018
    Duration     : 24.8s
    Success      : false
    Work Dir     : /home/vi/.nextflow/assets/nf-core/vipr/work
    Exit status  : 1
    Error report : Error executing process > 'decont (Decontaminating PDH203_GTGGCC)'

Caused by:
  Process `decont (Decontaminating PDH203_GTGGCC)` terminated with an error exit status (1)

Command executed:

  decont.py -i PDH203_GTGGCC_R1-trimmed.fastq.gz PDH203_GTGGCC_R2-trimmed.fastq.gz -t 4 -c 0.5 -r human_g1k_v37.fasta -o PDH203_GTGGCC_trimmed_decont;
  # since this is the last fastqc processing step, let's run fastqc here
  fastqc -t {task.cpus} PDH203_GTGGCC_trimmed_decont_1.fastq.gz PDH203_GTGGCC_trimmed_decont_2.fastq.gz;

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  Traceback (most recent call last):
    File "/opt/conda/bin/decont.py", line 449, in <module>
      assert shutil.which('bwa') and shutil.which('samtools')
  AttributeError: 'module' object has no attribute 'which'

Work dir:
  /home/vi/.nextflow/assets/nf-core/vipr/work/42/27852be75b8444b072ea29c2bbaa61

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.

I think, i had problem with the docker file. Do you recommend I will use docker or using conda for software dependencies ?.

Kiên

[ewels]

Hi @Ritzttk,

You have two problems here. The first you correctly diagnosed and fixed, is the missing docker image tag. This is resolved in #30 by essentially doing what you did above - removing the :1.0-dev prefix (though I replace it with :latest, which is the default).

After this, you have a warning that I've not seen before:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

Though this doesn't seem to be a blocking problem. Maybe @pditommaso has seen this before?

Then you also have a python error from the decont.py script:

Traceback (most recent call last):
  File "/opt/conda/bin/decont.py", line 449, in <module>
    assert shutil.which('bwa') and shutil.which('samtools')
AttributeError: 'module' object has no attribute 'which'

A quick bit of googling suggests that this is because the version of Python is too old. I've just specified Python 3.6 in the conda environment file (also in #30), hopefully this will solve the problem.

Phil

[pditommaso]

Never seen, but googling for it, it may be a problem with your docker version or the base image.

[ewels]

Hi @Ritzttk,

The new Docker hub image is now building. Hopefully when it's done the AttributeError problem should be fixed, but we'll see. Please let me know how you get on!

Cheers,

Phil

[Ritzttk]

Hi @ewels , The pipeline with new Docker image works now. Thank you and your team so much.

Kiên