[Bug]: I can not find docker images used for running the wf-human-variation workflows.

[jhxaz]

What happened?

First of all, thanks for your hard work, I think epi2me-labs workflows are excellent, both of them help me a lot. What confused me is that the Docker images link is missing, another problem is the introduction and help document seems too simple for Docker users, and I did spend a few days to study how the pipeline works, but unfortunately until now I don‘t know how the pipeline work in Docker mode. Could you please add more details on the page? Most of the user use Conda, but in my country the network always breaks, so Docker is better as you suggest.

Operating System

ubuntu 20.04

Workflow Execution

Command line

Workflow Execution - EPI2ME Labs Versions

No response

Workflow Execution - Execution Profile

Docker

Workflow Version

v0.22

Relevant log output

nextflow run epi2me-labs/wf-human-sv     -profile standard    --fastq demo_data/reads.fastq.gz     --reference demo_data/chr20_human_g1k_v37_part.fasta.gz     --target_bedfile demo_data/target.bed     --tr_bedfile demo_data/human_hs37d5.trf.bed     --out_dir ${OUTPUT}
N E X T F L O W  ~  version 22.04.5
Launching `https://github.com/epi2me-labs/wf-human-sv` [desperate_gutenberg] DSL2 - revision: 0418aa8f66 [master]
Core Nextflow options
  revision       : master
  runName        : desperate_gutenberg
  containerEngine: docker
  launchDir      : /mnt/memory2/database/SoftWare/Nanopore_epi2me-labs_wf-human-variation
  workDir        : /mnt/memory2/database/SoftWare/Nanopore_epi2me-labs_wf-human-variation/work
  projectDir     : /mnt/memory2/users/liuxiaozhao/.nextflow/assets/epi2me-labs/wf-human-sv
  userName       : liuxiaozhao
  profile        : standard
  configFiles    : /mnt/memory2/users/liuxiaozhao/.nextflow/assets/epi2me-labs/wf-human-sv/nextflow.config

Input/output options
  fastq          : demo_data/reads.fastq.gz
  reference      : demo_data/chr20_human_g1k_v37_part.fasta.gz
  out_dir        : tmp_test_out_dir
  tr_bedfile     : demo_data/human_hs37d5.trf.bed

Benchmarking options
  target_bedfile : demo_data/target.bed

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-human-sv for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

WARN: Access to undefined parameter `disable_ping` -- Initialise it to a default value eg. `params.disable_ping = some_value`
==============================================
Checking inputs
Checking fastq input.
Single file input detected.
executor >  local (2)
[-        ] process > handleSingleFile                   [  0%] 0 of 1
[-        ] process > fastq:readAlign:combineFilterFastq -
[-        ] process > fastq:readAlign:mapMM2             -
[-        ] process > fastq:variantCall:filterBam        -
[-        ] process > fastq:variantCall:sniffles2        -
[-        ] process > fastq:variantCall:mosdepth         -
[-        ] process > fastq:variantCall:filterCalls      -
[-        ] process > fastq:variantCall:sortVCF          -
[-        ] process > fastq:variantCall:indexVCF         -
[7e/a11683] process > fastq:runReport:getVersions        [  0%] 0 of 1
[f7/9b4a0d] process > fastq:runReport:getParams          [  0%] 0 of 1
executor >  local (3)
[-        ] process > handleSingleFile (1)               -
[-        ] process > fastq:readAlign:combineFilterFastq -
[-        ] process > fastq:readAlign:mapMM2             -
[-        ] process > fastq:variantCall:filterBam        -
[-        ] process > fastq:variantCall:sniffles2        -
[-        ] process > fastq:variantCall:mosdepth         -
[-        ] process > fastq:variantCall:filterCalls      -
[-        ] process > fastq:variantCall:sortVCF          -
[-        ] process > fastq:variantCall:indexVCF         -
[7e/a11683] process > fastq:runReport:getVersions        [100%] 1 of 1, failed: 1 ✘
[-        ] process > fastq:runReport:getParams          -
[-        ] process > fastq:runReport:report             -
[-        ] process > output                             -
=============================================
Running workflow | .fq input | standard mode.
Error executing process > 'fastq:runReport:getVersions'

Caused by:
  Process `fastq:runReport:getVersions` terminated with an error exit status (127)

Command executed:

  trap '' PIPE # suppress SIGPIPE without interfering with pipefail
  python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
  TRUVARI=$(which truvari)
  python $TRUVARI version | sed 's/ /,/' >> versions.txt
  mosdepth --version | sed 's/ /,/' >> versions.txt
  fastcat --version | sed 's/^/fastcat,/' >> versions.txt
  sniffles --version | head -n 1 | sed 's/ Version //' >> versions.txt
  bcftools --version | head -n 1 | sed 's/ /,/' >> versions.txt
  bedtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
  samtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
  minimap2 --version | head -n 1 | sed 's/^/minimap2,/' >> versions.txt
  echo `seqtk 2>&1 | head -n 3 | tail -n 1 | cut -d ':' -f 2 | sed 's/ /seqtk,/'` >> versions.txt

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 3: python: command not found

Work dir:
  /mnt/memory2/database/SoftWare/Nanopore_epi2me-labs_wf-human-variation/work/7e/a116832762c610b883a58ffe1fef16

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

[SamStudio8]

Hi @jhxaz! Our docker images can be found on Dockerhub (e.g. https://hub.docker.com/u/ontresearch), however Nextflow will take care of downloading and loading images as you have provided the -profile standard option, and so you don't need to worry about downloading the images yourself to get started.

You've actually tried to run the now-deprecated wf-human-sv pipeline which may explain the error that you've shared here. Please could you try again with the new wf-human-variation pipeline instead? You'll want to use nextflow run epi2me-labs/wf-human-variation and use the --sv option to enable SV calling.

[jhxaz]

Thanks for your quickly response, I realized that both Docker and Conda environment could be pulled automatically by nextflow run command, this is wonderful! Now I am testing the wf-human-variation pipeline you suggested. Another question is, the input files of the example code seems both for --snp and --sv mode, when I just run them separately, for examples only --sv mode, do I still need to give all the bam/bed/ref/--model files? And it seems fastq files are not the input files, then how does it works for sv mapping when I have fastq files? Could you please give the separately command for --snp and --sv mode?

examples, when we only run --snp or--sv mode separately, the command of input still the same?

nextflow run epi2me-labs/wf-human-variation \ -w ${OUTPUT}/workspace \ -profile standard \ --snp --sv \ --bam demo_data/demo.bam \ --bed demo_data/demo.bed \ --ref demo_data/demo.fasta \ --model demo_data/ont_r104_e81_sup_g5015 \ --out_dir ${OUTPUT}

[jhxaz]

@SamStudio8 I have run the example command for test data . but I still get the error Caused by: Process snp:getParams terminated with an error exit status (125)

$ nextflow run epi2me-labs/wf-human-variation \

-profile standard \
--snp --sv \
--bam demo_data/demo.bam \
--bed demo_data/demo.bed \
--ref demo_data/demo.fasta \
--model demo_data/ont_r104_e81_sup_g5015 \
--out_dir ${OUTPUT}

N E X T F L O W ~ version 22.04.5 Launching https://github.com/epi2me-labs/wf-human-variation [romantic_archimedes] DSL2 - revision: 898280e1ea [master] Core Nextflow options revision : master runName : romantic_archimedes containerEngine : docker container : ontresearch/wf-human-variation:sha90e1c9ae1b7cd51548c4c9b66df424fe3c3a81a8 launchDir : /mnt/memory2/database/SoftWare/Nanopore_epi2me-labs_wf-human-variation workDir : /mnt/memory2/database/SoftWare/Nanopore_epi2me-labs_wf-human-variation/work projectDir : /mnt/memory2/users/liuxiaozhao/.nextflow/assets/epi2me-labs/wf-human-variation userName : liuxiaozhao profile : standard configFiles : /mnt/memory2/users/liuxiaozhao/.nextflow/assets/epi2me-labs/wf-human-variation/nextflow.config

Basic Input/Output Options bam : demo_data/demo.bam ref : demo_data/demo.fasta bed : demo_data/demo.bed model : demo_data/ont_r104_e81_sup_g5015 out_dir : tmp_test_out_dir ubam_sort_threads : 3 ubam_bam2fq_threads: 1

wf-human-sv options sv : true

wf-human-snp options snp : true

Other parameters process_label : wfdefault

!! Only displaying parameters that differ from the pipeline defaults !!

If you use epi2me-labs/wf-human-variation for your analysis please cite:

• The nf-core framework https://doi.org/10.1038/s41587-020-0439-x

executor > local (5) [2a/307089] process > cram_cache (1) [ 0%] 0 of 1 [- ] process > snp:make_chunks - [- ] process > snp:pileup_variants - [- ] process > snp:aggregate_pileup_variants - [- ] process > snp:select_het_snps - [- ] process > snp:phase_contig - [- ] process > snp:get_qual_filter - [- ] process > snp:create_candidates - [- ] process > snp:evaluate_candidates - [- ] process > snp:aggregate_full_align_variants - [- ] process > snp:merge_pileup_and_full_vars - [- ] process > snp:aggregate_all_variants - [- ] process > snp:readStats - [15/f2ac31] process > snp:getVersions [ 0%] 0 of 1 [12/2aa0b7] process > snp:getParams [ 0%] 0 of 1 [- ] process > snp:vcfStats - [- ] process > snp:mosdepth - [- ] process > snp:makeReport - [- ] process > output_snp - [- ] process > sv:variantCall:filterBam - [- ] process > sv:variantCall:sniffles2 - [- ] process > sv:variantCall:mosdepth - [- ] process > sv:variantCall:filterCalls - [- ] process > sv:variantCall:sortVCF - [- ] process > sv:variantCall:indexVCF - [40/1b4683] process > sv:runReport:getVersions [ 0%] 0 of 1 [cc/dee338] process > sv:runReport:getParams [ 0%] 0 of 1 [- ] process > sv:runReport:report - [- ] process > output_sv - Error executing process > 'snp:getParams'

Caused by: Process snp:getParams terminated with an error exit status (125)

Command executed:

Output nextflow params object to JSON

      echo '{
  "help": false,
  "disable_ping": false,
  "threads": 4,
  "guppy_map_threads": 6,
  "guppy_basemod_threads": 2,
  "ubam_map_threads": 12,
  "ubam_sort_threads": 3,
  "ubam_bam2fq_threads": 1,
  "wfversion": "v0.2.2",
  "aws_image_prefix": null,
  "aws_queue": null,
  "out_dir": "tmp_test_out_dir",
  "report_name": "report",
  "process_label": "wfdefault",
  "snp": true,
  "sv": true,
  "methyl": false,
  "fast5_dir": null,
  "ubam": null,
  "bam": "demo_data/demo.bam",
  "ref": "demo_data/demo.fasta",
  "bed": "demo_data/demo.bed",
  "fast5_chunk_size": 100,
  "guppy_args": null,
  "guppy_no_align": false,
  "model": "demo_data/ont_r104_e81_sup_g5015",
  "call_retries": 5,
  "phase_vcf": false,
  "use_longphase": true,
  "use_longphase_intermediate": true,
  "sample_name": "SAMPLE",
  "ctg_name": "EMPTY",
  "include_all_ctgs": false,
  "ref_pct_full": 0.1,
  "var_pct_full": 0.7,
  "GVCF": false,
  "snp_min_af": 0.08,
  "indel_min_af": 0.15,
  "vcf_fn": "EMPTY",
  "min_contig_size": 0,
  "min_mq": 5,
  "min_cov": 2,
  "tr_bed": null,
  "sv_types": "DEL,INS",
  "min_sv_length": 30,
  "max_sv_length": 100000,
  "min_read_support": "auto",
  "min_read_support_limit": 2,
  "cluster_merge_pos": 150,
  "sniffles_args": null,
  "monochrome_logs": false,
  "validate_params": true,
  "show_hidden_params": false,
  "schema_ignore_params": "show_hidden_params,validate_params,monochrome_logs,aws_queue,aws_image_prefix,wfversion,wf,process_label,sv_types,min_sv_length,max_sv_length,min_read_support,min_read_support_limit,guppy_map_threads,guppy_basemod_threads,guppy_no_align",
  "wf": {
      "name": "wf-human-variation",
      "template_version": "195cab5",
      "example_cmd": [
          "--mode snp",
          "--bam demo_data/chr6_chr20.bam",
          "--bed demo_data/chr6_chr20.bed",
          "--ref demo_data/chr6_chr20.fasta",
          "--model demo_data/ont_r104_e81_sup_g5015"
      ],
      "e2l_base_tag": "sha90e1c9ae1b7cd51548c4c9b66df424fe3c3a81a8",
      "e2l_snp_tag": "shaa215006ce90c5d2ce97352b548400c59a5d0f67a",
      "e2l_sv_tag": "sha04c8557a38dff4f529c859ad55cb2865782802a1",
      "e2l_guppy_tag": "sha7ee822e4c0349299d7049ea3faa9671d38c27cd8",
      "e2l_methyl_tag": "sha90e1c9ae1b7cd51548c4c9b66df424fe3c3a81a8"
  }

}' > params.json

Command exit status: 125

Command output: (empty)

Command error: 3b65ec22a9e9: Retrying in 3 seconds 3b65ec22a9e9: Retrying in 2 seconds 3b65ec22a9e9: Retrying in 1 second 1cadfcaee51c: Retrying in 20 seconds 1cadfcaee51c: Retrying in 19 seconds 1cadfcaee51c: Retrying in 18 seconds 1cadfcaee51c: Retrying in 17 seconds 1cadfcaee51c: Retrying in 16 seconds 1cadfcaee51c: Retrying in 15 seconds 1cadfcaee51c: Retrying in 14 seconds 1cadfcaee51c: Retrying in 13 seconds 1cadfcaee51c: Retrying in 12 seconds 1cadfcaee51c: Retrying in 11 seconds 1cadfcaee51c: Retrying in 10 seconds 1cadfcaee51c: Retrying in 9 seconds executor > local (5) [2a/307089] process > cram_cache (1) [ 0%] 0 of 1 [- ] process > snp:make_chunks - [- ] process > snp:pileup_variants - [- ] process > snp:aggregate_pileup_variants - [- ] process > snp:select_het_snps - [- ] process > snp:phase_contig - [- ] process > snp:get_qual_filter - [- ] process > snp:create_candidates - [- ] process > snp:evaluate_candidates - [- ] process > snp:aggregate_full_align_variants - [- ] process > snp:merge_pileup_and_full_vars - [- ] process > snp:aggregate_all_variants - [- ] process > snp:readStats - [15/f2ac31] process > snp:getVersions [ 0%] 0 of 1 [12/2aa0b7] process > snp:getParams [100%] 1 of 1, failed: 1 ✘ [- ] process > snp:vcfStats - [- ] process > snp:mosdepth - [- ] process > snp:makeReport - [- ] process > output_snp - [- ] process > sv:variantCall:filterBam - [- ] process > sv:variantCall:sniffles2 - [- ] process > sv:variantCall:mosdepth - [- ] process > sv:variantCall:filterCalls - [- ] process > sv:variantCall:sortVCF - [- ] process > sv:variantCall:indexVCF - [40/1b4683] process > sv:runReport:getVersions [ 0%] 0 of 1 [cc/dee338] process > sv:runReport:getParams [ 0%] 0 of 1 [- ] process > sv:runReport:report - [- ] process > output_sv - Error executing process > 'snp:getParams'

Caused by: Process snp:getParams terminated with an error exit status (125)

[SamStudio8]

Hi @jhxaz! In response to your first follow-up, I wanted to say thanks for your feedback and that we'll improve our documentation to provide examples for running each arm of the workflow separately for the next release. Your point is interesting as it highlights functionality that is missing from Nextflow itself. As this workflow has grown to support multiple subworkflows we have found Nextflow has little logic for argument handling, such as the grouping and parsing one might get from something like Python's argparse. Some of this has been alleviated a little by our adoption of nf-core's schema, but the complexity of the options for this pipeline has given me some food for thought as to how we might better offer --help to users in future.

In the mean time, thanks for trying the wf-human-variation workflow as per the README. I see your error:

Caused by: Process snp:getParams terminated with an error exit status (125)

Exit 125 indicates a problem with the docker run command. From your error log it looks as though there might be a connectivity issue with Dockerhub (1cadfcaee51c: Retrying in 9 seconds), but it might be useful to double check the end of your .nextflow.log file to see if there are any other errors that might indicate why Docker was unable to start the container. I would suggest contacting your IT department/system administrator to investigate this further as without more information this does not look like a problem with the workflow itself.

[jhxaz]

@SamStudio8 Thanks a lot, as you suggestion, I found it really was a problem with the docker run command, I delete the docker image, and run the wf-human-variation again, everything seems well. Now all the problems have been solved, and I can't wait the next release which could run each arm of the workflow separately for SNPs and SVs calling.

Caused by:
Process snp:getParams terminated with an error exit status (125)

[SamStudio8]

Hi @jhxaz! Thanks for letting me know. I'm glad that's working for you now. The README of the latest version of the workflow (v0.3.1) has better instructions on running the arms of the workflow separately and some better error messages when required parameters are not provided for different subworkflows. I hope this helps. Don't forget to nextflow pull epi2me-labs/wf-human-variation to obtain the latest version.