Process `wf_dorado:dorado (7)` terminated with an error exit status (139)

istvankleijn commented 2 weeks ago

Operating System

Other Linux (please specify below)

Other Linux

Ubuntu 20.04.4

Workflow Version

v1.2.2

Workflow Execution

EPI2ME Desktop (Local)

Other workflow execution

No response

EPI2ME Version

V5.1.14

CLI command run

No response

Workflow Execution - CLI Execution Profile

None

What happened?

I tried running the workflow for the first time with demultiplexing included. The first three dorado tasks completed successfully, but then it crashed with error code 139.

Relevant log output

N E X T F L O W  ~  version 23.04.2
Launching `/data/epi2me/workflows/epi2me-labs/wf-basecalling/main.nf` [redacted] DSL2 - revision: f8cf5c234d
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-basecalling v1.2.2
--------------------------------------------------------------------------------
Core Nextflow options
  runName        : [redacted]
  containerEngine: docker
  launchDir      : /data/epi2me/instances/wf-basecalling_01J0V2WT5KC82NTCGK39G48FSC
  workDir        : /data/epi2me/instances/wf-basecalling_01J0V2WT5KC82NTCGK39G48FSC/work
  projectDir     : /data/epi2me/workflows/epi2me-labs/wf-basecalling
  userName       : [redacted]
  profile        : standard
  configFiles    : /data/epi2me/workflows/epi2me-labs/wf-basecalling/nextflow.config
Input Options
  input          : [redacted]
  ref            : /data/GRCh38-no_alt.fasta
Output Options
  out_dir        : /data/epi2me/instances/wf-basecalling_01J0V2WT5KC82NTCGK39G48FSC/output
  sample_name    : [redacted]
Basecalling options
  basecaller_cfg : dna_r10.4.1_e8.2_400bps_hac@v4.3.0
  remora_cfg     : dna_r10.4.1_e8.2_400bps_hac@v4.3.0_5mCG_5hmCG@v1
  barcode_kit    : SQK-NBD114-24
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-basecalling for your analysis please cite:
* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-basecalling v1.2.2.
--------------------------------------------------------------------------------
[ae/36f433] Submitted process > wf_dorado:make_mmi
[0b/280507] Submitted process > cram_cache
[e7/6bcafd] Submitted process > getVersions
[d3/5ae58f] Submitted process > getParams
[ca/dc076c] Submitted process > wf_dorado:dorado (4)
[0b/dd5be9] Submitted process > wf_dorado:dorado (5)
[50/307605] Submitted process > wf_dorado:dorado (3)
[b9/340f7d] Submitted process > wf_dorado:dorado (7)
[af/306e30] Submitted process > wf_dorado:dorado (6)
ERROR ~ Error executing process > 'wf_dorado:dorado (7)'
Caused by:
  Process `wf_dorado:dorado (7)` terminated with an error exit status (139)
Command executed:
  # CW-2569: convert the pod5s contextually
  if [[ "false" == "true" && "pod5" == "fast5" ]]; then
      pod5 convert fast5 ./*.fast5 --output . --threads 8 --one-to-one ./
  fi

  # Run dorado on the new pod5s
  dorado basecaller         ${DRD_MODELS_PATH}/dna_r10.4.1_e8.2_400bps_hac@v4.3.0         .         --modified-bases-models ${DRD_MODELS_PATH}/dna_r10.4.1_e8.2_400bps_hac@v4.3.0_5mCG_5hmCG@v1                           --kit-name SQK-NBD114-24                  --device cuda:all | samtools view --no-PG -b -o 6.ubam -

  # CW-2569: delete the pod5s, if emit not required.
  if [[ "false" == "true" && "pod5" == "fast5" ]]; then
      rm -r converted/
  fi
Command exit status:
  139
Command output:
  (empty)
Command error:
  [2024-06-20 16:12:49.860] [info] Running: "basecaller" "/models/dna_r10.4.1_e8.2_400bps_hac@v4.3.0" "." "--modified-bases-models" "/models/dna_r10.4.1_e8.2_400bps_hac@v4.3.0_5mCG_5hmCG@v1" "--kit-name" "SQK-NBD114-24" "--device" "cuda:all"
  [2024-06-20 16:12:49.884] [info] Normalised: chunksize 10000 -> 9996
  [2024-06-20 16:12:49.884] [info] Normalised: overlap 500 -> 498
  [2024-06-20 16:12:49.884] [info] > Creating basecall pipeline
  [2024-06-20 16:14:31.202] [info] cuda:0 using chunk size 9996, batch size 3200
  [2024-06-20 16:14:32.438] [info] cuda:0 using chunk size 4998, batch size 3200
  [W::bgzf_read_block] EOF marker is absent. The input may be truncated
  .command.sh: line 8:    29 Segmentation fault      (core dumped) dorado basecaller ${DRD_MODELS_PATH}/dna_r10.4.1_e8.2_400bps_hac@v4.3.0 . --modified-bases-models ${DRD_MODELS_PATH}/dna_r10.4.1_e8.2_400bps_hac@v4.3.0_5mCG_5hmCG@v1 --kit-name SQK-NBD114-24 --device cuda:all
          30 Done                    | samtools view --no-PG -b -o 6.ubam -
Work dir:
  /data/epi2me/instances/wf-basecalling_01J0V2WT5KC82NTCGK39G48FSC/work/b9/340f7d87b8d68545a23da984bf37ee
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
 -- Check '/data/epi2me/instances/wf-basecalling_01J0V2WT5KC82NTCGK39G48FSC/nextflow.log' file for details
WARN: Killing running tasks (1)

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

istvankleijn commented 2 weeks ago

This is probably the same problem as https://github.com/nanoporetech/dorado/issues/860 (mine are short reads as well), so it looks like it may have been fixed in dorado@v0.7.2. Any chance for a quick container update?

SamStudio8 commented 2 weeks ago

Hi @istvankleijn, sorry for the slow reply. Yes I suspect this may be due to the pre-existing issue for Dorado 0.7.1, we have a release with 0.7.2 in the works and it should be out at the start of next week.

epi2me-labs / wf-basecalling