search

epi2me-labs / wf-basecalling

Other
33 stars 13 forks source link

demultiplexing fails with CRAM #43

Open istvankleijn opened 3 months ago

istvankleijn commented 3 months ago

Operating System

Other Linux (please specify below)

Other Linux

Ubuntu 20.04.4

Workflow Version

v1.3.0

Workflow Execution

EPI2ME Desktop (Local)

Other workflow execution

No response

EPI2ME Version

V5.1.14

CLI command run

No response

Workflow Execution - CLI Execution Profile

None

What happened?

Running the workflow on a barcoded experiment, I encountered an error at the demultiplexing step, so I encounter: Missing output file(s) `demuxed/*.bam` expected by process `wf_dorado:split_calls` It seems something is wrong with the reference (see log). With previous workflow versions, I used output_bam = true so I am now redoing this to see if that was the problem.

Relevant log output

N E X T F L O W  ~  version 23.04.2
Launching `/data/epi2me/workflows/epi2me-labs/wf-basecalling/main.nf` [redacted] DSL2 - revision: 13c8f247e4
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-basecalling v1.3.0
--------------------------------------------------------------------------------
Core Nextflow options
  runName        : [redacted]
  containerEngine: docker
  launchDir      : /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF
  workDir        : /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/work
  projectDir     : /data/epi2me/workflows/epi2me-labs/wf-basecalling
  userName       : [redacted]
  profile        : standard
  configFiles    : /data/epi2me/workflows/epi2me-labs/wf-basecalling/nextflow.config
Input Options
  input          : [redacted]
  ref            : /data/GRCh38-no_alt.fasta
Output Options
  out_dir        : /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/output
  sample_name    : [redacted]
Basecalling options
  basecaller_cfg : dna_r10.4.1_e8.2_400bps_hac@v5.0.0
  remora_cfg     : dna_r10.4.1_e8.2_400bps_hac@v5.0.0_5mCG_5hmCG@v1
  barcode_kit    : SQK-NBD114-24
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-basecalling for your analysis please cite:
* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-basecalling v1.3.0.
--------------------------------------------------------------------------------
[7f/699eca] Submitted process > wf_dorado:make_mmi
[2c/f5c263] Submitted process > getParams
[67/f4b5ba] Submitted process > cram_cache
[1c/ef7509] Submitted process > getVersions
[75/11ef77] Submitted process > wf_dorado:dorado (3)
[fc/8348ac] Submitted process > wf_dorado:dorado (2)
[96/93e70f] Submitted process > wf_dorado:dorado (4)
[8c/b13617] Submitted process > wf_dorado:dorado (1)
[bf/2fbc68] Submitted process > wf_dorado:dorado (6)
[5a/393afe] Submitted process > wf_dorado:dorado (7)
[8a/8b6be8] Submitted process > wf_dorado:dorado (9)
[d1/e30a78] Submitted process > wf_dorado:dorado (11)
[80/bf8857] Submitted process > wf_dorado:dorado (12)
[30/1d0070] Submitted process > wf_dorado:dorado (8)
[bc/d08205] Submitted process > wf_dorado:dorado (10)
[50/f8841f] Submitted process > wf_dorado:dorado (5)
WARN: Killing running tasks (1)
N E X T F L O W  ~  version 23.04.2
Launching `/data/epi2me/workflows/epi2me-labs/wf-basecalling/main.nf` [romantic_bhaskara] DSL2 - revision: 13c8f247e4
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-basecalling v1.3.0
--------------------------------------------------------------------------------
Core Nextflow options
  runName        : romantic_bhaskara
  containerEngine: docker
  launchDir      : /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF
  workDir        : /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/work
  projectDir     : /data/epi2me/workflows/epi2me-labs/wf-basecalling
  userName       : [redacted]
  profile        : standard
  configFiles    : /data/epi2me/workflows/epi2me-labs/wf-basecalling/nextflow.config
Input Options
  input          : [redacted]
  ref            : /data/GRCh38-no_alt.fasta
Output Options
  out_dir        : /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/output
  sample_name    : [redacted]
Basecalling options
  basecaller_cfg : dna_r10.4.1_e8.2_400bps_hac@v5.0.0
  remora_cfg     : dna_r10.4.1_e8.2_400bps_hac@v5.0.0_5mCG_5hmCG@v1
  barcode_kit    : SQK-NBD114-24
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-basecalling for your analysis please cite:
* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-basecalling v1.3.0.
--------------------------------------------------------------------------------
[1c/ef7509] Cached process > getVersions
[67/f4b5ba] Cached process > cram_cache
[2c/f5c263] Cached process > getParams
[7f/699eca] Cached process > wf_dorado:make_mmi
[bf/2fbc68] Cached process > wf_dorado:dorado (6)
[96/93e70f] Cached process > wf_dorado:dorado (4)
[8c/b13617] Cached process > wf_dorado:dorado (1)
[5a/393afe] Cached process > wf_dorado:dorado (7)
[fc/8348ac] Cached process > wf_dorado:dorado (2)
[75/11ef77] Cached process > wf_dorado:dorado (3)
[30/1d0070] Cached process > wf_dorado:dorado (8)
[8a/8b6be8] Cached process > wf_dorado:dorado (9)
[bc/d08205] Cached process > wf_dorado:dorado (10)
[d1/e30a78] Cached process > wf_dorado:dorado (11)
[80/bf8857] Cached process > wf_dorado:dorado (12)
[c9/3b54bc] Submitted process > wf_dorado:dorado (5)
[66/2b4894] Submitted process > wf_dorado:dorado (13)
[4c/42603f] Submitted process > wf_dorado:dorado (14)
[e0/772385] Submitted process > wf_dorado:dorado (16)
[e1/446e5a] Submitted process > wf_dorado:dorado (15)
[5a/a3a1d5] Submitted process > wf_dorado:align_and_qsFilter (7)
[f3/beee7e] Submitted process > wf_dorado:align_and_qsFilter (8)
[67/33b033] Submitted process > bamstats (1)
[fe/b90df8] Submitted process > progressive_stats (1)
[f4/7bc32d] Submitted process > makeReport (1)
[38/de27bf] Submitted process > wf_dorado:align_and_qsFilter (6)
[3b/4380d4] Submitted process > bamstats (2)
[70/b4ff2f] Submitted process > progressive_stats (2)
[4c/d37115] Submitted process > makeReport (2)
[27/646817] Submitted process > wf_dorado:align_and_qsFilter (3)
[d3/5c1fbe] Submitted process > bamstats (3)
[92/d5f5d2] Submitted process > progressive_stats (3)
[51/05fedd] Submitted process > makeReport (3)
[59/938f74] Submitted process > wf_dorado:align_and_qsFilter (9)
[dd/dc0baf] Submitted process > bamstats (4)
[8e/bf44f6] Submitted process > progressive_stats (4)
[76/7bda01] Submitted process > makeReport (4)
[ac/f03d33] Submitted process > wf_dorado:align_and_qsFilter (4)
[69/10e205] Submitted process > bamstats (5)
[84/0cf970] Submitted process > progressive_stats (5)
[d7/127cc5] Submitted process > makeReport (5)
[ff/a5a314] Submitted process > wf_dorado:align_and_qsFilter (2)
[03/ffd602] Submitted process > bamstats (6)
[bf/89e7ab] Submitted process > progressive_stats (6)
[15/063fa6] Submitted process > makeReport (6)
[bd/804bb6] Submitted process > wf_dorado:align_and_qsFilter (1)
[3e/10268d] Submitted process > bamstats (7)
[5d/784c0d] Submitted process > progressive_stats (7)
[e1/3da906] Submitted process > makeReport (7)
[89/6acae2] Submitted process > wf_dorado:align_and_qsFilter (11)
[07/a49a86] Submitted process > bamstats (8)
[60/1b29c0] Submitted process > progressive_stats (8)
[0a/42b866] Submitted process > makeReport (8)
[92/4e2fda] Submitted process > wf_dorado:align_and_qsFilter (5)
[dc/b3c1d6] Submitted process > bamstats (9)
[c5/8443fc] Submitted process > progressive_stats (9)
[44/39c781] Submitted process > makeReport (9)
[4d/996f77] Submitted process > wf_dorado:align_and_qsFilter (10)
[3a/59d77d] Submitted process > bamstats (10)
[81/964ddb] Submitted process > progressive_stats (10)
[0e/50b23a] Submitted process > makeReport (10)
[e9/026240] Submitted process > wf_dorado:dorado (18)
[73/1d97e7] Submitted process > bamstats (11)
[c2/ce30a8] Submitted process > progressive_stats (11)
[e3/899fe1] Submitted process > makeReport (11)
[c8/d57ec6] Submitted process > wf_dorado:dorado (19)
[06/177d3a] Submitted process > wf_dorado:dorado (20)
[68/359311] Submitted process > wf_dorado:dorado (17)
[5f/9c7c7e] Submitted process > wf_dorado:dorado (21)
[8f/824cdd] Submitted process > wf_dorado:dorado (22)
[dd/915082] Submitted process > wf_dorado:dorado (24)
[f9/fad52a] Submitted process > wf_dorado:dorado (23)
[23/94d3bd] Submitted process > wf_dorado:dorado (25)
[a3/7c8494] Submitted process > wf_dorado:dorado (27)
[21/267b4a] Submitted process > wf_dorado:dorado (28)
[12/236697] Submitted process > wf_dorado:dorado (29)
[ee/65a68c] Submitted process > wf_dorado:dorado (30)
[a3/fce197] Submitted process > wf_dorado:dorado (26)
[3f/0b1dbf] Submitted process > wf_dorado:dorado (31)
[9a/19faed] Submitted process > wf_dorado:align_and_qsFilter (12)
[da/743843] Submitted process > wf_dorado:align_and_qsFilter (13)
[2e/2c8365] Submitted process > bamstats (12)
[1c/2a4e32] Submitted process > progressive_stats (12)
[8c/2d2390] Submitted process > makeReport (12)
[05/2ac0aa] Submitted process > wf_dorado:align_and_qsFilter (14)
[1f/7a08e0] Submitted process > bamstats (13)
[d4/4e6637] Submitted process > progressive_stats (13)
[86/0447c7] Submitted process > makeReport (13)
[57/e92b80] Submitted process > wf_dorado:align_and_qsFilter (15)
[ea/222693] Submitted process > bamstats (14)
[cc/b7170a] Submitted process > progressive_stats (14)
[6e/ce9e16] Submitted process > makeReport (14)
[1e/3a55f8] Submitted process > wf_dorado:align_and_qsFilter (16)
[05/c5d516] Submitted process > bamstats (15)
[f7/f9aa81] Submitted process > progressive_stats (15)
[33/cdf0f9] Submitted process > makeReport (15)
[b8/f7b650] Submitted process > wf_dorado:align_and_qsFilter (17)
[0a/79296a] Submitted process > bamstats (16)
[51/375e31] Submitted process > progressive_stats (16)
[91/10641e] Submitted process > makeReport (16)
[f6/19a924] Submitted process > wf_dorado:align_and_qsFilter (18)
[dd/0f65d0] Submitted process > bamstats (17)
[69/ead877] Submitted process > progressive_stats (17)
[2e/b168b2] Submitted process > makeReport (17)
[c5/a0123d] Submitted process > wf_dorado:align_and_qsFilter (19)
[04/395278] Submitted process > bamstats (18)
[de/2c6c8b] Submitted process > progressive_stats (18)
[eb/19efb8] Submitted process > makeReport (18)
[1e/790bef] Submitted process > wf_dorado:align_and_qsFilter (20)
[87/ffd95b] Submitted process > bamstats (19)
[1e/679008] Submitted process > progressive_stats (19)
[73/66774f] Submitted process > makeReport (19)
[b5/108b3a] Submitted process > wf_dorado:align_and_qsFilter (21)
[eb/2dc9db] Submitted process > bamstats (20)
[45/c08d82] Submitted process > progressive_stats (20)
[6e/a6a387] Submitted process > makeReport (20)
[3e/5a8408] Submitted process > wf_dorado:align_and_qsFilter (22)
[58/8ddb83] Submitted process > bamstats (21)
[1e/06695d] Submitted process > progressive_stats (21)
[80/af211b] Submitted process > makeReport (21)
[f3/600c44] Submitted process > wf_dorado:align_and_qsFilter (23)
[f7/89f061] Submitted process > bamstats (22)
[1b/6f3b49] Submitted process > progressive_stats (22)
[38/b5c44f] Submitted process > makeReport (22)
[8b/761d6e] Submitted process > wf_dorado:align_and_qsFilter (24)
[cc/82f955] Submitted process > bamstats (23)
[af/4aa0f9] Submitted process > progressive_stats (23)
[32/e5bc9e] Submitted process > makeReport (23)
[47/6e1d86] Submitted process > wf_dorado:align_and_qsFilter (25)
[55/9dc6eb] Submitted process > bamstats (24)
[b1/deafd8] Submitted process > progressive_stats (24)
[32/67891c] Submitted process > makeReport (24)
[cb/00fd3e] Submitted process > wf_dorado:align_and_qsFilter (26)
[ab/792f7f] Submitted process > bamstats (25)
[08/df58d7] Submitted process > progressive_stats (25)
[ef/72b48e] Submitted process > makeReport (25)
[b4/df4f0f] Submitted process > wf_dorado:align_and_qsFilter (27)
[c3/ac6d94] Submitted process > bamstats (26)
[7d/de864d] Submitted process > progressive_stats (26)
[cf/302ca8] Submitted process > makeReport (26)
[95/f92644] Submitted process > wf_dorado:align_and_qsFilter (28)
[54/d19349] Submitted process > bamstats (27)
[0d/e1b780] Submitted process > progressive_stats (27)
[77/3790e5] Submitted process > makeReport (27)
[af/5bc316] Submitted process > wf_dorado:align_and_qsFilter (29)
[3c/c2f805] Submitted process > bamstats (28)
[7e/b39159] Submitted process > progressive_stats (28)
[ab/fe63fc] Submitted process > makeReport (28)
[58/b991a6] Submitted process > wf_dorado:align_and_qsFilter (30)
[2b/090aba] Submitted process > bamstats (29)
[8e/21303c] Submitted process > progressive_stats (29)
[7e/b10eab] Submitted process > makeReport (29)
[13/415a4b] Submitted process > wf_dorado:align_and_qsFilter (31)
[ea/dfdd53] Submitted process > bamstats (30)
[9f/3789e3] Submitted process > progressive_stats (30)
[58/69ac31] Submitted process > makeReport (30)
[da/8c05f9] Submitted process > bamstats (31)
[4b/fb5c3e] Submitted process > wf_dorado:merge_pass_calls
[94/9f183b] Submitted process > wf_dorado:merge_fail_calls
[f1/5ffd85] Submitted process > progressive_stats (31)
[4c/5e86b6] Submitted process > makeReport (31)
[d7/c0fdbb] Submitted process > output_last
[0d/39f6ce] Submitted process > output_stream (8)
[d8/8d6cb9] Submitted process > output_stream (3)
[49/2220b3] Submitted process > output_stream (1)
[11/4ae751] Submitted process > output_stream (6)
[14/24208c] Submitted process > output_stream (4)
[df/d9c6da] Submitted process > output_stream (5)
[27/03d832] Submitted process > output_stream (9)
[e7/a15f99] Submitted process > output_stream (7)
[6c/c72cd7] Submitted process > output_stream (2)
[d9/6166dc] Submitted process > wf_dorado:split_calls
ERROR ~ Error executing process > 'wf_dorado:split_calls'
Caused by:
  Missing output file(s) `demuxed/*.bam` expected by process `wf_dorado:split_calls`
Command executed:
  dorado demux --output-dir demuxed --no-classify [redacted].pass.cram
Command exit status:
  0
Command output:
  (empty)
Command error:
  [2024-07-11 01:40:55.869] [info] Running: "demux" "--output-dir" "demuxed" "--no-classify" "[redacted].pass.cram"
  [2024-07-11 01:40:55.869] [info] num input files: 1
  [2024-07-11 01:40:55.871] [info] > starting barcode demuxing
  [W::find_file_url] Failed to open reference "https://www.ebi.ac.uk/ena/cram/md5/6aef897c3d6ff0c78aff06ac189178dd": Protocol not supported
  /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/work/4b/fb5c3ed3de90cf83e83f3d47522808/GRCh38-no_alt.fasta: No such file or directory
  [E::refs_load_fai] Failed to open reference file '/data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/work/4b/fb5c3ed3de90cf83e83f3d47522808/GRCh38-no_alt.fasta'
  [E::cram_get_ref] Failed to populate reference for id 0
  [E::cram_decode_slice] Unable to fetch reference #0:10001-72045

  [E::cram_next_slice] Failure to decode slice

  > Output records written: 0

  > Output records written: 0
  [2024-07-11 01:40:55.971] [info] > finished barcode demuxing
Work dir:
  /data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/work/d9/6166dc61a8ac65bf66a23cdaa1c0eb
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
 -- Check '/data/epi2me/instances/wf-basecalling_01J2E81B7YTCB9C5WQGB2Q5RTF/nextflow.log' file for details

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

istvankleijn commented 3 months ago

The workflow indeed completed successfully when I set output_bam = true.

RenzoTale88 commented 3 months ago

@istvankleijn thanks for letting us know of this issue, we will look into it.