report shows first chunk only

[istvankleijn]

Operating System

Ubuntu 22.04

Other Linux

No response

Workflow Version

wf-basecalling v1.1.5

Workflow Execution

EPI2ME Desktop application

EPI2ME Version

EPI2ME V5.1.9

CLI command run

No response

Workflow Execution - CLI Execution Profile

None

What happened?

At completion of the workflow, the basecalling report only shows the results from the first chunk of reads (100k reads from 25 files of 4000 reads each). These are not very representative for the whole experiment, because initial fast basecalling was performed during sequencing and the first chunk consists of failed reads only.

I would expect to see the read quality and read length histograms for all reads when the workflow finishes. I suppose it would also be nice if the report were generated on-the-fly rather than only at completion.

Relevant log output

N E X T F L O W  ~  version 23.04.2
Launching `/data/epi2me/workflows/epi2me-labs/wf-basecalling/main.nf` [redacted] DSL2 - revision: 6d842f59e6
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-basecalling v1.1.5
--------------------------------------------------------------------------------
Core Nextflow options
  runName        : [redacted]
  containerEngine: docker
  launchDir      : /data/epi2me/instances/wf-basecalling_01HPPWD4NTTC10YCS43PZ1Y4X8
  workDir        : /data/epi2me/instances/wf-basecalling_01HPPWD4NTTC10YCS43PZ1Y4X8/work
  projectDir     : /data/epi2me/workflows/epi2me-labs/wf-basecalling
  userName       : grid
  profile        : standard
  configFiles    : /data/epi2me/workflows/epi2me-labs/wf-basecalling/nextflow.config
Input Options
  input          : [redacted]
  ref            : /data/GRCh38-no_alt.fasta
Output Options
  out_dir        : /data/epi2me/instances/wf-basecalling_01HPPWD4NTTC10YCS43PZ1Y4X8/output
  sample_name    : [redacted]
  output_bam     : true
Basecalling options
  basecaller_cfg : dna_r10.4.1_e8.2_400bps_hac@v4.3.0
  remora_cfg     : dna_r10.4.1_e8.2_400bps_hac@v4.3.0_5mCG_5hmCG@v1
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-basecalling for your analysis please cite:
* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-basecalling v1.1.5.
--------------------------------------------------------------------------------
[f4/2387c4] Submitted process > getParams
[75/48d8fc] Submitted process > getVersions
[09/494947] Submitted process > wf_dorado:make_mmi
[0b/754d2d] Submitted process > cram_cache
[a1/e23631] Submitted process > wf_dorado:dorado (1)
[e2/913ef8] Submitted process > wf_dorado:dorado (2)
[7b/4fa78f] Submitted process > wf_dorado:dorado (3)
[d3/992461] Submitted process > wf_dorado:dorado (4)
[2a/cc153c] Submitted process > wf_dorado:dorado (5)
[70/a68063] Submitted process > wf_dorado:dorado (7)
[9d/469388] Submitted process > wf_dorado:dorado (6)
[d1/e2d963] Submitted process > wf_dorado:dorado (9)
[fb/9bbf16] Submitted process > wf_dorado:dorado (8)
[1d/fe9085] Submitted process > wf_dorado:dorado (10)
[95/80c526] Submitted process > wf_dorado:dorado (11)
[aa/55e517] Submitted process > wf_dorado:dorado (13)
[26/1e58a5] Submitted process > wf_dorado:dorado (12)
[ed/cf52e0] Submitted process > wf_dorado:dorado (14)
[60/c14d79] Submitted process > wf_dorado:dorado (15)
[e8/ef374f] Submitted process > wf_dorado:dorado (16)
[db/c9c7c2] Submitted process > wf_dorado:dorado (17)
[60/116e5e] Submitted process > wf_dorado:dorado (18)
[f0/151996] Submitted process > wf_dorado:dorado (19)
[17/c1185b] Submitted process > wf_dorado:dorado (20)
[62/7ef65c] Submitted process > wf_dorado:dorado (23)
[3d/0c9d79] Submitted process > wf_dorado:dorado (21)
[ff/0542d4] Submitted process > wf_dorado:dorado (24)
[11/48462f] Submitted process > wf_dorado:dorado (22)
[ea/a99379] Submitted process > wf_dorado:align_and_qsFilter (1)
[31/06336d] Submitted process > wf_dorado:align_and_qsFilter (2)
[2e/a53d29] Submitted process > bamstats (1)
[3a/8968ad] Submitted process > progressive_stats (1)
[13/0e8986] Submitted process > makeReport (1)
[ce/80dd58] Submitted process > output_last
[51/0316f6] Submitted process > wf_dorado:align_and_qsFilter (3)
[cf/f5a9c0] Submitted process > bamstats (2)
[fa/2138cd] Submitted process > progressive_stats (2)
[b5/461516] Submitted process > wf_dorado:align_and_qsFilter (4)
[d8/8e22d2] Submitted process > bamstats (3)
[3b/b7ef5b] Submitted process > progressive_stats (3)
[bf/4fdcc6] Submitted process > wf_dorado:align_and_qsFilter (5)
[b8/078ca0] Submitted process > bamstats (4)
[5b/ca0552] Submitted process > progressive_stats (4)
[44/e20aae] Submitted process > wf_dorado:align_and_qsFilter (6)
[51/f713b7] Submitted process > bamstats (5)
[94/7a279f] Submitted process > progressive_stats (5)
[a2/83e9a7] Submitted process > wf_dorado:align_and_qsFilter (7)
[23/537e1d] Submitted process > bamstats (6)
[64/aa6d58] Submitted process > progressive_stats (6)
[fe/579fdd] Submitted process > wf_dorado:align_and_qsFilter (8)
[2b/96829b] Submitted process > bamstats (7)
[e9/6ebf1b] Submitted process > progressive_stats (7)
[52/5f6c88] Submitted process > wf_dorado:align_and_qsFilter (9)
[14/a9f15a] Submitted process > bamstats (8)
[6a/a1d9fc] Submitted process > progressive_stats (8)
[18/d865a2] Submitted process > wf_dorado:align_and_qsFilter (10)
[85/cbf9f5] Submitted process > bamstats (9)
[ef/aecc34] Submitted process > progressive_stats (9)
[99/28e19a] Submitted process > wf_dorado:align_and_qsFilter (11)
[71/66c4d6] Submitted process > bamstats (10)
[31/30d800] Submitted process > progressive_stats (10)
[2d/e5b819] Submitted process > wf_dorado:align_and_qsFilter (12)
[e5/8e4a19] Submitted process > bamstats (11)
[02/6e1111] Submitted process > progressive_stats (11)
[c4/69db68] Submitted process > wf_dorado:align_and_qsFilter (13)
[ed/7de2e9] Submitted process > bamstats (12)
[e2/0d5ed9] Submitted process > progressive_stats (12)
[25/ba5904] Submitted process > wf_dorado:align_and_qsFilter (14)
[28/d957a8] Submitted process > bamstats (13)
[0d/95092f] Submitted process > progressive_stats (13)
[59/8ceabe] Submitted process > wf_dorado:align_and_qsFilter (15)
[03/11808c] Submitted process > bamstats (14)
[cf/e8f7ec] Submitted process > progressive_stats (14)
[38/7ac221] Submitted process > wf_dorado:align_and_qsFilter (16)
[f6/54a447] Submitted process > bamstats (15)
[a8/d4bcb0] Submitted process > progressive_stats (15)
[33/f5ec7e] Submitted process > wf_dorado:align_and_qsFilter (17)
[80/78a8f5] Submitted process > bamstats (16)
[e2/558ae8] Submitted process > progressive_stats (16)
[3f/81e887] Submitted process > wf_dorado:align_and_qsFilter (18)
[0c/9e6b1c] Submitted process > bamstats (17)
[e6/63efd6] Submitted process > progressive_stats (17)
[c3/1f412f] Submitted process > wf_dorado:align_and_qsFilter (19)
[f6/2661ff] Submitted process > bamstats (18)
[d6/23e15d] Submitted process > progressive_stats (18)
[3a/262f80] Submitted process > wf_dorado:align_and_qsFilter (20)
[e0/338fc0] Submitted process > bamstats (19)
[1d/c9e2dd] Submitted process > progressive_stats (19)
[06/97b789] Submitted process > wf_dorado:align_and_qsFilter (21)
[06/8f776e] Submitted process > bamstats (20)
[40/de823b] Submitted process > progressive_stats (20)
[6d/abf331] Submitted process > wf_dorado:align_and_qsFilter (22)
[d8/5d6643] Submitted process > bamstats (21)
[52/422d0b] Submitted process > progressive_stats (21)
[21/48c308] Submitted process > wf_dorado:align_and_qsFilter (23)
[f0/edb2c8] Submitted process > bamstats (22)
[dd/8d8abf] Submitted process > progressive_stats (22)
[0d/4c7c82] Submitted process > wf_dorado:align_and_qsFilter (24)
[1e/0aef75] Submitted process > bamstats (23)
[d1/aa0db0] Submitted process > progressive_stats (23)
[cf/df86cd] Submitted process > bamstats (24)
[b6/bae571] Submitted process > wf_dorado:merge_pass_calls
[03/7a737d] Submitted process > wf_dorado:merge_fail_calls
[fa/aa3ca5] Submitted process > progressive_stats (24)
[85/3e8af2] Submitted process > output_stream (2)
[d6/a01d34] Submitted process > output_stream (3)
[e0/7fd818] Submitted process > output_stream (4)
[4f/5a5715] Submitted process > output_stream (1)
[5f/dc1c9e] Submitted process > output_stream (6)
[eb/195c1f] Submitted process > output_stream (7)
[47/166798] Submitted process > output_stream (5)
[66/8aadb2] Submitted process > output_stream (8)
[7c/695d61] Submitted process > output_stream (9)
Waiting for file transfers to complete (3 files)

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

[SamStudio8]

I suppose it would also be nice if the report were generated on-the-fly rather than only at completion

Thanks for reporting this @istvankleijn, it is our intention that reports are generated on the fly, it looks like our dynamic reporting may not be working. We'll take a look at this for the next release.

[istvankleijn]

Just to say that this seems to be fixed in v1.1.7, thanks! The final report includes all the data and the report is updated after each alignment process. There is no reporting on-the-fly during basecalling itself, so for the last few hours of the workflow running, the GPU is idle while alignment and reporting occurs. This way I can start a second iteration of wf-basecalling if I need to without the two iterations interfering with each other on the GPU, so I hope this is intentional :)