html report generation errors out when there are 0 classified reads

osilander commented 7 months ago

Operating System

CentOS 7

Other Linux

No response

Workflow Version

wf-16s v1.1.0-g137d59e

Workflow Execution

Command line

EPI2ME Version

No response

CLI command run

# AG1268-test1 directory contains barcode directories
nextflow run epi2me-labs/wf-16s --fastq /nesi/nobackup/uoa03387/AG1268-test/AG1268-test1 -profile singularity

Workflow Execution - CLI Execution Profile

singularity

What happened?

The pipeline generates most output files (.tsv, params.json, versions.txt,execution/) but fails to generate the html report if any barcode contains zero classified reads (even if the barcode itself has reads). In addition, if a barcode has zero classified reads, there is no column in the .tsv for that barcode. In the case below, this is barcode 14. The pipeline works fine for groups of barcodes in which each barcode has at least one classified read.

ERROR ~ Error executing process > 'minimap_pipeline:makeReport (1)'

Caused by:
  Process `minimap_pipeline:makeReport (1)` terminated with an error exit status (1)

Relevant log output

executor >  local (27)
[7f/7ce854] process > fastcat (3)                    [100%] 10 of 10 ✔
[skipped  ] process > prepare_databases:download_... [100%] 1 of 1, stored: 1 ✔
[skipped  ] process > prepare_databases:download_... [100%] 1 of 1, stored: 1 ✔
[18/459a90] process > minimap_pipeline:run_common... [100%] 1 of 1 ✔
[f2/a80798] process > minimap_pipeline:run_common... [100%] 1 of 1 ✔
[7e/0b7c19] process > minimap_pipeline:minimap (b... [100%] 10 of 10 ✔
[4b/471d04] process > minimap_pipeline:createAbun... [100%] 1 of 1 ✔
[f9/a77948] process > minimap_pipeline:makeReport... [100%] 1 of 1, failed: 1 ✘
[29/22b418] process > minimap_pipeline:output_res... [100%] 3 of 3 ✔
ERROR ~ Error executing process > 'minimap_pipeline:makeReport (1)'

Caused by:
  Process `minimap_pipeline:makeReport (1)` terminated with an error exit status (1)

Command executed:

  workflow-glue report         "wf-16s-report.html"         --workflow_name wf-16s         --versions versions         --params params.json         --read_stats read_stats/*         --lineages lineages         --abundance_table "abundance_table_genus.tsv"         --taxonomic_rank "G"         --pipeline "minimap2"         --abundance_threshold "1"        --n_taxa_barplot "9"

Command exit status:
  1

Command output:
  (empty)

Command error:
  [00:14:15 - matplotlib] Matplotlib created a temporary cache directory at /dev/shm/jobs/43965414/matplotlib-_ahlkudb because the default path (/home/osilande/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
  [00:14:16 - matplotlib.font_manager] generated new fontManager
  /home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/__init__.py:30: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    logger.warn(f"Could not load {name} due to missing module {e.name}")
  [00:14:17 - workflow_glue] Could not load abundance_tables due to missing module anytree
  [00:14:17 - workflow_glue] Starting entrypoint.
  [00:14:21 - workflow_glue.Report    ] Sample barcode18.
  [00:14:21 - EChrtPlotr] Skipping axis label fixing
  [00:14:21 - workflow_glue.Report    ] Sample barcode15.
  [00:14:21 - EChrtPlotr] Skipping axis label fixing
  [00:14:21 - workflow_glue.Report    ] Sample barcode17.
  [00:14:21 - EChrtPlotr] Skipping axis label fixing
  [00:14:21 - workflow_glue.Report    ] Sample barcode12.
  [00:14:21 - EChrtPlotr] Skipping axis label fixing
  [00:14:21 - workflow_glue.Report    ] Sample barcode10.
  [00:14:21 - EChrtPlotr] Skipping axis label fixing
  [00:14:21 - workflow_glue.Report    ] Sample barcode14.
  Traceback (most recent call last):
    File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow-glue", line 7, in <module>
      cli()
    File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/__init__.py", line 72, in cli
      args.func(args)
    File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/report.py", line 178, in main
      plt = ezc.sunburst(
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/plots/sunburst.py", line 62, in sunburst
      if data[0].get('value'):
  IndexError: list index out of range

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

nggvs commented 7 months ago

Hi @osilander , Thank you very much for using the pipeline, I'll take a look, if all the reads for a barcode are unclassified shouldn't be failing (is supposed to be the case01/barcode03 in the test_data), so might be something else, but I'll take a look to reproduce it just in case. As it seems that you have looked the files, could you confirm me that for barcode14, contains unclassified reads or no reads at all? Could you paste the content of the file: work/4b/471d04..../lineages/barcode14 ?

Thank you very much in advance

osilander commented 7 months ago

Hi, Thanks for your quick response. I've since run the pipeline a bunch of different times so don't even have that work folder anymore. However, I now can't get the issue to replicate so please close it :)

nggvs commented 7 months ago

Ok, thank you! Just open a new issue if you find something else. Thank you for using the workflow!

epi2me-labs / wf-16s