Closed yanchunzhang closed 7 months ago
Also fails during report generating step (abundance tables generated fine) but as far as I can tell only when using a sample sheet.
executor > slurm (181)
[c5/827031] process > validate_sample_sheet [100%] 1 of 1, cached: 1 ✔
[b6/89f166] process > fastcat (85) [100%] 89 of 89 ✔
[skipped ] process > prepare_databases:download_unpack_taxonomy [100%] 1 of 1, stored: 1 ✔
[skipped ] process > prepare_databases:download_reference_ref2taxid [100%] 1 of 1, stored: 1 ✔
[a3/ace151] process > minimap_pipeline:run_common:getVersions [100%] 1 of 1, cached: 1 ✔
[b9/13634a] process > minimap_pipeline:run_common:getParams [100%] 1 of 1, cached: 1 ✔
[25/baee67] process > minimap_pipeline:minimap (119C) [100%] 89 of 89 ✔
[67/a62a4f] process > minimap_pipeline:createAbundanceTables [100%] 1 of 1 ✔
[1f/c3f2e4] process > minimap_pipeline:makeReport (1) [100%] 1 of 1, failed: 1 ✘
[33/e21ba8] process > minimap_pipeline:output_results (3) [100%] 3 of 3, cached: 2 ✔
ERROR ~ Error executing process > 'minimap_pipeline:makeReport (1)'
Caused by:
Process `minimap_pipeline:makeReport (1)` terminated with an error exit status (1)
Command executed:
workflow-glue report "wf-16s-report.html" --workflow_name wf-16s --versions versions --params params.json --read_stats read_stats/* --lineages lineages --abundance_table "abundance_table_genus.tsv" --taxonomic_rank "G" --pipeline "minimap2" --abundance_threshold "1" --n_taxa_barplot "9"
Command exit status:
1
Command output:
(empty)
Command error:
[09:04:50 - matplotlib] Matplotlib created a temporary cache directory at /dev/shm/jobs/44149314/matplotlib-446td809 because the default path (/home/osilande/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
[09:04:51 - matplotlib.font_manager] generated new fontManager
/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/__init__.py:30: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn(f"Could not load {name} due to missing module {e.name}")
[09:04:52 - workflow_glue] Could not load abundance_tables due to missing module anytree
[09:04:52 - workflow_glue] Starting entrypoint.
Traceback (most recent call last):
File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow-glue", line 7, in <module>
cli()
File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/__init__.py", line 72, in cli
args.func(args)
File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/report.py", line 121, in main
sample_reads = ezc.barplot(data=report_utils.per_sample_stats(
File "/home/osilande/.nextflow/assets/epi2me-labs/wf-16s/bin/workflow_glue/report_utils/report_utils.py", line 143, in per_sample_stats
return df_allstats.sort_values(by=['sample_name'])
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/util/_decorators.py", line 331, in wrapper
return func(*args, **kwargs)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/frame.py", line 6923, in sort_values
indexer = nargsort(
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/sorting.py", line 438, in nargsort
indexer = non_nan_idx[non_nans.argsort(kind=kind)]
TypeError: '<' not supported between instances of 'str' and 'int'
Work dir:
/scale_wlg_nobackup/filesets/nobackup/uoa03387/AG1268/work/1f/c3f2e4e42b56b4dc22498f64e5cc6f
Compare to below, the same sample but no sample sheet and thus with no step [c5/827031] process > validate_sample_sheet [100%] 1 of 1, cached: 1 ✔
)
executor > slurm (182)
[d1/617b24] process > fastcat (83) [100%] 89 of 89 ✔
[skipped ] process > prepare_databases:download_unpack_taxonomy [100%] 1 of 1, stored: 1 ✔
[skipped ] process > prepare_databases:download_reference_ref2taxid [100%] 1 of 1, stored: 1 ✔
[a3/ace151] process > minimap_pipeline:run_common:getVersions [100%] 1 of 1, cached: 1 ✔
[b9/13634a] process > minimap_pipeline:run_common:getParams [100%] 1 of 1, cached: 1 ✔
[86/ace595] process > minimap_pipeline:minimap (barcode65) [100%] 89 of 89 ✔
[a6/de756e] process > minimap_pipeline:createAbundanceTables [100%] 1 of 1 ✔
[64/1db8f3] process > minimap_pipeline:makeReport (1) [100%] 1 of 1 ✔
[23/cbc45d] process > minimap_pipeline:output_results (4) [100%] 4 of 4, cached: 2 ✔
Completed at: 01-Mar-2024 08:43:15
Duration : 1h 31m 19s
CPU hours : 252.8 (0% cached)
Succeeded : 182
Cached : 4
Hi @osilander , Could you open a different issue for your problem as it seems to be different from the initial one and that help me to keep track on the different issues. The error of the latest report is 137 which usually is related with not having enough memory, if you have enough memory, then you can use an extra file to override the default value. I'll provide you more info in the issue you open.
Thank you very much for using the pipeline!
Should I also open a different issue for the report above?
Hi I just updated the wf to v1.1.1, and got the same error once the wf runs minimap_pipeline::makeReport.
Checking inputs.
Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
Minimap2 pipeline.
Preparing databases.
Using default taxonomy database.
Using a default database.
[skipping] Stored process > prepare_databases:download_unpack_taxonomy
[skipping] Stored process > prepare_databases:download_reference_ref2taxid
[c2/066821] Submitted process > fastcat (4)
[bc/681b96] Submitted process > fastcat (7)
[9a/38f988] Submitted process > minimap_pipeline:run_common:getParams
[0a/28d2c6] Submitted process > fastcat (9)
[9f/1c04d4] Submitted process > fastcat (11)
[4e/edc2d7] Submitted process > fastcat (8)
[04/bb9d0f] Submitted process > minimap_pipeline:run_common:getVersions
[05/bd7879] Submitted process > fastcat (10)
[c8/ad8e39] Submitted process > fastcat (2)
[bc/efc61d] Submitted process > fastcat (5)
[4b/118e79] Submitted process > fastcat (6)
[72/1ea865] Submitted process > fastcat (3)
[e7/e7db87] Submitted process > fastcat (1)
[3d/7646b3] Submitted process > fastcat (12)
[df/5cf77f] Submitted process > minimap_pipeline:output_results (1)
[ba/018fe3] Submitted process > fastcat (13)
[fc/dd2b41] Submitted process > fastcat (14)
[ff/a790c2] Submitted process > fastcat (15)
[12/59d0d0] Submitted process > minimap_pipeline:output_results (2)
[26/77df73] Submitted process > fastcat (16)
[9b/7f5b21] Submitted process > minimap_pipeline:minimap (barcode16)
[87/2658e8] Submitted process > minimap_pipeline:minimap (barcode14)
[b9/7d6447] Submitted process > minimap_pipeline:minimap (barcode13)
[c7/43cf95] Submitted process > minimap_pipeline:minimap (barcode09)
[42/106603] Submitted process > minimap_pipeline:minimap (barcode15)
[4b/69dddb] Submitted process > minimap_pipeline:minimap (barcode02)
[1d/8bfa97] Submitted process > minimap_pipeline:minimap (barcode06)
[7a/0732b9] Submitted process > minimap_pipeline:minimap (barcode08)
[7a/bd8042] Submitted process > minimap_pipeline:minimap (barcode03)
[ca/1e2bd5] Submitted process > minimap_pipeline:minimap (barcode01)
[22/c79408] Submitted process > minimap_pipeline:minimap (barcode17)
[bf/0de3c6] Submitted process > minimap_pipeline:minimap (barcode04)
[5b/ec85a1] Submitted process > minimap_pipeline:minimap (barcode18)
[36/ee45ad] Submitted process > minimap_pipeline:minimap (barcode21)
[5f/968d43] Submitted process > minimap_pipeline:minimap (barcode23)
[72/6496f2] Submitted process > minimap_pipeline:minimap (barcode24)
[e6/5704fb] Submitted process > minimap_pipeline:createAbundanceTables
[46/7f2474] Submitted process > minimap_pipeline:output_results (3)
[eb/486250] Submitted process > minimap_pipeline:makeReport (1)
ERROR ~ Error executing process > 'minimap_pipeline:makeReport (1)'
Caused by:
Process minimap_pipeline:makeReport (1)
terminated with an error exit status (1)
Command executed:
workflow-glue report "wf-16s-report.html" --workflow_name wf-16s --versions versions --params params.json --read_stats read_stats/ --lineages lineages --abundance_table "abundance_table_genus.tsv" --taxonomic_rank "G" --pipeline "minimap2" --abundance_threshold "1" --n_taxa_barplot "9"
Command exit status:
1
Command output:
(empty)
Command error:
File "/home/epi2melabs/conda/lib/python3.8/site-packages/ezcharts/components/fastcat.py", line 374, in load_stats
df = pd.read_csv(
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/util/_decorators.py", line 211, in wrapper
return func(args, *kwargs)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/util/_decorators.py", line 331, in wrapper
return func(args, **kwargs)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 611, in _read
return parser.read(nrows)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1778, in read
) = self._engine.read( # type: ignore[attr-defined]
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 230, in read
chunks = self._reader.read_low_memory(nrows)
File "pandas/_libs/parsers.pyx", line 808, in pandas._libs.parsers.TextReader.read_low_memory
File "pandas/_libs/parsers.pyx", line 890, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 1037, in pandas._libs.parsers.TextReader._convert_column_data
File "pandas/_libs/parsers.pyx", line 1130, in pandas._libs.parsers.TextReader._convert_tokens
ValueError: invalid literal for int() with base 10: 'input'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/c/Users/jornb/epi2melabs/workflows/epi2me-labs/wf-16s/bin/workflow-glue", line 7, in bash .command.run
-- Check '/mnt/c/Users/jornb/epi2melabs/instances/wf-16s_01HRCWSZTM8848P11XBF38JCJX/nextflow.log' file for details
Hi @Jorn-Bethke, Could you open a new issue for your error as it seems to be a different one? Thank you very much in advance for using the workflow!
Hi @nggvs thanks for the update to v 1.1.2 problem solved when generating reports, working fine!
Oh those are good news! Glad to hear that! I'll close the issue in that case. Please feel free to open a new one is you detect something else is happening. Thank you for using the workflow!
Operating System
CentOS 7
Other Linux
No response
Workflow Version
v1.0.0
Workflow Execution
Command line
EPI2ME Version
No response
CLI command run
nextflow run epi2me-labs/wf-16s --fastq wf-16s-demo/test_data/ -profile standard -without-docker
Workflow Execution - CLI Execution Profile
None
What happened?
The pipeline faild at the last step 'making report'
Relevant log output
Application activity log entry
No response