search

epi2me-labs / wf-single-cell

Other
74 stars 39 forks source link

Error while running stringtie step #63

Closed ddiez closed 6 months ago

ddiez commented 11 months ago

Operating System

Ubuntu Linux

Other Linux

Ubuntu 23.10

Workflow Version

prerelease

Workflow Execution

EPI2ME Desktop application

EPI2ME Version

No response

CLI command run

nextflow run epi2me-labs/wf-single-cell \
   -r prerelease \
   -w workspace \
   -profile standard \
   --fastq ../data/final \
   --single_cell_sample_sheet samples.txt \
   --ref_genome_dir ~/10x/refdata-gex/refdata-gex-GRCh38-2020-A \
   --out_dir output \
   --plot_umaps \
   --umap_n_repeats 1

Workflow Execution - CLI Execution Profile

None

What happened?

The workflow stops with an error during stingtie step. The input data is the concatenated fastq files from 2 P2SOLO flow cells. The same workflow on 2 Promethion FC data worked OK a few days before. It is not clear whether it is related to #22 since in my cases it stops with an error rather than being stuck. The error message suggests to look into .command.sh in the working directory where the error happens. This is the content:

#!/bin/bash -euo pipefail
# Data from 3prime and multiome kits must be flipped to the transcript strand before building transcriptome.
workflow-glue process_bam_for_stringtie align.bam chr17          | tee >(stringtie -L -c 2 -p 4 -G chr.gtf -l stringtie             -o stringtie.gff - )         | samtools fastq > reads.fastq
# Get transcriptome sequence
gffread -g ref_genome.fa -w transcriptome.fa stringtie.gff

Relevant log output

...
Dec-02 21:20:24.049 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Dec-02 21:20:24.049 [Task submitter] INFO  nextflow.Session - [3b/6e1038] Submitted process > pipeline:process_bams:stringtie (7)
Dec-02 21:20:38.792 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 49; name: pipeline:process_bams:stringtie (24); status: COMPLETED; exit: 141; error: -; workDir: /media/diez/Biodev/projects/covid-sex/nanopore/wf-single-cell/workspace/63/b457607ed38fe116b458b890a41c48]
Dec-02 21:20:38.797 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:process_bams:stringtie (24); work-dir=/media/diez/Biodev/projects/covid-sex/nanopore/wf-single-cell/workspace/63/b457607ed38fe116b458b890a41c48
  error [nextflow.exception.ProcessFailedException]: Process `pipeline:process_bams:stringtie (24)` terminated with an error exit status (141)
Dec-02 21:20:38.799 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Dec-02 21:20:38.799 [Task submitter] INFO  nextflow.Session - [ed/73c93d] Submitted process > pipeline:process_bams:stringtie (16)
Dec-02 21:20:38.916 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'pipeline:process_bams:stringtie (24)'

Caused by:
  Process `pipeline:process_bams:stringtie (24)` terminated with an error exit status (141)

Command executed:

  # Data from 3prime and multiome kits must be flipped to the transcript strand before building transcriptome.
  workflow-glue process_bam_for_stringtie align.bam chr17          | tee >(stringtie -L -c 2 -p 4 -G chr.gtf -l stringtie             -o stringtie.gff - )         | samtools fastq > reads.fastq
  # Get transcriptome sequence
  gffread -g ref_genome.fa -w transcriptome.fa stringtie.gff

Command exit status:
  141

Command output:
  (empty)

Command error:
  [12:18:38 - matplotlib.font_manager] generated new fontManager
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
    @numba.jit()
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
    @numba.jit()
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
    @numba.jit()
  /home/epi2melabs/conda/lib/python3.8/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
    @numba.jit()
  [12:18:58 - workflow_glue] Starting entrypoint.
  .command.sh: line 3:    37 Killed                  stringtie -L -c 2 -p 4 -G chr.gtf -l stringtie -o stringtie.gff -
  [M::bam2fq_mainloop] discarded 0 singletons
  [M::bam2fq_mainloop] processed 3769431 reads
  Traceback (most recent call last):
    File "/home/diez/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow-glue", line 7, in <module>
      cli()
    File "/home/diez/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/__init__.py", line 72, in cli
      args.func(args)
    File "/home/diez/.nextflow/assets/epi2me-labs/wf-single-cell/bin/workflow_glue/process_bam_for_stringtie.py", line 41, in main
      bam_out.write(align)
    File "pysam/libcalignmentfile.pyx", line 1708, in pysam.libcalignmentfile.AlignmentFile.write
    File "pysam/libcalignmentfile.pyx", line 1740, in pysam.libcalignmentfile.AlignmentFile.write
  OSError: sam_write1 failed with error code -1

Work dir:
  /media/diez/Biodev/projects/covid-sex/nanopore/wf-single-cell/workspace/63/b457607ed38fe116b458b890a41c48

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Dec-02 21:20:38.924 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `pipeline:process_bams:stringtie (24)` terminated with an error exit status (141)
Dec-02 21:20:38.950 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:
[process] pipeline:process_bams:get_contigs
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:extract_barcodes
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (value) bound ; channel: bc_longlist_dir
  port 2: (cntrl) -     ; channel: $

[process] pipeline:process_bams:combine_uncorrect_bcs
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:generate_whitelist
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:assign_barcodes
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:stringtie
  status=ACTIVE
  port 0: (value) bound ; channel: ref_genome.fa
  port 1: (value) bound ; channel: ref_genome.fa.fai
  port 2: (queue) OPEN  ; channel: -
  port 3: (cntrl) -     ; channel: $

[process] pipeline:process_bams:align_to_transcriptome
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:assign_features
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:cluster_umis
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:tag_bams
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:combine_tag_files
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:combine_final_tag_files
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:umi_gene_saturation
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:construct_expression_matrix
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:process_expression_matrix
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:umap_reduce_expression_matrix
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:process_bams:pack_images
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:prepare_report_data
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] pipeline:makeReport
  status=ACTIVE
  port 0: (value) bound ; channel: versions
  port 1: (value) bound ; channel: params.csv
  port 2: (queue) OPEN  ; channel: read_stats.csv
  port 3: (queue) OPEN  ; channel: survival.tsv
  port 4: (queue) OPEN  ; channel: wf_summary.tsv
  port 5: (queue) OPEN  ; channel: umap_dirs
  port 6: (value) OPEN  ; channel: images
  port 7: (value) bound ; channel: umap_genes
  port 8: (cntrl) -     ; channel: $

[process] output
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: -
  port 1: (cntrl) -     ; channel: $

[process] output_report
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: fname
  port 1: (cntrl) -     ; channel: $

Dec-02 21:20:40.282 [main] DEBUG nextflow.Session - Session await > all processes finished
Dec-02 21:20:40.282 [main] DEBUG nextflow.Session - Session await > all barriers passed
Dec-02 21:20:40.287 [Actor Thread 631] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Dec-02 21:20:40.320 [Actor Thread 718] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Dec-02 21:20:40.320 [Actor Thread 719] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Dec-02 21:20:40.325 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Dec-02 21:20:40.585 [main] WARN  n.processor.TaskPollingMonitor - Killing running tasks (9)
Dec-02 21:20:40.666 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=32; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=71; submittedCount=1; runningCount=-1; retriesCount=0; abortedCount=9; succeedDuration=14d 18h 50m 56s; failedDuration=8m 18s; cachedDuration=0ms;loadCpus=-4; loadMemory=0; peakRunning=9; peakCpus=50; peakMemory=494 GB; ]
Dec-02 21:20:40.666 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Dec-02 21:20:40.670 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Dec-02 21:20:41.455 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Dec-02 21:20:41.534 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Dec-02 21:20:41.582 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Application activity log entry

No response

nrhorner commented 11 months ago

Hi @ddiez

Sorry that your havign issues with the workflow. It may well be related to #22. There is a fix on the way shortly to remove this step from the workflow. I will let you know when it's ready.

ddiez commented 11 months ago

Thanks for the heads up. I am actually rerunning this again because I used 3prime,v3 as kit options, but I realized my data is multiomics so needed to fix that. Maybe this is unrelated and won't fix the problem but I will let you know of the outcome.

ddiez commented 11 months ago

Just to update on this, setting the right kit type and version did not help to solve this problem.

ddiez commented 6 months ago

Using the latest prerelease in a cluster with more resources I was able to get the dataset in this issue to finish successfully. Although I am not sure if the issues with stringtie are completely solved and it is just a question of enough resources, I think this issue can be closed.