makeReport runs out of memory

[jdiazsupsi]

Operating System

Ubuntu 22.04

Other Linux

No response

Workflow Version

v1.0.0-gc66a485

Workflow Execution

Command line

EPI2ME Version

No response

CLI command run

nextflow run epi2me-labs/wf-transcriptomes --fastq myfiles/fastq --ref_genome $ref --transcriptome-source reference-guided --ref_annotation $annot_gtf --direct_rna --de_analysis --sample_sheet sample_sheet.csv --threads 20 -profile standard

Workflow Execution - CLI Execution Profile

standard (default)

What happened?

Hi, I have been trying to run the workflow on six bacterial samples from a novel species and everything runs smoothly until the step in which it tries to create the report, where the docker container runs out of memory and gets killed. I have been monitoring the usage of resources with docker stats and indeed, memory goes up until it hits the limit, 2 GB (as specified in the main.nf script https://github.com/epi2me-labs/wf-transcriptomes/blob/c66a48525feb2ac4e8896776257715ba2b09a21a/main.nf#L381C1-L381C1). Is there a simple way to run the pipeline changing the 2 GB value set on main.nf?

I am not sure of what exactly makes the memory go up for this process. The test data ran through with no issue so it could be something specific to my dataset.

This issue could also be related to this closed one https://github.com/epi2me-labs/wf-transcriptomes/issues/22. Would it make sense to increase the default value in the pipeline a higher one to cover more use cases? After all, the pipeline requirements in the README specify a minimum of 16 GB.

Thanks for developing the workflow and for the support! Juan.

Relevant log output

[eb/5ed5f9] process > pipeline:makeReport (1)                                                [100%] 1 of 1, failed: 1 ✘
[3f/d160e7] process > output (35)                                                            [ 65%] 36 of 55, cached: 36
ERROR ~ Error executing process > 'pipeline:makeReport (1)'

Caused by:
  Process `pipeline:makeReport (1)` terminated with an error exit status (137)

Command executed:

  if [ -f "de_report/OPTIONAL_FILE" ]; then
      dereport=""
  else
      dereport="--de_report true --de_stats "seqkit/*""
      mv de_report/*.g*f* de_report/stringtie_merged.gtf
  fi
  if [ -f "gff_annotation/OPTIONAL_FILE" ]; then
      OPT_GFF=""
  else
      OPT_GFF="--gffcompare_dir cad_rep2_gffcompare cad_o2_rep1_gffcompare cad_rep3_gffcompare cad_o2_rep2_gffcompare cad_o2_rep3_gffcompare cad_rep1_gffcompare --gff_annotation gff_annotation/*"

  fi
  if [ -f "jaffal_csv/OPTIONAL_FILE" ]; then
      OPT_JAFFAL_CSV=""
  else
      OPT_JAFFAL_CSV="--jaffal_csv jaffal_csv/*"
  fi
  if [ -f "aln_stats/OPTIONAL_FILE" ]; then
      OPT_ALN=""
  else
      OPT_ALN="--alignment_stats aln_stats/*"
  fi
  if [ -f "pychopper_report/OPTIONAL_FILE" ]; then
      OPT_PC_REPORT=""
  else
      OPT_PC_REPORT="--pychop_report pychopper_report/*"
  fi
  workflow-glue report --report wf-transcriptomes-report.html     --versions versions.txt     --params params.json     $OPT_ALN     $OPT_PC_REPORT     --sample_ids cad_o2_rep3 cad_rep1 cad_o2_rep1 cad_rep3
cad_rep2 cad_o2_rep2     --stats per_read_stats/*     $OPT_GFF     --isoform_table_nrows 5000     $OPT_JAFFAL_CSV     $dereport

Command exit status:
  137

Command output:
  (empty)

Command error:
  10814 of 21628 (50%)
  11031 of 21628 (51%)
  11247 of 21628 (52%)
  11463 of 21628 (53%)
  11680 of 21628 (54%)
  11896 of 21628 (55%)
  12112 of 21628 (56%)
  12328 of 21628 (57%)
  12545 of 21628 (58%)
  12761 of 21628 (59%)
  12977 of 21628 (60%)
  13194 of 21628 (61%)
  13410 of 21628 (62%)
  13626 of 21628 (63%)
  13842 of 21628 (64%)
  14059 of 21628 (65%)
  14275 of 21628 (66%)
  14491 of 21628 (67%)
  14708 of 21628 (68%)
  14924 of 21628 (69%)
  15140 of 21628 (70%)
  15356 of 21628 (71%)
  15573 of 21628 (72%)
  15789 of 21628 (73%)
  16005 of 21628 (74%)
  16221 of 21628 (75%)
  16438 of 21628 (76%)
  16654 of 21628 (77%)
  16870 of 21628 (78%)
  17087 of 21628 (79%)
  17303 of 21628 (80%)
  17519 of 21628 (81%)
  17735 of 21628 (82%)
  17952 of 21628 (83%)
  18168 of 21628 (84%)
  18384 of 21628 (85%)
  18601 of 21628 (86%)
  18817 of 21628 (87%)
  19033 of 21628 (88%)
  19249 of 21628 (89%)
  19466 of 21628 (90%)
  19682 of 21628 (91%)
  19898 of 21628 (92%)
  20115 of 21628 (93%)
  20331 of 21628 (94%)
  20547 of 21628 (95%)
  20763 of 21628 (96%)
  20980 of 21628 (97%)
  21196 of 21628 (98%)
  .command.sh: line 29:    34 Killed                  workflow-glue report --report wf-transcriptomes-report.html --versions versions.txt --params params.json $OPT_ALN $OPT_PC_REPORT --sample_ids cad_o2_rep3 cad_rep1 cad_o2_rep1 cad_rep3 cad_rep2 cad_o2_rep2 --stats per_read_stats/* $OPT_GFF --isoform_table_nrows 5000 $OPT_JAFFAL_CSV $dereport

Work dir:
  /home/bioinfoserver/projects/juan/cadagno-genomics/transcriptome/work/eb/5ed5f906eb9932c74fb08cd47e66d4

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Application activity log entry

No response

[nrhorner]

Hi @jdiazsupsi

Sorry that you are experiencing issues with the workflow. We'll get this sorted ASAP

[mdhitch]

Hit this issue too...

@nrhorner, after your help and support on this one too.

Many thanks

Matt

[nrhorner]

Hi @mdhitch

Thanks for letting us know. We are working on getting a fix out for this issue. I'll let you know when it's ready Neil

[audrey-gibert]

Hello, I have the same issue (12 samples to analyze..) To solve this temporarily, you can edit the local.config file of the in-error workflow with this :

executor.$local.memory = "10 GB"
process {
    withName: 'makeReport' {
        memory = 10.GB
    }
}

Of course, the value of 10Gb can be adapted to your available memory. And the memory in the the process part can not be superior to the one specified with local.memory parameter

[jdiazsupsi]

Thanks a lot @audrey-gibert ! Modifying the nexflow.config as you said worked and the report was generated properly. I monitored the docker container with docker stats and saw the ram increase slowly to 4.2 GB. Then it exited successfully

[nrhorner]

HI @jdiazsupsi Thanks for reporting that, your feedback is useful to us.

This should be fixed in the next release.

[sarahjeeeze]

Hi, this should now be fixed in the latest release v1.1.0

[sarahjeeeze]

Closing as this is now released.