ERROR ~ Error executing process > `pipeline:differential_expression:map_transcriptome (4)` terminated with an error exit status (137)

[RPdavies]

I get the below error when running on a HPC with 120 cores and 500GB RAM. I read that 137 relates to memory so I increased the allocation to 250G using the -c parameter and a mem.config file, as suggested here: https://github.com/epi2me-labs/wf-transcriptomes/issues/64#issuecomment-1921039910

However, the same error persists:

ERROR ~ Error executing process > 'pipeline:differential_expression:map_transcriptome (4)'
Caused by:
Process pipeline:differential_expression:map_transcriptome (4) terminated with an error exit status (137)

Command executed:
minimap2 -t 200 -ax splice -uf -p 1.0 "genome_index.mmi" "seqs.fastq.gz" | samtools view -Sb > "output.bam"
samtools sort -@ 200 "output.bam" -o "B3-2Gy-24h_reads_aln_sorted.bam"
Command exit status:
137
Command output:
(empty)
Command error:
[WARNING] Indexing parameters (-k, -w or -H) overridden by parameters used in the prebuilt index.
[M::main::2.1840.54] loaded/built the index for 206723 target sequence(s)
[M::mm_mapopt_update::2.3450.57] mid_occ = 190
[M::mm_idx_stat] kmer size: 14; skip: 10; is_hpc: 0; #seq: 206723
[M::mm_idx_stat::2.4250.59] distinct minimizers: 12256745 (29.72% are singletons); average occurrences: 5.903; average spacing: 5.440; t otal length: 393634404
[M::worker_pipeline::33.41345.94] mapped 628018 sequences
[M::worker_pipeline::53.41042.86] mapped 628797 sequences [M::worker_pipeline::72.73941.64] mapped 629812 sequences
[M::worker_pipeline::92.55140.78] mapped 628171 sequences
[M::worker_pipeline::112.88340.13] mapped 630448 sequences
[M::worker_pipeline::133.08839.68] mapped 629206 sequences
[M::worker_pipeline::153.22739.23] mapped 629394 sequences
[M::worker_pipeline::172.54439.09] mapped 627077 sequences
[M::worker_pipeline::192.28338.92] mapped 629971 sequences
[M::worker_pipeline::212.57738.75] mapped 627593 sequences
[M::worker_pipeline::232.90238.57] mapped 629078 sequences
[M::worker_pipeline::253.16438.42] mapped 629005 sequences
[M::worker_pipeline::273.32238.37] mapped 629411 sequences
[M::worker_pipeline::292.87938.33] mapped 629842 sequences
[M::worker_pipeline::313.29738.23] mapped 629964 sequences
[M::worker_pipeline::333.49238.14] mapped 628898 sequences
[M::worker_pipeline::353.24838.11] mapped 628806 sequences
[M::worker_pipeline::373.31838.09] mapped 627001 sequences
[M::worker_pipeline::392.73638.11] mapped 626155 sequences
[M::worker_pipeline::413.05138.00] mapped 626833 sequences
[M::worker_pipeline::433.24837.94] mapped 626935 sequences
[M::worker_pipeline::453.07437.90] mapped 628324 sequences
[M::worker_pipeline::473.16737.86] mapped 629846 sequences
[M::worker_pipeline::492.48637.32] mapped 631375 sequences
[M::worker_pipeline::504.04436.47] mapped 401398 sequences
[M::main] Version: 2.24-r1122
[M::main] CMD: minimap2 -t 200 -ax splice -uf -p 1.0 genome_index.mmi seqs.fastq.gz
[M::main] Real time: 504.397 sec; CPU: 18384.409 sec; Peak RSS: 11.430 GB
[bam_sort_core] merging from 0 files and 200 in-memory blocks...
.command.sh: line 3: 13677 Killed samtools sort -@ 200 "output.bam" -o "B3-2Gy-24h_reads_aln_sorted.bam"

[RPdavies]

Full command can be seen here:

nextflow run epi2me-labs/wf-transcriptomes -resume -c mem.config\ --fastq /buffer_drive/DGE-old-kit/input \ --de_analysis \ --ref_annotation /home/cmb-group/upni/media/cmb-group/venus/human_reference/transcriptome/Homo_sapiens.GRCh38.112.gtf \ --ref_transcriptome /home/cmb-group/upni/media/cmb-group/venus/human_reference/transcriptome/Homo_sapiens.GRCh38.cdna.all.fa \ --transcriptome_source precomputed \ --minimap2_index_opts '-k 14 -uf' \ --sample_sheet /buffer_drive/DGE-old-kit/sample_sheet.csv \ --ref_genome /home/cmb-group/upni/media/cmb-group/venus/human_reference/transcriptome/Homo_sapiens.GRCh38.cdna.all.fa \ --outdir /buffer_drive/DGE-old-kit/output \ --threads 200 \ (base) root@cmbhpcchi02:/buffer_drive/DGE-old-kit# cat nextflow-instructions nextflow run epi2me-labs/wf-transcriptomes -resume -c mem.config\ --fastq /buffer_drive/DGE-old-kit/input \ --de_analysis \ --ref_annotation /home/cmb-group/upni/media/cmb-group/venus/human_reference/transcriptome/Homo_sapiens.GRCh38.112.gtf \ --ref_transcriptome /home/cmb-group/upni/media/cmb-group/venus/human_reference/transcriptome/Homo_sapiens.GRCh38.cdna.all.fa \ --transcriptome_source precomputed \ --minimap2_index_opts '-k 14 -uf' \ --sample_sheet /buffer_drive/DGE-old-kit/sample_sheet.csv \ --ref_genome /home/cmb-group/upni/media/cmb-group/venus/human_reference/transcriptome/Homo_sapiens.GRCh38.cdna.all.fa \ --outdir /buffer_drive/DGE-old-kit/output \ --threads 200 \

[SamStudio8]

[bam_sort_core] merging from 0 files and 200 in-memory blocks...

Have you set a thread parameter to 200?

[SamStudio8]

@RPdavies Thanks for the update, I see you have provided --threads 200. Each thread during mapping and merging is going to take some amount of your memory. You may wish to provide a more sensible number of threads which should alleviate the memory pressure of the workflow.

[RPdavies]

Thanks for the quick reply. Reported peak RSS was 11.430 GB, so I thought increasing availability to 250G using mem.config would suffice. I will repeat with fewer threads and get back to you, cheers

[RPdavies]

@SamStudio8 I resumed the run using only 20 threads as per your recommendation, it's been frozen however over the past 12 hours or so:

This is epi2me-labs/wf-transcriptomes v1.4.0-gd2ddad2.

---

Reference Transcriptome provided will be used for differential expression.
Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
executor > local (12)
[39/599bc8] validate_sample_sheet | 1 of 1, cached: 1 ✔
[3a/4cb782] fastcat (30) | 30 of 30, cached: 30 ✔
[d6/cb1969] pipeline:preprocess_ref_annotation | 1 of 1, cached: 1 ✔
[f0/b0a3c0] pipeline:collectFastqIngressResultsInDir (30) | 30 of 30, cached: 30 ✔
[b0/fac849] pipeline:getVersions | 1 of 1, cached: 1 ✔
[d1/8caca7] pipeline:getParams | 1 of 1, cached: 1 ✔
[- ] pipeline:preprocess_reads | 0 of 30
[82/56e532] pipeline:check_annotation_strand | 1 of 1, cached: 1 ✔
[87/0f20ac] pipeline:preprocess_ref_transcriptome (1) | 1 of 1, cached: 1 ✔
[f4/eef6ee] pip…erential_expression:checkSampleSheetCondition (1) | 1 of 1, cached: 1 ✔
[0d/8d933d] pip…_expression:build_minimap_index_transcriptome (1) | 1 of 1, cached: 1 ✔
[02/90cb9a] pip…ne:differential_expression:map_transcriptome (26) | 15 of 30, cached: 15
[- ] pipeline:differential_expression:count_transcripts | 0 of 15
[- ] pipeline:differential_expression:mergeCounts -
[- ] pipeline:differential_expression:mergeTPM -
[- ] pipeline:differential_expression:deAnalysis - [- ] pipeline:differential_expression:plotResults - [- ] pipeline:makeReport - [- ] publish_results -