samtools sort: couldn't allocate memory for bam_mem

[royfrancis]

Running into the following error:

[W::bam_hdr_read] EOF marker is absent. The input is probably truncated
[E::bam_hdr_read] Invalid BAM binary header
[bam_cat] ERROR: couldn't read header for '/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//zUMIs_output/.tmpMap//tmp.30dpf.11.Aligned.toTranscriptome.out.bam'.
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated
[E::bam_hdr_read] Invalid BAM binary header
[bam_cat] ERROR: couldn't read header for '/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//zUMIs_output/.tmpMap//tmp.30dpf.11.Aligned.out.bam'.

And sample/zUMIs_output/expression/ is empty. Any ideas on what might be wrong?

zUMIs version 2.9.7c

Complete output

``` Welcome! You provided these parameters: YAML file: 30dpf.yaml zUMIs directory: /crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/zUMIs STAR executable STAR samtools executable samtools pigz executable pigz Rscript executable Rscript RAM limit: 1024 zUMIs version 2.9.7c Sat 14 Jan 21:19:33 CET 2023 WARNING: The STAR version used for mapping is 2.7.3a and the STAR index was created using the version 2.7.1a. This may lead to an error while mapping. If you encounter any errors at the mapping stage, please make sure to create the STAR index using STAR 2.7.3a. Filtering... Welcome! Welcome! Welcome! Welcome! Welcome! Welcome! Welcome! Welcome! Welcome! Sat 14 Jan 22:52:51 CET 2023 Welcome! [1] "22847507 reads were assigned to barcodes that do not correspond to intact cells." [1] "Found 13786 daughter barcodes that can be binned into 384 parent barcodes." [1] "Binned barcodes correspond to 22685672 reads." Mapping... Welcome! [1] "2023-01-14 22:59:27 CET" Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:29 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:29 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:29 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:29 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:33 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:35 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:37 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:37 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:37 ..... loading genome Jan 14 22:59:29 ..... started STAR run Jan 14 22:59:49 ..... loading genome Jan 14 23:00:55 ... generating Suffix Array index Jan 14 23:00:56 ... generating Suffix Array index Jan 14 23:00:56 ... generating Suffix Array index Jan 14 23:00:57 ... generating Suffix Array index Jan 14 23:00:58 ... generating Suffix Array index Jan 14 23:00:59 ... generating Suffix Array index Jan 14 23:00:59 ... generating Suffix Array index Jan 14 23:01:00 ... generating Suffix Array index Jan 14 23:01:01 ... generating Suffix Array index Jan 14 23:01:02 ... generating Suffix Array index Jan 14 23:03:59 ... completed Suffix Array index Jan 14 23:03:59 ..... processing annotations GTF Jan 14 23:04:02 ... completed Suffix Array index Jan 14 23:04:02 ..... processing annotations GTF Jan 14 23:04:04 ... completed Suffix Array index Jan 14 23:04:04 ..... processing annotations GTF Jan 14 23:04:04 ... completed Suffix Array index Jan 14 23:04:04 ..... processing annotations GTF Jan 14 23:04:05 ..... inserting junctions into the genome indices Jan 14 23:04:08 ..... inserting junctions into the genome indices Jan 14 23:04:10 ..... inserting junctions into the genome indices Jan 14 23:04:10 ..... inserting junctions into the genome indices Jan 14 23:04:16 ... completed Suffix Array index Jan 14 23:04:16 ..... processing annotations GTF Jan 14 23:04:22 ..... inserting junctions into the genome indices Jan 14 23:04:24 ... completed Suffix Array index Jan 14 23:04:24 ..... processing annotations GTF Jan 14 23:04:25 ... completed Suffix Array index Jan 14 23:04:25 ..... processing annotations GTF Jan 14 23:04:28 ... completed Suffix Array index Jan 14 23:04:28 ..... processing annotations GTF Jan 14 23:04:31 ..... inserting junctions into the genome indices Jan 14 23:04:32 ..... inserting junctions into the genome indices Jan 14 23:04:35 ..... inserting junctions into the genome indices Jan 14 23:04:39 ... completed Suffix Array index Jan 14 23:04:39 ..... processing annotations GTF Jan 14 23:04:41 ... completed Suffix Array index Jan 14 23:04:41 ..... processing annotations GTF Jan 14 23:04:46 ..... inserting junctions into the genome indices Jan 14 23:04:48 ..... inserting junctions into the genome indices Jan 14 23:13:19 ..... started mapping Jan 14 23:13:32 ..... started mapping Jan 14 23:13:43 ..... started mapping Jan 14 23:13:46 ..... started mapping Jan 14 23:14:28 ..... started mapping Jan 14 23:14:32 ..... started mapping Jan 14 23:14:41 ..... started mapping Jan 14 23:14:51 ..... started mapping Jan 14 23:14:53 ..... started mapping Jan 14 23:14:59 ..... started mapping Jan 15 00:14:15 ..... finished mapping Jan 15 00:14:16 ..... finished successfully Jan 15 00:19:54 ..... finished mapping Jan 15 00:19:56 ..... finished successfully Jan 15 00:20:23 ..... finished mapping Jan 15 00:20:24 ..... finished successfully Jan 15 00:21:59 ..... finished mapping Jan 15 00:22:01 ..... finished successfully Jan 15 00:22:53 ..... finished mapping Jan 15 00:22:55 ..... finished successfully Jan 15 00:23:01 ..... finished mapping Jan 15 00:23:03 ..... finished successfully Jan 15 00:23:11 ..... finished mapping Jan 15 00:23:13 ..... finished successfully Jan 15 00:25:09 ..... finished mapping Jan 15 00:25:12 ..... finished successfully Jan 15 03:12:00 ..... finished mapping Jan 15 03:12:01 ..... finished successfully Jan 15 04:54:04 ..... finished mapping Jan 15 04:54:07 ..... finished successfully [W::bam_hdr_read] EOF marker is absent. The input is probably truncated [E::bam_hdr_read] Invalid BAM binary header [bam_cat] ERROR: couldn't read header for '/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//zUMIs_output/.tmpMap//tmp.30dpf.11.Aligned.toTranscriptome.out.bam'. [W::bam_hdr_read] EOF marker is absent. The input is probably truncated [E::bam_hdr_read] Invalid BAM binary header [bam_cat] ERROR: couldn't read header for '/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//zUMIs_output/.tmpMap//tmp.30dpf.11.Aligned.out.bam'. Sun 15 Jan 04:55:00 CET 2023 Counting... Welcome! [1] "2023-01-15 04:55:17 CET" [1] "666666666 Reads per chunk" [1] "Loading reference annotation from:" [1] "/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//30dpf.final_annot.gtf" [1] "Annotation loaded!" Warning message: `as_quosure()` requires an explicit environment as of rlang 0.3.0. Please supply `env`. This warning is displayed once per session. [1] "Assigning reads to features (ex)" ========== _____ _ _ ____ _____ ______ _____ ===== / ____| | | | _ \| __ \| ____| /\ | __ \ ===== | (___ | | | | |_) | |__) | |__ / \ | | | | ==== \___ \| | | | _ <| _ /| __| / /\ \ | | | | ==== ____) | |__| | |_) | | \ \| |____ / ____ \| |__| | ========== |_____/ \____/|____/|_| \_\______/_/ \_\_____/ Rsubread 1.32.4 //========================== featureCounts setting ===========================\\ || || || Input files : 1 BAM file || || P 30dpf.filtered.tagged.Aligned.out.bam || || || || Annotation : R data.frame || || Assignment details : .featureCounts.bam || || (Note that files are saved to the output directory) || || || || Dir for temp files : . || || Threads : 10 || || Level : meta-feature level || || Paired-end : yes || || Multimapping reads : counted || || Multiple alignments : primary alignment only || || Multi-overlapping reads : not counted || || Min overlapping bases : 1 || || || || Chimeric reads : not counted || || Both ends mapped : not required || || || \\===================== http://subread.sourceforge.net/ ======================// //================================= Running ==================================\\ || || || Load annotation file .Rsubread_UserProvidedAnnotation_pid28421 ... || || Features : 272318 || || Meta-features : 32522 || || Chromosomes/contigs : 302 || || || || Process BAM file 30dpf.filtered.tagged.Aligned.out.bam... || || Paired-end reads are included. || || Assign alignments (paired-end) to features... || || || || WARNING: reads from the same pair were found not adjacent to each || || other in the input (due to read sorting by location or || || reporting of multi-mapping read pairs). || || || || Pairing up the read pairs. || || || || Total alignments : 186884281 || || Successfully assigned alignments : 129832662 (69.5%) || || Running time : 4.56 minutes || || || || || \\===================== http://subread.sourceforge.net/ ======================// [1] "Assigning reads to features (in)" ========== _____ _ _ ____ _____ ______ _____ ===== / ____| | | | _ \| __ \| ____| /\ | __ \ ===== | (___ | | | | |_) | |__) | |__ / \ | | | | ==== \___ \| | | | _ <| _ /| __| / /\ \ | | | | ==== ____) | |__| | |_) | | \ \| |____ / ____ \| |__| | ========== |_____/ \____/|____/|_| \_\______/_/ \_\_____/ Rsubread 1.32.4 //========================== featureCounts setting ===========================\\ || || || Input files : 1 BAM file || || P 30dpf.filtered.tagged.Aligned.out.bam.ex.f ... || || || || Annotation : R data.frame || || Assignment details : .featureCounts.bam || || (Note that files are saved to the output directory) || || || || Dir for temp files : . || || Threads : 10 || || Level : meta-feature level || || Paired-end : yes || || Multimapping reads : counted || || Multiple alignments : primary alignment only || || Multi-overlapping reads : not counted || || Min overlapping bases : 1 || || || || Chimeric reads : not counted || || Both ends mapped : not required || || || \\===================== http://subread.sourceforge.net/ ======================// //================================= Running ==================================\\ || || || Load annotation file .Rsubread_UserProvidedAnnotation_pid28421 ... || || Features : 223948 || || Meta-features : 25038 || || Chromosomes/contigs : 255 || || || || Process BAM file 30dpf.filtered.tagged.Aligned.out.bam.ex.featureCount ... || || Paired-end reads are included. || || Assign alignments (paired-end) to features... || || || || WARNING: reads from the same pair were found not adjacent to each || || other in the input (due to read sorting by location or || || reporting of multi-mapping read pairs). || || || || Pairing up the read pairs. || || || || Total alignments : 186884281 || || Successfully assigned alignments : 11864221 (6.3%) || || Running time : 4.86 minutes || || || || || \\===================== http://subread.sourceforge.net/ ======================// [1] "2023-01-15 05:05:50 CET" [1] "Coordinate sorting intermediate bam file..." samtools sort: couldn't allocate memory for bam_mem [E::hts_open_format] Failed to open file "/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//30dpf.filtered.Aligned.GeneTagged.sorted.bam" : No such file or directory samtools index: failed to open "/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//30dpf.filtered.Aligned.GeneTagged.sorted.bam": No such file or directory [1] "2023-01-15 05:05:51 CET" [1] "Hamming distance collapse in barcode chunk 1 out of 1" Error in value[[3L]](cond) : failed to open BamFile: file(s) do not exist: ‘/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//30dpf.filtered.Aligned.GeneTagged.sorted.bam’ Calls: reads2genes_new ... tryCatch -> tryCatchList -> tryCatchOne -> Execution halted Sun 15 Jan 05:05:52 CET 2023 Welcome! Loading required package: yaml Loading required package: Matrix [1] "loomR found" Error in gzfile(file, "rb") : cannot open the connection Calls: rds_to_loom -> readRDS -> gzfile In addition: Warning message: In gzfile(file, "rb") : cannot open compressed file '/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//zUMIs_output/expression/30dpf.dgecounts.rds', probable reason 'No such file or directory' Execution halted Sun 15 Jan 05:05:56 CET 2023 Descriptive statistics... Welcome! [1] "I am loading useful packages for plotting..." [1] "2023-01-15 05:05:56 CET" Error in gzfile(file, "rb") : cannot open the connection Calls: readRDS -> gzfile In addition: Warning message: In gzfile(file, "rb") : cannot open compressed file '/crex/proj/snicxxx/nobackup/nbis/data/processed/zumis/30dpf//zUMIs_output/expression/30dpf.dgecounts.rds', probable reason 'No such file or directory' Execution halted ```

[cziegenhain]

Hi,

I would recommending running with significantly lower RAM limit! you are allocating 1 TB. Assuming one of the STAR jobs doesn't make it because of that leaving you with the truncated bam file warning and then eventually samtools sort crashes for lack of memory.

There is typically only small speedup after allowing 100 - 200 GB of RAM anyways

Best, C

[royfrancis]

I suspected RAM as well since the fastqs are huge. It's about 270 million reads per fastq. I gave it a 1TB of RAM, But looking at the job profile, it doesn't seem to be using all of it.

[cziegenhain]

270M reads is not much at all, this will run super fine with 100GB memory. RAM usage can peak occasionally and zUMIs can't control too precisely eg. samtools. it's good practice to leave some buffer between the memory limit set in zUMIs and the physical limit of the node you are running on.

[royfrancis]

I downsampled to 1mil reads just to test RAM usage. I gave it 128GB of RAM and it failed with same error. I suspect specifications in yaml might play a role.

Since my node has 20 cores and 128GB of RAM, I set

num_threads: 20
mem_limit: 126

I am not sure if zumis respects the max memory limit specified here.

[cziegenhain]

In your log before it was clearly a samtools issue, the RAM usage is rounded so you can see this was the source of the fail: samtools sort: couldn't allocate memory for bam_mem

On a node of 128GB RAM, I would set eg. 80 GB and you'll be fine.

[royfrancis]

Finally this worked! Thanks very much!