iosonofabio
commented
3 years ago Memory exhausted, the Python interpreter gives up before even throwing an error. Get a bigger machine or, since htseq-count is trivially parallelizable, split the bam files
When I use htseq-count to process my bam files, I met the following issue:
66200000 alignment record pairs processed killedMy environment is Ubuntu 20.04 loaded on VMware Workstation 16, 8Tb storage, 32Gb RAM.I have 140 bam files required for counting. I tried "for loop" to output my data.
for i in $(seq 1 140); do htseq-count --format bam --order pos --stranded no --minaqual 10 --type exon --idattr gene_id --mode union /home/tomas/project/bam/a/a${i}.bam /home/tomas/project/ref/gencode.v38.annotation.gtf>/home/tomas/project/expression/a/a${i}_gene.tsv; doneMost of them works. But there are 14 "killed" failures, resulting in 14 " .tsv" files with 0 byte.
I'm not sure if it is the problem with RAM because I opened 4 terminals and worked on it parallelly (e.g 1-40, 41-70, 71-100, 101-140). But I checked the memory during that time, there's no warning of memory exhausation.
May I ask why it happens and how to deal with it? thank you