search

jfarek / xatlas

BSD 3-Clause "New" or "Revised" License
29 stars 3 forks source link

memory footprint #2

Closed slw287r closed 4 years ago

slw287r commented 6 years ago

Hi Jesse,

What about xatlas' memory usage? I just run it with my WES data (average coverage depth of 30x) and xatlas consumed 180G memory (RES) and 318G (VIRT) before I kill it. The log shows it's Processing records for region 3:197954977-197955050.

The command I used:

xatlas -r $hg19 -i wes.bam -s wes -p wes -t 8 -c $wes_bed -m 50 -n 25 -M 300 -z

Best, Richard

jfarek commented 6 years ago

Hi,

For 30x WES, xAtlas should use around 3-4G of resident memory.

Which version of htslib are you using? And is this reproducible if you omit the -t option? I suspect that this might be a memory leak found in some older versions of htslib related to the BGZF decompression thread pool, since I might have seen similar behavior before with xAtlas built with older htslib versions. xAtlas uses htslib's BGZF decompression thread pool to enable multithreaded BAM decompression when using the -t option.

slw287r commented 6 years ago

Hi, Leave out the -t option works like a charm!

BTW, I'm using the latest htslib 1.8 in compiling xAtlas. I'll try some older versions to see if the problem persists.