Open DarioS opened 9 months ago
@DarioS How much memory are you providing to Java via the -Xmx
option, and how much physical memory do you have available? You can see how to pass the -Xmx
option in to GATK here: https://github.com/broadinstitute/gatk?tab=readme-ov-file#jvmoptions
-Xmx52g
was used. Compute node has 1.5 TB physical RAM. I use af-only-gnomad.hg38.vcf.gz
for -V
and -L
.
@DarioS You could try increasing the size of the Java heap (say, doubling it to 104g). Does your bam/cram have extremely high depth?
I copied 60× BAM file to an interactive Linux server with 768 GB physical RAM and eighty cores and used version 4.5.0.0.
%Cpu(s): 1.3 us, 0.0 sy, 0.1 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 754.5 total, 52.1 free, 107.3 used, 600.3 buff/cache
GiB Swap: 931.3 total, 924.9 free, 6.4 used. 647.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
171365 dario 20 0 35.0g 31.3g 23040 S 100.0 4.1 32:18.12 java
I removed -Xmx
and using top
to see the process is consistently at about 32 GB. So, -Xmx
is irrelevant to the problem.
12:15:04.531 INFO ProgressMeter - Current Locus Elapsed Minutes Loci Processed Loci/Minute
12:57:32.208 INFO GetPileupSummaries - Shutting down engine
[January 13, 2024 at 12:57:32 PM AEDT] org.broadinstitute.hellbender.tools.walkers.contamination.GetPileupSummaries done. Elapsed time: 50.36 minutes.
Runtime.totalMemory()=20753416192
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
What does "reallocation of scalar replaced objects" mean? I don't think it could possibly have run out of memory.
I am in a similar boat. Xmx has a default value which is small. Using a specified 448 GB limit shows that this module is inefficient.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
172833 thind 20 0 468.7g 378.7g 31360 S 99.9 50.2 29:16.31 java
The analysis dies a few seconds later because GATK tries to create impossibly-large Java array.
org.broadinstitute.hellbender.tools.walkers.contamination.GetPileupSummaries done. Elapsed time: 25.63 minutes.
Runtime.totalMemory()=481036337152
java.lang.OutOfMemoryError: Required array length 2147483640 + 16 is too large
at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
I can independently reproduce Dario's problem on the same Linux server.
CPU usage was high.
However, it never proceeds past the first interval.
There is a memory error in some log files but only after many hours and no intervals processed.