TreeAnnotator error: java.lang.OutOfMemoryError: GC overhead limit exceeded

[SticaC]

I am using command line TreeAnnotator on LogCombined tree files that range from size 3.2G to 12G. I keep getting the error message "java.lang.OutOfMemoryError: GC overhead limit exceeded" even when allocated up to 50G memory to the program and running on 16 cpus. When it errors out, it doesnt even seem to be using all of the memory that has been allocated, usually only about 19G.

Is there a fix for this issue?

Thanks

[rbouckaert]

Try running treeannotator with the -lowMem option.

[SticaC]

Hello,

I am already running it using the lowMem option. Upon increasing the available memory to 100G on 16 cpus,

treeannotator -burnin 10 -lowMem -heights ca ${INPUT} ${SAMPLEID}_MCC.trees

I have received the following message:

          TreeAnnotator v2.6.3, 2002-2020
                MCMC Output analysis
                         by
       Andrew Rambaut and Alexei J. Drummond

         Institute of Evolutionary Biology
              University of Edinburgh
                 a.rambaut@ed.ac.uk

           Department of Computer Science
               University of Auckland
              alexei@cs.auckland.ac.nz

Please cite: Heled and Bouckaert: Looking for trees in the forest: summary tree from posterior samples. BMC Evolutionary Biology 2013 13:221. Processing 81005 trees from file after ignoring first 10% = 9000 trees.

Total number of trees 90005, where 81005 are used. Total unique clades: 7847

Finding maximum credibility tree... Analyzing 81005 trees... 0 25 50 75 100 |--------------|--------------|--------------|--------------|

---

Highest Log Clade Credibility: -292.061722067245 Collecting node information... 0 25 50 75 100 |--------------|--------------|--------------|--------------|

---

Annotating target tree... Setting node heights... 0 25 50 75 100 |--------------|--------------|--------------|--------------| java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at beast.app.beastapp.BeastLauncher.run(Unknown Source) at beast.app.treeannotator.TreeAnnotatorLauncher.main(Unknown Source) Caused by: java.lang.OutOfMemoryError: Java heap space at beast.app.treeannotator.TreeAnnotator.setTreeHeightsByCA(Unknown Source) at beast.app.treeannotator.TreeAnnotator.(Unknown Source) at beast.app.treeannotator.TreeAnnotator.main(Unknown Source) ... 6 more PBS Job 9745005.pbs CPU time : 04:49:03 Wall time : 00:46:02 Mem usage : 13473516kb

[rbouckaert]

@SticaC there should not be a problem when using -lowMem using this many trees, and the job suggests only 13473516kb or approximately 13GB is used. When you set the available memory to 100G, do you set this in the treeannotator script, that is, do you change the -Xmx8g at the last line in the script to -Xmx100g, or do you run this on a cluster where you assign the job 100G. The memory available to treeannotator is limited by how much is specified in the -Xmx directive, so perhaps that is causing some problems here?

[XiaXiaTianTian]

@SticaC there should not be a problem when using -lowMem using this many trees, and the job suggests only 13473516kb or approximately 13GB is used. When you set the available memory to 100G, do you set this in the treeannotator script, that is, do you change the -Xmx8g at the last line in the script to -Xmx100g, or do you run this on a cluster where you assign the job 100G. The memory available to treeannotator is limited by how much is specified in the -Xmx directive, so perhaps that is causing some problems here?

Could you please show example command for this? I tried several ways to add -Xmx100g but all failed. Thanks a lot!

Xia

[XiaXiaTianTian]

java -Xmx16g /ds3200_1/users_root/beast/lib/launcher.jar beast.app.treeannotator.TreeAnnotator -burnin 20 -heights mean aa.trees out.txt

/ds3200_1/users_root/beast/bin/treeannotator -Xmx16g -burnin 20 -heights mean aa.trees out.txt

...

[rbouckaert]

The -Xmx directive is an option for java, so the first entry is correct:

java -Xmx16g /ds3200_1/users_root/beast/lib/launcher.jar beast.app.treeannotator.TreeAnnotator -burnin 20 -heights mean aa.trees out.txt

You can also simply edit the treeannotator script, which on the last line contains this line:

"$JAVA" -Dlauncher.wait.for.exit=true -Xms256m -Xmx8g -Djava.library.path="$BEAST_LIB" -Duser.language=en -cp "$BEAST_LIB/launcher.jar" beast.pkgmgmt.launcher.TreeAnnotatorLauncher $*

and replace the 8 in -Xmx8g with whatever you require.

[XiaXiaTianTian]

The -Xmx directive is an option for java, so the first entry is correct:

java -Xmx16g /ds3200_1/users_root/beast/lib/launcher.jar beast.app.treeannotator.TreeAnnotator -burnin 20 -heights mean aa.trees out.txt

You can also simply edit the treeannotator script, which on the last line contains this line:

"$JAVA" -Dlauncher.wait.for.exit=true -Xms256m -Xmx8g -Djava.library.path="$BEAST_LIB" -Duser.language=en -cp "$BEAST_LIB/launcher.jar" beast.pkgmgmt.launcher.TreeAnnotatorLauncher $*

and replace the 8 in -Xmx8g with whatever you require.

Got it. Thank you so much!