Generational Shenandoah non promoting objects to Old Gen

[chevaris]

Describe the bug

Running Apache Geode (In memory DB) with Amazon Coretto early builds with Generational Shenandoah (amazon-corretto-17.0.1.12.1-06f6f03f6a-linux-x64.tar.gz).

Apache Geode is an in memory DB, and I am running another program to populate data in the DB

Using JConsole is trivial to observe that Old Generation is completely empty and GC not able to promote objects to Old Gen, even after many minutes and keep populating Geode with many many thousands of entries and after many Young Gen collections.

Only when Young Gen is almost exhausted a very minor portion of referenced objects look to be promoted to Old Gen.

Geode Server JVM options can be seen in /home/eceejcr/Downloads/amazon-corretto-17.0.1.12.1-linux-x64/bin/java -server -classpath /home/eceejcr/Development/geode/apache-geode-1.14.0/lib/geode-core-1.14.0.jar:/home/eceejcr/Development/geode/apache-geode-1.14.0/lib/geode-dependencies.jar -Dgemfire.start-dev-rest-api=false -Dgemfire.locators=127.0.0.1[10334] -Dgemfire.use-cluster-configuration=true -Xmx6144m -Xms6144m -Xmn1024m -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahGCMode=generational -XX:InitialTenuringThreshold=5 -Xlog:gc*:file=gc.log:time,uptime:filecount=5,filesize=20M --add-opens java.base/java.nio=ALL-UNNAMED -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806 org.apache.geode.distributed.ServerLauncher start server2 --server-port=0

I know that heuristics are missing in generational shenandoah, so maybe my JVM options are not correct

To Reproduce

Install Apache Geode, and try to populate it

Expected behavior

Objects promotion to old gen to happen gradually like with any other generational collector like G1

Screenshots

Platform information

OS: manjaro
Version : corretto-17.0.1.12.1-linux-x64 early build with generational shenandoah

[earthling-amzn]

Hi - thank you for trying out the generational mode.

TL;DR

Try increasing -XX:ShenandoahTenuredRegionUsageBias from it's default of 192 to it's maximum of 256.

Explanation

The generational mode for Shenandoah works a little differently from other collectors. It only ages objects which belong to regions that are selected for collection. This means that not all objects will age on every cycle. Also, just like single generation mode, it will evacuate objects from the collection set into regions toward the opposite end of the heap from where new objects are allocated. Over time, these regions in the back of the heap will tend to fill with surviving objects which makes them less likely to be included in the collection set (and so, less likely to be promoted). For this reason, we bias these regions to be included in the collection set. The bias makes them appear to have more garbage than they really do by scaling the actual garbage with a configurable multiplier. This flag is called: ShenandoahTenuredRegionUsageBias and the default is 192 (the application of the multiplier avoids floating point math, so it should be expressed as a whole number). The value of this flag is effectively divided by 128, so the default setting scales the apparent garbage by (192/128) ^ (region age - tenuring age). Please let us know if changing this results in more objects being promoted. I'd also like to know if this slower promotion rate is causing other problems for your application? or is the collector just not behaving like you expected?

[chevaris]

Hi - Thanks for the fast response and also thanks for working on this feature (I think could be very relevant for some use cases)

OBSERVATIONS WITH NEW PARAMETER (1GB Young Gen, 6GB total heap)

I do not observe any significant difference after changing the parameter. Basically Young Gen is completely exhausted with referenced objects while Old Gen is almost empty. Neglictable promotion only happens when Young gen is almost exhausted

If you see the options that I pasted in the ticket I am using 6GB of heap and for the purpose of the test I provided 1GB to the Young Gen.

Regarding application behavior, it works as expected from a functional point of view. Problems are not functional:

• In practice available memory is just 1GB. Basically the app is an memory DB, so this is a problem because the capacity is dramatically reduced.
• The JVM is using tons of cycles to collect Young Gen. Actually when Young is under heavy pressure GC is taking most of the CPU. Basically Young Generation cycles are back to back, but GC is not able to collect almost nothing because all the memory is referenced and contains objects that should be in the Old Gen.

From a functional perspective, notice that -XX:InitialTenuringThreshold is set to 5, and looks that GC is not honoring that option.

OBSERVATIONS WITH DIFFERENT CONFIGURATION (2GB Young Gen, 6GB total heap)

I made a second try increasing the Young Gen size to 2GB (Young Gen 33% memory - Old Gen 66% aprox) with the parameter recommended, and it shows the same behavior. No promotion at all to Old Gen until Young Gen is exhausted, and when promotion is done is neglictable.

After 30 mins and Young Gen completely exhausted only 5MB promoted to Old Gen

I attache figures from jconsole

Hope it helps.

[earthling-amzn]

Thank you for trying the parameter changes. We've noticed that generational mode doesn't promote objects at the same rate as other collectors and I agree with you that it's not what we want here. We'll work on adding more visibility into the decisions to promote (or not to promote) and we'll also develop mechanisms to increase the promotion rate (or increase the size of young gen).

[chevaris]

Thanks. Just let me know if I can help testing when you have some new binaries

[benty-amzn]

This should be improved in the latest release. Please reopen this issue if you continue to encounter this behavior.