OpenJDK java/lang/StrictMath/ExhaustingTests OutOfMemoryError: Java heap space

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/42 java/lang/StrictMath/ExhaustingTests.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/42/openjdk_test_output.tar.gz

18:31:08  java.lang.OutOfMemoryError: Java heap space
18:31:08    at java.base/java.lang.Thread.getThreads(Native Method)
18:31:08    at java.base/java.lang.Thread.getAllThreads(Thread.java:2632)
18:31:08    at java.base/java.lang.ThreadGroup.activeCount(ThreadGroup.java:377)
18:31:08    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMThreadGroup.liveThreads(MainActionHelper.java:443)
18:31:08    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMThreadGroup.cleanup(MainActionHelper.java:401)
18:31:08    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMThreadGroup.access$000(MainActionHelper.java:366)
18:31:08    at com.sun.javatest.regtest.agent.MainActionHelper.runClass(MainActionHelper.java:235)
18:31:08    at com.sun.javatest.regtest.agent.AgentServer.doMain(AgentServer.java:317)
18:31:08    at com.sun.javatest.regtest.agent.AgentServer.run(AgentServer.java:242)
18:31:08    at com.sun.javatest.regtest.agent.AgentServer.main(AgentServer.java:71)

Perhaps related to https://github.com/eclipse-openj9/openj9/issues/17910

[pshipton]

@dmitripivkine can you pls look and see what's keeping the heap full.

[dmitripivkine]

There is no system core unfortunately, snap traces and javacore only. I will take a look but not sure is there something useful.

[dmitripivkine]

There is excessive GC case:

3STHSTTYPE     22:30:56:639561502 GMT j9mm.101 -   J9AllocateIndexableObject() returning NULL! 184 bytes requested for object of class 000003FF902E0500 from memory space 'Flat' id=000003FF900B65D0
3STHSTTYPE     22:30:56:639561032 GMT j9mm.84 -   Forcing J9AllocateIndexableObject() to fail due to excessive GC
3STHSTTYPE     22:30:56:639355452 GMT j9mm.82 -   Excessive GC raised!
3STHSTTYPE     22:30:56:639353659 GMT j9mm.468 -   Cycle End: type 3 approximateFreeMemorySize 497025024
3STHSTTYPE     22:30:56:639305006 GMT j9mm.346 -   PGC end: workstackoverflow=0 overflowcount=0 heapFree=497025024/536870912

[dmitripivkine]

Taking this item to GC queue. We need to double check Excessive GC condition for Balanced. In one hand there is very little memory can be released by PGC (Copy Forward) but from another hand there is no attempt to perform GMP.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_OMR_testList_0/27 jdk_lang_j9_0 java/lang/StrictMath/ExhaustingTests.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_s390x_linux_OMR_testList_0/27/openjdk_test_output.tar.gz

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Release_testList_0/21 - ub20-390-6 jdk_lang_j9_0 java/lang/StrictMath/ExhaustingTests.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Release_testList_0/21/openjdk_test_output.tar.gz

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/39

[dmitripivkine]

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/39

This failure might be the same as https://github.com/eclipse-openj9/openj9/issues/19831. Excessive GC with Balanced due Eden size seems reduced to 2 regions:

06:00:54.272267961                 0x0       j9mm.346  Event       PGC end: workstackoverflow=0 overflowcount=0 heapFree=457179136/536870912
06:00:54.272368365                 0x0       j9mm.468  Event       Cycle End: type 3 approximateFreeMemorySize 457179136
06:00:54.272373965                 0x0       j9mm.82   Event       Excessive GC raised!
06:00:54.272400530      *0x3fef1092a00       j9mm.294  Entry      >MM_SchedulingDelegate_getNextTaxationThreshold
06:00:54.272400808       0x3fef1092a00       j9mm.295  Exit       <MM_SchedulingDelegate_getNextTaxationThreshold index=7516, Eden=1048576, threshold=1048576, doGlobal=0, doPGC=1

There is no system core preserved, trying to find details from javacore/snap traces.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_Nightly_testList_0/340

[pshipton]

There will never be a system core as the tests run with -Xdump:system:none. Some of the tests intentionally cause OOM. We'd have to grind it and repeat the problem without this option.