search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

testSoftMxDisclaimMemory Memory didn't release back to OS #10732

Open pshipton opened 4 years ago

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk15_j9_extended.functional_ppc64le_linux_Nightly_testList_0/57 testSoftMxDisclaimMemory_2 ub18-ppcle-1 variation: Mode601 JVM_OPTIONS: -Xcompressedrefs -Xjit -Xgcpolicy:optthruput 

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3416391680 is less than or equal to previous memory size 3401515008
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3416588288 is less than or equal to previous memory size 3416391680
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2020/09/27 02:01:57 - please wait.
...
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.management/com.ibm.java.lang.management.internal.MemoryMXBeanImpl.getHeapMemoryUsageImpl(Native Method)
    at java.management/com.ibm.java.lang.management.internal.MemoryMXBeanImpl.getHeapMemoryUsage(MemoryMXBeanImpl.java:390)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:137)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.TestNG.privateMain(TestNG.java:1354)
    at org.testng.TestNG.main(TestNG.java:1323)
pshipton commented 4 years ago

@dmitripivkine fyi

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk15_j9_extended.functional_ppc64le_linux_Nightly_testList_0/58 ub16-ppcle-1

pshipton commented 4 years ago

First occurrence: OpenJ9: 150f183 OMR: 79c250b OpenJDK: 39a43ea

Previous build: OpenJ9: 150f183 OMR: 79c250b OpenJDK: 39a43ea

dmitripivkine commented 4 years ago

First occurrence: OpenJ9: 150f183 OMR: 79c250b OpenJDK: 39a43ea

Previous build: OpenJ9: 150f183 OMR: 79c250b OpenJDK: 39a43ea

looks like mentioned builds are the same

dmitripivkine commented 4 years ago

Softmx option works: according GC verbose log heap contraction occur as expected

Screen Shot 2020-09-29 at 4 26 22 PM
pshipton commented 4 years ago

looks like mentioned builds are the same

Yup. Seems we just got unlucky a couple of times, this didn't re-occur last night.

dmitripivkine commented 4 years ago

Test checks actual memory available in OS. It is sensitive to running environment. We use to have similar failures in the past.

dmitripivkine commented 4 years ago

Also there are OOM "heap memory" at the end of the test. Test instructed to contract heap to ~84m:

1STHEAPTYPE    Object Memory
NULL           id                 start              end                size               space/region
1STHEAPSPACE   0x000078D3E40BFF50         --                 --                 --         Flat
1STHEAPREGION  0x000078D3E40C0020 0x00000000C0000000 0x00000000C5110000 0x0000000005110000 Flat/Tenured Region

and this seems not enough for test + test framework itself

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk15_j9_extended.functional_s390x_linux_Nightly/71/ testSoftMxDisclaimMemory_zlinux_64_2 ub16-390-2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4439527424 is less than or equal to previous memory size 4420001792
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4438700032 is less than or equal to previous memory size 4438413312
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2020/10/08 08:06:21 - please wait.
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk15_j9_extended.functional_ppc64le_linux_Release_testList_0/14 testSoftMxDisclaimMemory_2 ub16-ppcle-1

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 1057488896 is less than or equal to previous memory size 1055850496
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 1058471936 is less than or equal to previous memory size 1057488896
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/01/02 17:07:39 - please wait.
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk15_j9_extended.functional_ppc64le_linux_Nightly_testList_0/141/ testSoftMxDisclaimMemory_2 ub18-ppcle-2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4031512576 is less than or equal to previous memory size 4026466304
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4033150976 is less than or equal to previous memory size 4031512576
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2020/12/30 02:17:51 - please wait.
dmitripivkine commented 3 years ago

@llxia FYI: Test has requested heap contraction and it succeeds. However seems size of heap after contraction is not enough to complete test framework run, so we got OOM at the very end of test. The solution might be setting watermark higher. Alternatively heap size can be restored to original as soon as requested contraction is detected. However an idea that test expected that released from contracted object heap memory should be seen in OS Free is extremely unreliable.

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk15_j9_extended.functional_ppc64le_linux_Nightly_testList_1/155 testSoftMxDisclaimMemory_2 ub18-ppcle-2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4984340480 is less than or equal to previous memory size 4974575616
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4984143872 is less than or equal to previous memory size 4984340480
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/01/13 03:03:08 - please wait.
llxia commented 3 years ago

Yes, this test is very unreliable. I will try to see if I can adjust it.

lumpfish commented 3 years ago

Seen on openj9-0.26.0-m2 build:

openjdk version "1.8.0_292-internal"
OpenJDK Runtime Environment (build 1.8.0_292-internal-202104050821-b09)
Eclipse OpenJ9 VM (build openj9-0.26.0-m2, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20210405_990 (JIT enabled, AOT enabled)
OpenJ9   - 4ce9ec8a1
OMR      - b8996ac2c
JCL      - e21e76d905 based on jdk8u292-b09)
12:51:57  ===============================================
12:51:57  Running test testSoftMxDisclaimMemory_2 ...
12:51:57  ===============================================
12:51:57  testSoftMxDisclaimMemory_2 Start Time: Mon Apr  5 11:51:55 2021 Epoch Time (ms): 1617623515637
12:51:57  "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
12:51:57  JVMSHRC005I No shared class caches available
12:51:57  JVMSHRC005I No shared class caches available
12:51:57  cache cleanup done
12:51:57  variation: Mode601
12:51:57  JVM_OPTIONS:  -Xcompressedrefs -Xjit -Xgcpolicy:optthruput 
12:51:57  { itercnt=1; \
12:51:57  mkdir -p "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/output_16176134823666/testSoftMxDisclaimMemory_2"; \
12:51:57  cd "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/output_16176134823666/testSoftMxDisclaimMemory_2"; \
12:51:57  export LD_LIBRARY_PATH="/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdkbinary/openjdk-test-image/openj9:/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdkbinary/j2sdk-image/jre/bin/../lib/amd64/default:/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdkbinary/j2sdk-image/jre/bin/../lib/amd64/j9vm:"; \
12:51:57  "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdkbinary/j2sdk-image/bin/java"  -Xcompressedrefs -Xjit -Xgcpolicy:optthruput  \
12:51:57  -Xmx1024m -Xsoftmx512m -Xmn1m -XX:+DisclaimVirtualMemory -verbose:gc -Xverbosegclog:"/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/output_16176134823666/testSoftMxDisclaimMemory_2"/vgc.log \
12:51:57  -cp "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../../jvmtest/TestConfig/resources:/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/lib/testng.jar:/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/lib/jcommander.jar:/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../../jvmtest/functional/JLM_Tests/jlm_tests.jar" \
12:51:57  org.testng.TestNG -d "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/output_16176134823666/testSoftMxDisclaimMemory_2" "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../../jvmtest/functional/JLM_Tests/testng.xml" \
12:51:57  -testnames testSoftMxDisclaimMemory \
12:51:57  -groups level.extended \
12:51:57  -excludegroups d.*.linux_x86-64_mxdptrs,d.*.arch.x86,d.*.os.linux,d.*.bits.64,d.*.generic-all; \
12:51:57  if [ $? -eq 0 ] ; then echo ""; echo "testSoftMxDisclaimMemory_2""_PASSED"; echo ""; cd /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/..; rm -f -r "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/output_16176134823666/testSoftMxDisclaimMemory_2"; else echo ""; echo "testSoftMxDisclaimMemory_2""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TKG/output_16176134823666/TestTargetResult";
12:51:57  [IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /home/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_linux/openjdk-tests/TKG/../TestConfig/resources/excludes/latest_exclude_8.txt
12:51:57  ...
12:51:57  ... TestNG 6.14.2 by C?dric Beust (cedric@beust.com)
12:51:57  ...
12:51:57  
12:53:43  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 50376044544 is less than or equal to previous memory size 50601791488
12:53:51  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 50166038528 is less than or equal to previous memory size 50165157888
12:53:58  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 50121924608 is less than or equal to previous memory size 50166038528
12:54:06  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 50101964800 is less than or equal to previous memory size 50121924608
12:54:13  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 50076467200 is less than or equal to previous memory size 50101964800
12:54:13  FAILED: testDisclaimMemoryEffects(5)
12:54:13  java.lang.AssertionError: Test has failed after 5 attempts
12:54:13    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
12:54:13    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
12:54:13    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
12:54:13    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:54:13    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:54:13    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:54:13    at java.lang.reflect.Method.invoke(Method.java:498)
12:54:13    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
12:54:13    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
12:54:13    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
12:54:13    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
12:54:13    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
12:54:13    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
12:54:13    at org.testng.TestRunner.privateRun(TestRunner.java:648)
12:54:13    at org.testng.TestRunner.run(TestRunner.java:505)
12:54:13    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
12:54:13    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
12:54:13    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
12:54:13    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
12:54:13    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
12:54:13    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
12:54:13    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
12:54:13    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
12:54:13    at org.testng.TestNG.runSuites(TestNG.java:1049)
12:54:13    at org.testng.TestNG.run(TestNG.java:1017)
12:54:13    at org.testng.TestNG.privateMain(TestNG.java:1354)
12:54:13    at org.testng.TestNG.main(TestNG.java:1323)
12:54:13  
12:54:13  
12:54:13  ===============================================
12:54:13      testSoftMxDisclaimMemory
12:54:13      Tests run: 1, Failures: 1, Skips: 0
12:54:13  ===============================================
andrew-m-leonard commented 3 years ago

@pshipton occurrence on GA build: https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.functional_aarch64_linux/141/consoleFull

llxia commented 3 years ago

@OscarQQ could you take a look? Thanks

pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.functional_s390x_linux_Nightly_testList_0/38 - ub18-390-1 testDefaultDisclaimMemory_zlinux_1 JVM_OPTIONS: -Xgcpolicy:optavgpause -Xjit -Xnocompressedrefs 

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3191304192 is less than or equal to previous memory size 3322134528
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2679894016 is less than or equal to previous memory size 3191304192
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2363199488 is less than or equal to previous memory size 2679894016
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2176286720 is less than or equal to previous memory size 2363199488
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 1693995008 is less than or equal to previous memory size 2176286720
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
JasonFengJ9 commented 3 years ago

Observed in an internal build Test_openjdk17_j9_extended.functional_s390x_linux_Personal/34/tapResults/ (rhel7s390x-3-7)

09:01:55  openjdk version "17-internal" 2021-09-14
09:01:55  OpenJDK Runtime Environment (build 17-internal+0-adhoc.jenkins.BuildJDK17s390xlinuxPersonal)
09:01:55  Eclipse OpenJ9 VM (build v0.28.0-release-1b582d44560, JRE 17 Linux s390x-64-Bit Compressed References 20210915_44 (JIT enabled, AOT enabled)
09:01:55  OpenJ9   - 1b582d44560
09:01:55  OMR      - 377314cb0c7
09:01:55  JCL      - 41171df43e2 based on jdk-17+35)

===============================================
Running test testSoftMxDisclaimMemory_zlinux_64_0 ...
===============================================
testSoftMxDisclaimMemory_zlinux_64_0 Start Time: Wed Sep 15 06:25:46 2021 Epoch Time (ms): 1631712346253
variation: Mode110
JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 164184064 is less than or equal to previous memory size 194506752
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 121544704 is less than or equal to previous memory size 162086912
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 55619584 is less than or equal to previous memory size 120528896
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 45568000 is less than or equal to previous memory size 55619584
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.TestNG.privateMain(TestNG.java:1354)
    at org.testng.TestNG.main(TestNG.java:1323)

===============================================
    testSoftMxDisclaimMemory
    Tests run: 1, Failures: 1, Skips: 0
===============================================

testSoftMxDisclaimMemory_zlinux_64_0_FAILED
dmitripivkine commented 3 years ago

the only check we can do here to differentiate possible JVM problem from environment behaviour it look to gc verbose log to confirm heap is contracted actually. I believe these tests have been improved to store log file. I will check it

dmitripivkine commented 3 years ago
Screen Shot 2021-09-15 at 5 27 32 PM

Heap has been contracted below 256m as requested, however test can not see this memory returned back to OS. I think this is environment problem. In general there is no guarantee memory is going to be visible as free, OS or Hypervisor might not release it back immediately.

pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0/106 - win2012-x86-10 testSoftMxNotDisclaimMemory_3 -Xjit -Xgcpolicy:balanced -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0/106/functional_test_output.tar.gz

02:03:40  [SoftmxAdvanceTest] [WARN]    Generate Java core dump after forcing GC.
02:03:40  JVMDUMP034I User requested Java dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\javacore.20210923.010325.2916.0001.txt' through com.ibm.jvm.Dump.JavaDump
02:03:40  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\javacore.20210923.010325.2916.0001.txt
02:03:40  JVMDUMP034I User requested Heap dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\heapdump.20210923.010325.2916.0002.phd' through com.ibm.jvm.Dump.HeapDump
02:03:40  JVMDUMP010I Heap dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\heapdump.20210923.010325.2916.0002.phd
02:03:40  JVMDUMP034I User requested System dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\core.20210923.010325.2916.0003.dmp' through com.ibm.jvm.Dump.SystemDump
02:03:40  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\core.20210923.010325.2916.0003.dmp
02:03:40  [SoftmxAdvanceTest] [ERROR]   FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
02:08:43  [SoftmxAdvanceTest] [WARN]    Generate Java core dump after forcing GC.
02:08:43  JVMDUMP034I User requested Java dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\javacore.20210923.010828.2916.0004.txt' through com.ibm.jvm.Dump.JavaDump
02:08:43  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\javacore.20210923.010828.2916.0004.txt
02:08:43  JVMDUMP034I User requested Heap dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\heapdump.20210923.010828.2916.0005.phd' through com.ibm.jvm.Dump.HeapDump
02:08:43  JVMDUMP010I Heap dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\heapdump.20210923.010828.2916.0005.phd
02:08:43  JVMDUMP034I User requested System dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\core.20210923.010828.2916.0006.dmp' through com.ibm.jvm.Dump.SystemDump
02:08:43  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\core.20210923.010828.2916.0006.dmp
02:08:43  [SoftmxAdvanceTest] [ERROR]   FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
02:08:43  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory shouldn't release back to OS! Post memory size 5721952256 is greater than previous memory size 5183815680
02:08:44  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory shouldn't release back to OS! Post memory size 5991362560 is greater than previous memory size 5721952256
02:13:47  [SoftmxAdvanceTest] [WARN]    Generate Java core dump after forcing GC.
02:13:47  JVMDUMP034I User requested Java dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\javacore.20210923.011344.2916.0007.txt' through com.ibm.jvm.Dump.JavaDump
02:13:47  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\javacore.20210923.011344.2916.0007.txt
02:13:47  JVMDUMP034I User requested Heap dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\heapdump.20210923.011344.2916.0008.phd' through com.ibm.jvm.Dump.HeapDump
02:13:47  JVMDUMP010I Heap dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\heapdump.20210923.011344.2916.0008.phd
02:13:47  JVMDUMP034I User requested System dump using 'F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\core.20210923.011344.2916.0009.dmp' through com.ibm.jvm.Dump.SystemDump
02:13:47  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323720753856\testSoftMxNotDisclaimMemory_3\core.20210923.011344.2916.0009.dmp
02:13:47  [SoftmxAdvanceTest] [ERROR]   FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
02:13:47  FAILED: testDisclaimMemoryEffects(5)
02:13:47  java.lang.AssertionError: Test has failed after 5 attempts
02:13:47    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
02:13:47    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
02:13:47    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
02:13:47    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
02:13:47    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
02:13:47    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
02:13:47    at java.lang.reflect.Method.invoke(Method.java:498)
02:13:47    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
02:13:47    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
02:13:47    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
02:13:47    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
02:13:47    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
02:13:47    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
02:13:47    at org.testng.TestRunner.privateRun(TestRunner.java:648)
02:13:47    at org.testng.TestRunner.run(TestRunner.java:505)
02:13:47    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
02:13:47    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
02:13:47    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
02:13:47    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
02:13:47    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
02:13:47    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
02:13:47    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
02:13:47    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
02:13:47    at org.testng.TestNG.runSuites(TestNG.java:1049)
02:13:47    at org.testng.TestNG.run(TestNG.java:1017)
02:13:47    at org.testng.TestNG.privateMain(TestNG.java:1354)
02:13:47    at org.testng.TestNG.main(TestNG.java:1323)
pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0/106 - win2012-x86-7 testSoftMxNotDisclaimMemory_3 -Xjit -Xgcpolicy:balanced -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0/106/functional_test_output.tar.gz

03:43:34  [SoftmxAdvanceTest] [WARN]    Generate Java core dump after forcing GC.
03:43:34  JVMDUMP034I User requested Java dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\javacore.20210923.024318.7296.0001.txt' through com.ibm.jvm.Dump.JavaDump
03:43:34  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\javacore.20210923.024318.7296.0001.txt
03:43:34  JVMDUMP034I User requested Heap dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\heapdump.20210923.024318.7296.0002.phd' through com.ibm.jvm.Dump.HeapDump
03:43:34  JVMDUMP010I Heap dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\heapdump.20210923.024318.7296.0002.phd
03:43:34  JVMDUMP034I User requested System dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\core.20210923.024318.7296.0003.dmp' through com.ibm.jvm.Dump.SystemDump
03:43:34  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\core.20210923.024318.7296.0003.dmp
03:43:34  [SoftmxAdvanceTest] [ERROR]   FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
03:48:37  [SoftmxAdvanceTest] [WARN]    Generate Java core dump after forcing GC.
03:48:37  JVMDUMP034I User requested Java dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\javacore.20210923.024821.7296.0004.txt' through com.ibm.jvm.Dump.JavaDump
03:48:37  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\javacore.20210923.024821.7296.0004.txt
03:48:37  JVMDUMP034I User requested Heap dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\heapdump.20210923.024821.7296.0005.phd' through com.ibm.jvm.Dump.HeapDump
03:48:37  JVMDUMP010I Heap dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\heapdump.20210923.024821.7296.0005.phd
03:48:37  JVMDUMP034I User requested System dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\core.20210923.024821.7296.0006.dmp' through com.ibm.jvm.Dump.SystemDump
03:48:37  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\core.20210923.024821.7296.0006.dmp
03:48:37  [SoftmxAdvanceTest] [ERROR]   FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
03:48:37  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory shouldn't release back to OS! Post memory size 5679521792 is greater than previous memory size 5074493440
03:48:39  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory shouldn't release back to OS! Post memory size 5947752448 is greater than previous memory size 5679521792
03:53:41  [SoftmxAdvanceTest] [WARN]    Generate Java core dump after forcing GC.
03:53:41  JVMDUMP034I User requested Java dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\javacore.20210923.025337.7296.0007.txt' through com.ibm.jvm.Dump.JavaDump
03:53:41  JVMDUMP010I Java dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\javacore.20210923.025337.7296.0007.txt
03:53:41  JVMDUMP034I User requested Heap dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\heapdump.20210923.025337.7296.0008.phd' through com.ibm.jvm.Dump.HeapDump
03:53:41  JVMDUMP010I Heap dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\heapdump.20210923.025337.7296.0008.phd
03:53:41  JVMDUMP034I User requested System dump using 'F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\core.20210923.025337.7296.0009.dmp' through com.ibm.jvm.Dump.SystemDump
03:53:41  JVMDUMP010I System dump written to F:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly_testList_0\aqa-tests\TKG\output_16323772561385\testSoftMxNotDisclaimMemory_3\core.20210923.025337.7296.0009.dmp
03:53:41  [SoftmxAdvanceTest] [ERROR]   FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
03:53:41  FAILED: testDisclaimMemoryEffects(5)
03:53:41  java.lang.AssertionError: Test has failed after 5 attempts
03:53:41    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
03:53:41    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
03:53:41    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
03:53:41    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
03:53:41    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
03:53:41    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
03:53:41    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
03:53:41    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
03:53:41    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
03:53:41    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
03:53:41    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
03:53:41    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
03:53:41    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
03:53:41    at org.testng.TestRunner.privateRun(TestRunner.java:648)
03:53:41    at org.testng.TestRunner.run(TestRunner.java:505)
03:53:41    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
03:53:41    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
03:53:41    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
03:53:41    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
03:53:41    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
03:53:41    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
03:53:41    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
03:53:41    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
03:53:41    at org.testng.TestNG.runSuites(TestNG.java:1049)
03:53:41    at org.testng.TestNG.run(TestNG.java:1017)
03:53:41    at org.testng.TestNG.privateMain(TestNG.java:1354)
03:53:41    at org.testng.TestNG.main(TestNG.java:1323)
pshipton commented 3 years ago

@dmitripivkine this failed on both jdk8 and 11 last night, may be a new problem.

pshipton commented 3 years ago

Note the open jdk17 builds are running the 0.28 release stream and not testing head.

dmitripivkine commented 3 years ago

@dmitripivkine this failed on both jdk8 and 11 last night, may be a new problem.

I checked both failures - heap is contracted as expected

pshipton commented 3 years ago

We got one on jdk17 (0.28 release stream) as well, in the first open testing on aarch64.

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/1 testSoftMxDisclaimMemory_2 -Xcompressedrefs -Xjit -Xgcpolicy:optthruput

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/1/functional_test_output.tar.gz

[SoftmxAdvanceTest] [WARN]  Generate Java core dump after forcing GC.
JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/javacore.20210923.022704.107685.0001.txt' through com.ibm.jvm.Dump.JavaDump
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/javacore.20210923.022704.107685.0001.txt
JVMDUMP034I User requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/heapdump.20210923.022704.107685.0002.phd' through com.ibm.jvm.Dump.HeapDump
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/heapdump.20210923.022704.107685.0002.phd
JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/core.20210923.022704.107685.0003.dmp' through com.ibm.jvm.Dump.SystemDump
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/core.20210923.022704.107685.0003.dmp
[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3472621568 is less than or equal to previous memory size 3475177472
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/09/23 02:27:28 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/core.20210923.022728.107685.0004.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/core.20210923.022728.107685.0004.dmp
JVMDUMP032I JVM requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/heapdump.20210923.022728.107685.0005.phd' in response to an event
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/heapdump.20210923.022728.107685.0005.phd
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/javacore.20210923.022728.107685.0006.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/javacore.20210923.022728.107685.0006.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/Snap.20210923.022728.107685.0007.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_1/aqa-tests/TKG/output_16323741006436/testSoftMxDisclaimMemory_2/Snap.20210923.022728.107685.0007.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1045)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:990)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:685)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.functional_x86-64_windows_Nightly_testList_0/107 - win2012-x86-2 https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_x86-64_windows_Nightly/107 - win2012-x86-8 testSoftMxNotDisclaimMemory_3 -Xjit -Xgcpolicy:balanced -Xnocompressedrefs

@dmitripivkine I have to think this is related to the recent GC change when it occurs every night now. You might run a grinder before and after the change.

pshipton commented 3 years ago

This continues to occur in every head stream build/test, but did not occur in the 0.29 build which doesn't have the recent GC change. Set as a blocker for the 0.30 release.

dmitripivkine commented 3 years ago

I just double check again: Heap has been contracted as requested

Screen Shot 2021-09-27 at 9 52 28 AM

The difference between old and new behaviour before heap allowed to expand above 512m (because -Xmx1024m), now it never exceeds softmx level

I think we should double check test for expected memory usage changes first

@pshipton Is this test failed with Balanced only?

JasonFengJ9 commented 3 years ago

Observed in an internal build job/Test_openjdk17_j9_extended.functional_x86-64_windows/49/ (win16x64rt-1-8).

openjdk version "17" 2021-09-14
IBM Semeru Runtime Open Edition 17.0.0.0-m1 (build 17+35)
Eclipse OpenJ9 VM 17.0.0.0-m1 (build master-d6256c10f, JRE 17 Windows Server 2016 amd64-64-Bit Compressed References 20210925_38 (JIT enabled, AOT enabled)
OpenJ9   - d6256c10f
OMR      - 82faf49a1
JCL      - ff6a3e831b based on jdk-17+35)

[2021-09-25T23:28:11.713Z] Running test testSoftMxNotDisclaimMemory_3 ...
[2021-09-25T23:28:12.439Z] ===============================================
[2021-09-25T23:28:12.439Z] testSoftMxNotDisclaimMemory_3 Start Time: Sat Sep 25 16:28:11 2021 Epoch Time (ms): 1632612491906
[2021-09-25T23:28:12.439Z] "C:/Users/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_windows/openjdkbinary/j2sdk-image\\bin\\java" -Xshareclasses:destroyAll; "C:/Users/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_windows/openjdkbinary/j2sdk-image\\bin\\java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
[2021-09-25T23:28:13.947Z] JVMSHRC005I No shared class caches available
[2021-09-25T23:28:14.683Z] JVMSHRC005I No shared class caches available
[2021-09-25T23:28:14.684Z] cache cleanup done
[2021-09-25T23:28:14.684Z] variation: Mode501
[2021-09-25T23:28:15.412Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:balanced -Xnocompressedrefs 

[2021-09-25T23:44:23.792Z] [SoftmxAdvanceTest] [ERROR]  FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[2021-09-25T23:44:23.792Z] FAILED: testDisclaimMemoryEffects(5)
[2021-09-25T23:44:23.792Z] java.lang.AssertionError: Test has failed after 5 attempts
[2021-09-25T23:44:23.792Z]  at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
[2021-09-25T23:44:23.792Z]  at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
[2021-09-25T23:44:23.792Z]  at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
[2021-09-25T23:44:23.792Z]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2021-09-25T23:44:23.792Z]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[2021-09-25T23:44:23.792Z]  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-09-25T23:44:23.792Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[2021-09-25T23:44:23.792Z]  at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
[2021-09-25T23:44:23.792Z]  at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
[2021-09-25T23:44:23.792Z]  at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
[2021-09-25T23:44:23.792Z]  at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
[2021-09-25T23:44:23.792Z]  at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
[2021-09-25T23:44:23.792Z]  at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
[2021-09-25T23:44:23.792Z]  at org.testng.TestRunner.privateRun(TestRunner.java:648)
[2021-09-25T23:44:23.792Z]  at org.testng.TestRunner.run(TestRunner.java:505)
[2021-09-25T23:44:23.792Z]  at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
[2021-09-25T23:44:23.792Z]  at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
[2021-09-25T23:44:23.792Z]  at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
[2021-09-25T23:44:23.792Z]  at org.testng.SuiteRunner.run(SuiteRunner.java:364)
[2021-09-25T23:44:23.792Z]  at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[2021-09-25T23:44:23.792Z]  at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
[2021-09-25T23:44:23.792Z]  at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
[2021-09-25T23:44:23.792Z]  at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
[2021-09-25T23:44:23.792Z]  at org.testng.TestNG.runSuites(TestNG.java:1049)
[2021-09-25T23:44:23.792Z]  at org.testng.TestNG.run(TestNG.java:1017)
[2021-09-25T23:44:23.792Z]  at org.testng.TestNG.privateMain(TestNG.java:1354)
[2021-09-25T23:44:23.792Z]  at org.testng.TestNG.main(TestNG.java:1323)
[2021-09-25T23:44:23.792Z] 
[2021-09-25T23:44:23.792Z] 
[2021-09-25T23:44:23.792Z] ===============================================
[2021-09-25T23:44:23.792Z]     testSoftMxDisclaimMemory
[2021-09-25T23:44:23.792Z]     Tests run: 1, Failures: 1, Skips: 0
[2021-09-25T23:44:23.792Z] ===============================================
[2021-09-25T23:44:23.792Z] 
[2021-09-25T23:44:23.792Z] 
[2021-09-25T23:44:23.792Z] ===============================================
[2021-09-25T23:44:23.792Z] JLM_Tests_Suite
[2021-09-25T23:44:23.792Z] Total tests run: 1, Failures: 1, Skips: 0
[2021-09-25T23:44:23.792Z] ===============================================
[2021-09-25T23:44:23.792Z] 
[2021-09-25T23:44:24.523Z] 
[2021-09-25T23:44:24.523Z] testSoftMxNotDisclaimMemory_3_FAILED

This is in JDK17 head w/ -Xgcpolicy:balanced.

pshipton commented 3 years ago

Is this test failed with Balanced only?

Yes, failing nightly only in mode 3 which uses -Xgcpolicy:balanced

pshipton commented 3 years ago

Sounds like the test needs to be updated then?

pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/36 testSoftMxDisclaimMemory_2 -Xcompressedrefs -Xjit -Xgcpolicy:optthruput

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/36/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4692049920 is less than or equal to previous memory size 4685627392
[SoftmxAdvanceTest] [WARN]  Generate Java core dump after forcing GC.
JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/javacore.20210927.235735.29477.0001.txt' through com.ibm.jvm.Dump.JavaDump
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/javacore.20210927.235735.29477.0001.txt
JVMDUMP034I User requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/heapdump.20210927.235735.29477.0002.phd' through com.ibm.jvm.Dump.HeapDump
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/heapdump.20210927.235735.29477.0002.phd
JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/core.20210927.235736.29477.0003.dmp' through com.ibm.jvm.Dump.SystemDump
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/core.20210927.235736.29477.0003.dmp
[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/09/27 23:57:55 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/core.20210927.235755.29477.0004.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/core.20210927.235755.29477.0004.dmp
JVMDUMP032I JVM requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/heapdump.20210927.235755.29477.0005.phd' in response to an event
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/heapdump.20210927.235755.29477.0005.phd
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/javacore.20210927.235755.29477.0006.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/javacore.20210927.235755.29477.0006.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/Snap.20210927.235755.29477.0007.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/aqa-tests/TKG/output_16327962942855/testSoftMxDisclaimMemory_2/Snap.20210927.235755.29477.0007.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1045)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:990)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:685)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.TestNG.privateMain(TestNG.java:1354)
    at org.testng.TestNG.main(TestNG.java:1323)
pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/2 - cent8-aarch64-1 testSoftMxDisclaimMemory_2 -Xcompressedrefs -Xjit -Xgcpolicy:optthruput

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/2/functional_test_output.tar.gz

[SoftmxAdvanceTest] [WARN]  Generate Java core dump after forcing GC.
JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/javacore.20211010.052259.300119.0001.txt' through com.ibm.jvm.Dump.JavaDump
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/javacore.20211010.052259.300119.0001.txt
JVMDUMP034I User requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/heapdump.20211010.052259.300119.0002.phd' through com.ibm.jvm.Dump.HeapDump
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/heapdump.20211010.052259.300119.0002.phd
JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/core.20211010.052259.300119.0003.dmp' through com.ibm.jvm.Dump.SystemDump
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/core.20211010.052259.300119.0003.dmp
[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4000514048 is less than or equal to previous memory size 3996712960
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3999924224 is less than or equal to previous memory size 3999531008
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/10/10 05:23:24 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/core.20211010.052324.300119.0004.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/core.20211010.052324.300119.0004.dmp
JVMDUMP032I JVM requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/heapdump.20211010.052324.300119.0005.phd' in response to an event
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/heapdump.20211010.052324.300119.0005.phd
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/javacore.20211010.052324.300119.0006.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/javacore.20211010.052324.300119.0006.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/Snap.20211010.052324.300119.0007.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Release_testList_0/aqa-tests/TKG/output_16338534859169/testSoftMxDisclaimMemory_2/Snap.20211010.052324.300119.0007.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1045)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:990)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:685)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/152 - cent8-aarch64-2 testSoftMxDisclaimMemory_1 -Xgcpolicy:optavgpause -Xjit -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/152/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4086824960 is less than or equal to previous memory size 4080140288
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4085252096 is less than or equal to previous memory size 4086824960
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2022/03/20 23:13:31 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/core.20220320.231331.3967505.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/core.20220320.231331.3967505.0001.dmp
JVMDUMP032I JVM requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/heapdump.20220320.231331.3967505.0002.phd' in response to an event
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/heapdump.20220320.231331.3967505.0002.phd
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/javacore.20220320.231331.3967505.0003.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/javacore.20220320.231331.3967505.0003.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/Snap.20220320.231331.3967505.0004.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16478310469979/testSoftMxDisclaimMemory_1/Snap.20220320.231331.3967505.0004.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1046)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:990)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:685)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
JasonFengJ9 commented 2 years ago

Similar error observed at JDK18 milestone 2 build - job/Test_openjdk18_j9_extended.functional_ppc64le_linux_testList_2/31/(p10rhel053)

openjdk version "18" 2022-03-22
IBM Semeru Runtime Open Edition 18.0.1.0-m2 (build 18+37)
Eclipse OpenJ9 VM 18.0.1.0-m2 (build openj9-0.32.0-m2, JRE 18 Linux ppc64le-64-Bit Compressed References 20220406_55 (JIT enabled, AOT enabled)
OpenJ9   - 959fc5030
OMR      - ab24b6666
JCL      - ea5f594899a based on jdk-18+37)

[2022-04-07T04:01:03.913Z] variation: Mode601
[2022-04-07T04:01:03.913Z] JVM_OPTIONS:  -Xcompressedrefs -Xjit -Xgcpolicy:optthruput 

[2022-04-07T04:01:14.270Z] [SoftmxAdvanceTest] [ERROR]  FAIL: Memory shouldn't release back to OS! Post memory size 2858156032 is greater than previous memory size 2649358336
[2022-04-07T04:01:20.705Z] [SoftmxAdvanceTest] [ERROR]  FAIL: Memory shouldn't release back to OS! Post memory size 3048996864 is greater than previous memory size 2858156032
[2022-04-07T04:01:27.714Z] [SoftmxAdvanceTest] [ERROR]  FAIL: Memory shouldn't release back to OS! Post memory size 3167944704 is greater than previous memory size 3048996864
[2022-04-07T04:01:34.995Z] [SoftmxAdvanceTest] [ERROR]  FAIL: Memory shouldn't release back to OS! Post memory size 3243835392 is greater than previous memory size 3167944704
[2022-04-07T04:01:41.844Z] [SoftmxAdvanceTest] [ERROR]  FAIL: Memory shouldn't release back to OS! Post memory size 3436642304 is greater than previous memory size 3243835392
[2022-04-07T04:01:41.844Z] FAILED: testDisclaimMemoryEffects(5)
[2022-04-07T04:01:41.844Z] java.lang.AssertionError: Test has failed after 5 attempts
[2022-04-07T04:01:41.844Z]  at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
[2022-04-07T04:01:41.844Z]  at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
[2022-04-07T04:01:41.844Z]  at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
[2022-04-07T04:01:41.844Z]  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[2022-04-07T04:01:41.844Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:577)
[2022-04-07T04:01:41.844Z]  at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
[2022-04-07T04:01:41.844Z]  at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
[2022-04-07T04:01:41.844Z]  at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
[2022-04-07T04:01:41.844Z]  at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
[2022-04-07T04:01:41.844Z]  at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
[2022-04-07T04:01:41.844Z]  at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
[2022-04-07T04:01:41.844Z]  at org.testng.TestRunner.privateRun(TestRunner.java:648)
[2022-04-07T04:01:41.844Z]  at org.testng.TestRunner.run(TestRunner.java:505)
[2022-04-07T04:01:41.844Z]  at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
[2022-04-07T04:01:41.844Z]  at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
[2022-04-07T04:01:41.844Z]  at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
[2022-04-07T04:01:41.844Z]  at org.testng.SuiteRunner.run(SuiteRunner.java:364)
[2022-04-07T04:01:41.844Z]  at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[2022-04-07T04:01:41.844Z]  at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
[2022-04-07T04:01:41.844Z]  at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
[2022-04-07T04:01:41.844Z]  at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
[2022-04-07T04:01:41.844Z]  at org.testng.TestNG.runSuites(TestNG.java:1049)
[2022-04-07T04:01:41.844Z]  at org.testng.TestNG.run(TestNG.java:1017)
[2022-04-07T04:01:41.844Z]  at org.testng.TestNG.privateMain(TestNG.java:1354)
[2022-04-07T04:01:41.844Z]  at org.testng.TestNG.main(TestNG.java:1323)
[2022-04-07T04:01:41.844Z] 
[2022-04-07T04:01:41.844Z] 
[2022-04-07T04:01:41.844Z] ===============================================
[2022-04-07T04:01:41.844Z]     testSoftMxDisclaimMemory
[2022-04-07T04:01:41.844Z]     Tests run: 1, Failures: 1, Skips: 0
[2022-04-07T04:01:41.844Z] ===============================================
[2022-04-07T04:01:41.844Z] 
[2022-04-07T04:01:41.844Z] 
[2022-04-07T04:01:41.844Z] ===============================================
[2022-04-07T04:01:41.844Z] JLM_Tests_Suite
[2022-04-07T04:01:41.844Z] Total tests run: 1, Failures: 1, Skips: 0
[2022-04-07T04:01:41.844Z] ===============================================
[2022-04-07T04:01:41.844Z] 
[2022-04-07T04:01:41.844Z] 
[2022-04-07T04:01:41.844Z] testSoftMxNotDisclaimMemory_2_FAILED
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/89 - cent8-aarch64-2 testSoftMxDisclaimMemory_2 -Xcompressedrefs -Xjit -Xgcpolicy:optthruput

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/89/functional_test_output.tar.gz

[SoftmxAdvanceTest] [WARN]  Generate Java core dump after forcing GC.
JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/javacore.20220621.234146.971505.0001.txt' through com.ibm.jvm.Dump.JavaDump
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/javacore.20220621.234146.971505.0001.txt
JVMDUMP034I User requested Heap dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/heapdump.20220621.234146.971505.0002.phd' through com.ibm.jvm.Dump.HeapDump
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/heapdump.20220621.234146.971505.0002.phd
JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/core.20220621.234146.971505.0003.dmp' through com.ibm.jvm.Dump.SystemDump
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/core.20220621.234146.971505.0003.dmp
[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2373320704 is less than or equal to previous memory size 2371551232
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2022/06/21 23:42:04 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/core.20220621.234204.971505.0004.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/core.20220621.234204.971505.0004.dmp
JVMDUMP032I JVM requested Heap dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/heapdump.20220621.234204.971505.0005.phd' in response to an event
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/heapdump.20220621.234204.971505.0005.phd
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/javacore.20220621.234204.971505.0006.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/javacore.20220621.234204.971505.0006.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/Snap.20220621.234204.971505.0007.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk18_j9_extended.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_16558670324072/testSoftMxDisclaimMemory_2/Snap.20220621.234204.971505.0007.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1046)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:994)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:687)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
JasonFengJ9 commented 2 years ago 
openjdk version "11.0.16" 2022-07-19
IBM Semeru Runtime Open Edition 11.0.16.0-m2 (build 11.0.16+7)
Eclipse OpenJ9 VM 11.0.16.0-m2 (build openj9-0.33.0-m2, JRE 11 Linux s390x-64-Bit Compressed References 20220712_410 (JIT enabled, AOT enabled)
OpenJ9   - e7fc0d2f1
OMR      - e25299388
JCL      - f20da0ba5b based on jdk-11.0.16+7)

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.functional_s390x_linux_testList_2/253/

[2022-07-12T18:27:15.735Z] variation: Mode501
[2022-07-12T18:27:15.735Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:balanced -Xnocompressedrefs 

[2022-07-12T18:42:52.457Z] FAILED: testDisclaimMemoryEffects(5)
[2022-07-12T18:42:52.457Z] java.lang.AssertionError: Test has failed after 5 attempts
[2022-07-12T18:42:52.457Z]  at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
[2022-07-12T18:42:52.457Z]  at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
[2022-07-12T18:42:52.457Z]  at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
[2022-07-12T18:42:52.457Z]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2022-07-12T18:42:52.457Z]  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2022-07-12T18:42:52.457Z]  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2022-07-12T18:42:52.458Z]  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2022-07-12T18:42:52.458Z]  at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
[2022-07-12T18:42:52.458Z]  at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
[2022-07-12T18:42:52.458Z]  at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
[2022-07-12T18:42:52.458Z]  at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
[2022-07-12T18:42:52.458Z]  at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
[2022-07-12T18:42:52.458Z]  at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
[2022-07-12T18:42:52.458Z]  at org.testng.TestRunner.privateRun(TestRunner.java:648)
[2022-07-12T18:42:52.458Z]  at org.testng.TestRunner.run(TestRunner.java:505)
[2022-07-12T18:42:52.458Z]  at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
[2022-07-12T18:42:52.458Z]  at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
[2022-07-12T18:42:52.458Z]  at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
[2022-07-12T18:42:52.458Z]  at org.testng.SuiteRunner.run(SuiteRunner.java:364)
[2022-07-12T18:42:52.458Z]  at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[2022-07-12T18:42:52.458Z]  at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
[2022-07-12T18:42:52.458Z]  at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
[2022-07-12T18:42:52.458Z]  at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
[2022-07-12T18:42:52.458Z]  at org.testng.TestNG.runSuites(TestNG.java:1049)
[2022-07-12T18:42:52.458Z]  at org.testng.TestNG.run(TestNG.java:1017)
[2022-07-12T18:42:52.458Z]  at org.testng.TestNG.privateMain(TestNG.java:1354)
[2022-07-12T18:42:52.458Z]  at org.testng.TestNG.main(TestNG.java:1323)
[2022-07-12T18:42:52.458Z] 
[2022-07-12T18:42:52.458Z] 
[2022-07-12T18:42:52.458Z] ===============================================
[2022-07-12T18:42:52.458Z]     testSoftMxDisclaimMemory
[2022-07-12T18:42:52.458Z]     Tests run: 1, Failures: 1, Skips: 0
[2022-07-12T18:42:52.458Z] ===============================================
[2022-07-12T18:42:52.458Z] 
[2022-07-12T18:42:52.458Z] 
[2022-07-12T18:42:52.458Z] ===============================================
[2022-07-12T18:42:52.458Z] JLM_Tests_Suite
[2022-07-12T18:42:52.458Z] Total tests run: 1, Failures: 1, Skips: 0
[2022-07-12T18:42:52.458Z] ===============================================
[2022-07-12T18:42:52.458Z] 
[2022-07-12T18:42:52.458Z] 
[2022-07-12T18:42:52.458Z] testSoftMxNotDisclaimMemory_zlinux_64_3_FAILED
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.functional_s390x_linux_Nightly_testList_1/329/ - rh7-390-4 testDefaultDisclaimMemory_zlinux_3 -Xjit -Xgcpolicy:balanced -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk8_j9_extended.functional_s390x_linux_Nightly_testList_1/329/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3889741824 is less than or equal to previous memory size 3926867968
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3876507648 is less than or equal to previous memory size 3889741824
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.functional_s390x_linux_Nightly_testList_1/114 - rh7-390-2 testDefaultDisclaimMemory_zlinux_3

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk18_j9_extended.functional_s390x_linux_Nightly_testList_1/114/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2410889216 is less than or equal to previous memory size 2762022912
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_ppc64le_linux_Nightly_testList_1/363 testDefaultDisclaimMemory_2

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.functional_ppc64le_linux_Nightly_testList_1/363/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2774532096 is less than or equal to previous memory size 2927034368
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2526871552 is less than or equal to previous memory size 2774532096
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2475556864 is less than or equal to previous memory size 2526871552
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2415591424 is less than or equal to previous memory size 2475556864
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2312699904 is less than or equal to previous memory size 2415591424
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_ppc64le_linux_Nightly_testList_1/41 testSoftMxDisclaimMemory_2

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_ppc64le_linux_Nightly_testList_1/41/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 1744961536 is less than or equal to previous memory size 

[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!

FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1054)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:994)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:688)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:366)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/323 testSoftMxDisclaimMemory_2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4068409344 is less than or equal to previous memory size 4058578944
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4071358464 is less than or equal to previous memory size 4067819520
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4071751680 is less than or equal to previous memory size 4071358464
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4071358464 is less than or equal to previous memory size 4071751680
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4072144896 is less than or equal to previous memory size 4071358464
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_aarch64_linux_Nightly_testList_1/295 testDefaultDisclaimMemory_2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4526702592 is less than or equal to previous memory size 4521197568
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4098555904 is less than or equal to previous memory size 4526112768
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4103798784 is less than or equal to previous memory size 4098555904
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4100259840 is less than or equal to previous memory size 4103798784
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4092592128 is less than or equal to previous memory size 4100259840
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_s390x_linux_Nightly_testList_1/443 testDefaultDisclaimMemory_zlinux_1

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4219842560 is less than or equal to previous memory size 4236763136
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4200697856 is less than or equal to previous memory size 4216680448
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4058271744 is less than or equal to previous memory size 4200648704
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4025245696 is less than or equal to previous memory size 4058271744
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 3949391872 is less than or equal to previous memory size 4022083584
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/378 testSoftMxDisclaimMemory_2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 2841378816 is less than or equal to previous memory size 2830893056
[SoftmxAdvanceTest] [WARN]  Generate Java core dump after forcing GC.
JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_16734940088399/testSoftMxDisclaimMemory_2/javacore.20230111.234948.4320.0001.txt' through com.ibm.jvm.Dump.JavaDump
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_16734940088399/testSoftMxDisclaimMemory_2/javacore.20230111.234948.4320.0001.txt
JVMDUMP034I User requested Heap dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_16734940088399/testSoftMxDisclaimMemory_2/heapdump.20230111.234948.4320.0002.phd' through com.ibm.jvm.Dump.HeapDump
JVMDUMP010I Heap dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_16734940088399/testSoftMxDisclaimMemory_2/heapdump.20230111.234948.4320.0002.phd
JVMDUMP034I User requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_16734940088399/testSoftMxDisclaimMemory_2/core.20230111.234948.4320.0003.dmp' through com.ibm.jvm.Dump.SystemDump
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_16734940088399/testSoftMxDisclaimMemory_2/core.20230111.234948.4320.0003.dmp
[SoftmxAdvanceTest] [ERROR]     FAIL: Heap can't shrink to the reset max heap size within 5 minutes!
...
FAILED: testDisclaimMemoryEffects(5)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1046)
    at java.base/java.util.Collection.stream(Collection.java:743)
    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:990)
    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:685)
    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/375 testSoftMxDisclaimMemory_LP4k_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.functional_aarch64_linux_Nightly_testList_0/375/functional_test_output.tar.gz

20:05:43  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 4885184512 is less than or equal to previous memory size 4881448960
20:06:08  [SoftmxAdvanceTest] [ERROR]   FAIL: Memory didn't release back to OS! Post memory size 4887150592 is less than or equal to previous memory size 4885184512
20:06:08  JVMDUMP055I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError", exception "Java heap space" at 2023/02/01 20:06:07 - please wait.
...
20:06:08  JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
20:06:08  FAILED: testDisclaimMemoryEffects(5)
20:06:08  java.lang.OutOfMemoryError: Java heap space
20:06:08    at java.base/java.util.HashMap$Values.spliterator(HashMap.java:1046)
20:06:08    at java.base/java.util.Collection.stream(Collection.java:743)
20:06:08    at java.management/java.lang.management.ManagementFactory$PlatformMBeanFinder.findSingleton(ManagementFactory.java:990)
20:06:08    at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:685)
20:06:08    at java.management/java.lang.management.ManagementFactory.getRuntimeMXBean(ManagementFactory.java:365)
20:06:08    at j9vm.test.softmx.SoftmxAdvanceTest.runTestDisclaimMemoryEffects(SoftmxAdvanceTest.java:114)
20:06:08    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:64)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_aarch64_linux_Nightly_testList_1/363 testDefaultDisclaimMemory_2

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.functional_aarch64_linux_Nightly_testList_1/363/functional_test_output.tar.gz

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4162584576 is less than or equal to previous memory size 4592893952
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4163764224 is less than or equal to previous memory size 4162387968
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4165926912 is less than or equal to previous memory size 4163764224
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4164157440 is less than or equal to previous memory size 4165926912
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4164550656 is less than or equal to previous memory size 4164157440
FAILED: testDisclaimMemoryEffects(5)
java.lang.AssertionError: Test has failed after 5 attempts
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at j9vm.test.softmx.SoftmxAdvanceTest.testDisclaimMemoryEffects(SoftmxAdvanceTest.java:70)
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_0/427 testSoftMxDisclaimMemory_2

[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4463394816 is less than or equal to previous memory size 4454088704
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4465950720 is less than or equal to previous memory size 4463394816
[SoftmxAdvanceTest] [ERROR]     FAIL: Memory didn't release back to OS! Post memory size 4451663872 is less than or equal to previous memory size 4465950720