search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

OpenJDK java/lang/StringBuilder/HugeCapacity Unexpected exit from test [exit code: 137] #17323

Open pshipton opened 1 year ago

pshipton commented 1 year ago

This was seen once before, https://github.com/eclipse-openj9/openj9/issues/17203#issuecomment-1511962790

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/529 - ub20-390-4 jdk_lang_j9_0 -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs java/lang/StringBuilder/HugeCapacity.java

I don't spot any diagnostics.

00:36:55  ACTION: main -- Failed. Unexpected exit from test [exit code: 137]
00:36:55  REASON: User specified action: run main/othervm -Xmx5G -XX:+CompactStrings HugeCapacity 
00:36:55  TIME:   17.562 seconds
00:36:55  messages:
00:36:55  command: main -Xmx5G -XX:+CompactStrings HugeCapacity
00:36:55  reason: User specified action: run main/othervm -Xmx5G -XX:+CompactStrings HugeCapacity 
00:36:55  Mode: othervm [/othervm specified]
00:36:55  elapsed time (seconds): 17.562
00:36:55  configuration:
00:36:55  STDOUT:
00:36:55  STDERR:
00:36:55  rerun:
00:36:55  cd /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/scratch/0 && \
00:36:55  DISPLAY=:0 \
00:36:55  HOME=/home/jenkins \
00:36:55  LANG=en_US.UTF-8 \
00:36:55  PATH=/bin:/usr/bin:/usr/sbin \
00:36:55  CLASSPATH=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/jtreg.jar \
00:36:55      /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image/bin/java \
00:36:55          -Dtest.vm.opts='-ea -esa -Xmx512m -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs' \
00:36:55          -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-Xdump:system:none -J-Xdump:heap:none -J-Xdump:system:events=gpf+abort+traceassert+corruptcache -J-XX:-JITServerTechPreviewMessage -J-Xjit -J-Xgcpolicy:balanced -J-Xnocompressedrefs' \
00:36:55          -Dtest.compiler.opts= \
00:36:55          -Dtest.java.opts= \
00:36:55          -Dtest.jdk=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image \
00:36:55          -Dcompile.jdk=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image \
00:36:55          -Dtest.timeout.factor=8.0 \
00:36:55          -Dtest.nativepath=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
00:36:55          -Dtest.root=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk \
00:36:55          -Dtest.name=java/lang/StringBuilder/HugeCapacity.java \
00:36:55          -Dtest.file=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder/HugeCapacity.java \
00:36:55          -Dtest.src=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder \
00:36:55          -Dtest.src.path=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder \
00:36:55          -Dtest.classes=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d \
00:36:55          -Dtest.class.path=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d \
00:36:55          -Dtest.class.path.prefix=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder \
00:36:55          -ea \
00:36:55          -esa \
00:36:55          -Xmx512m \
00:36:55          -Xdump:system:none \
00:36:55          -Xdump:heap:none \
00:36:55          -Xdump:system:events=gpf+abort+traceassert+corruptcache \
00:36:55          -XX:-JITServerTechPreviewMessage \
00:36:55          -Xjit \
00:36:55          -Xgcpolicy:balanced \
00:36:55          -Xnocompressedrefs \
00:36:55          -Djava.library.path=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
00:36:55          -Xmx5G \
00:36:55          -XX:+CompactStrings \
00:36:55          com.sun.javatest.regtest.agent.MainWrapper /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/java/lang/StringBuilder/HugeCapacity.d/main.0.jta
00:36:55  
00:36:55  TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
00:36:55  --------------------------------------------------
00:40:13  Test results: passed: 731; failed: 1
pshipton commented 1 year ago

40x https://openj9-jenkins.osuosl.org/job/Grinder/2307/ - passed

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_OpenJDK11/68 - ub20-390-6

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_OMR/4 - ub20-390-5 https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/536 - ub20-390-3

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/537 - ub20-390-6

pshipton commented 1 year ago

@dmitripivkine not sure if this belongs to GC. The failure started showing up when we starting testing with -Xgcpolicy:balanced not long ago. We've only seen it on zlinux. It's only seen on jdk11 because the test is excluded from jdk16 (17) due to https://github.com/eclipse-openj9/openj9/issues/10582

dmitripivkine commented 1 year ago

@pshipton @llxia Is there any way to run test differently to get failure details to look at? I can not find any results stored, just error message. There is no reason to run grinders until we figure out how to get meaningful results.

pshipton commented 1 year ago

Checking if the failure can be reproduced when running just the failing test. Using EXTRA_OPTIONS -Xgcpolicy:balanced -Xnocompressedrefs -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError 50x x 5 grinder https://openj9-jenkins.osuosl.org/job/Grinder/2476/ - passed on ub20-390-1, 3, 4, 5, 6 50x x 5 grinder https://openj9-jenkins.osuosl.org/job/Grinder/2477/ - passed on ub20-390-3, 5, 6, rh7-390-1, 4

pshipton commented 1 year ago

Try running all jdk_lang_j9_0 Using EXTRA_OPTIONS -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError, which I expect will create a number of core files for passing tests. 3x x 5 https://openj9-jenkins.osuosl.org/job/Grinder/2478/

pshipton commented 1 year ago

There is a test mismatch in the grinder, but HugeCapacity also failed in 2/5 of the runs (runs 0, 4 on ub20-390-3, 4). The other runs occurred on the rhel7 machines.

The test does only seem to fail on the ub20 machines. It seems like maybe there isn't enough memory to create the JVM and run the test. All the machines have about 8GB, but the ub20 machines seem to be configured with no swap space, while the rhel7 machines do have swap space. I'll set this as an infra issue.

JasonFengJ9 commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Release_testList_1/5/consoleFull

04:35:04  TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
04:35:04  --------------------------------------------------
04:37:39  Test results: passed: 730; failed: 1
04:37:55  Report written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Release_testList_1/aqa-tests/TKG/output_16900990055977/jdk_lang_j9_0/report/html/report.html
04:37:55  Results written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Release_testList_1/aqa-tests/TKG/output_16900990055977/jdk_lang_j9_0/work
04:37:55  Error: Some tests failed or other problems occurred.
04:37:55  -----------------------------------
04:37:55  jdk_lang_j9_0_FAILED
pshipton commented 10 months ago

Also fails on rh8-390-2, which also has no swap space. Since we aren't making any progress fixing the machines I will exclude the test until it can pass.

              total        used        free      shared  buff/cache   available
Mem:        7696108     1538712     5004988      368900     1152408     5672452
Swap:             0           0           0

https://github.com/adoptium/aqa-tests/pull/4940

pshipton commented 9 months ago

The machines are updated to have swap space, and seem to work. Unexclude the test. https://github.com/adoptium/aqa-tests/pull/5033

JasonFengJ9 commented 2 days ago

openjdk11_j9_sanity.openjdk_s390x_linux(rhel9s390xrt-2)

[2024-11-02T03:21:55.033Z] variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode501
[2024-11-02T03:21:55.033Z] JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs -Xverbosegclog 

[2024-11-02T03:55:01.772Z] TEST: java/lang/StringBuilder/HugeCapacity.java

[2024-11-02T03:55:01.773Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
[2024-11-02T03:55:01.773Z] --------------------------------------------------
[2024-11-02T04:00:20.340Z] Test results: passed: 738; failed: 1
[2024-11-02T04:01:01.759Z] Report written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux/aqa-tests/TKG/output_17305122238337/jdk_lang_j9_0/report/html/report.html
[2024-11-02T04:01:01.759Z] Results written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux/aqa-tests/TKG/output_17305122238337/jdk_lang_j9_0/work
[2024-11-02T04:01:01.759Z] Error: Some tests failed or other problems occurred.
[2024-11-02T04:01:01.759Z] -----------------------------------
[2024-11-02T04:01:01.759Z] jdk_lang_j9_0_FAILED