search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

JIT failure in GatherersMapConcurrentTest vmState=0x0005ff04 #20267

Open keithc-ca opened 1 month ago

keithc-ca commented 1 month ago

Failure link

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal/4

vmState [0x5ff04]: {J9VMSTATE_JIT} {InstructionSelection}

Failure output (captured from console output)

[2024-09-30T19:22:09.443Z] STARTED    GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[2] Config[streamSize=10, parallel=true]'
[2024-09-30T19:22:09.443Z] #0: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xae5a5e) [0x3ff7ce65a5e]
[2024-09-30T19:22:09.444Z] #1: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xaf3a00) [0x3ff7ce73a00]
[2024-09-30T19:22:09.444Z] #2: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1798bc) [0x3ff7c4f98bc]
[2024-09-30T19:22:09.444Z] #3: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x31c36) [0x3ff7d8b1c36]
[2024-09-30T19:22:09.444Z] #4: [0x3ff559f66e0]
[2024-09-30T19:22:09.444Z] #5: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x68829c) [0x3ff7ca0829c]
[2024-09-30T19:22:09.444Z] #6: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xbc0856) [0x3ff7cf40856]
[2024-09-30T19:22:09.444Z] #7: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6885f6) [0x3ff7ca085f6]
[2024-09-30T19:22:09.444Z] #8: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xbbacc2) [0x3ff7cf3acc2]
[2024-09-30T19:22:09.444Z] #9: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6885f6) [0x3ff7ca085f6]
[2024-09-30T19:22:09.444Z] #10: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x14d47a) [0x3ff7c4cd47a]
[2024-09-30T19:22:09.444Z] #11: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6998fc) [0x3ff7ca198fc]
[2024-09-30T19:22:09.444Z] #12: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x69a74e) [0x3ff7ca1a74e]
[2024-09-30T19:22:09.444Z] #13: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x695b88) [0x3ff7ca15b88]
[2024-09-30T19:22:09.444Z] #14: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6c785c) [0x3ff7ca4785c]
[2024-09-30T19:22:09.444Z] #15: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x195eee) [0x3ff7c515eee]
[2024-09-30T19:22:09.444Z] #16: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x197018) [0x3ff7c517018]
[2024-09-30T19:22:09.444Z] #17: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32b68) [0x3ff7d8b2b68]
[2024-09-30T19:22:09.444Z] #18: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x19467a) [0x3ff7c51467a]
[2024-09-30T19:22:09.444Z] #19: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x194c04) [0x3ff7c514c04]
[2024-09-30T19:22:09.444Z] #20: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x19368a) [0x3ff7c51368a]
[2024-09-30T19:22:09.444Z] #21: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193b7c) [0x3ff7c513b7c]
[2024-09-30T19:22:09.444Z] #22: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193c14) [0x3ff7c513c14]
[2024-09-30T19:22:09.444Z] #23: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32b68) [0x3ff7d8b2b68]
[2024-09-30T19:22:09.444Z] #24: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x194056) [0x3ff7c514056]
[2024-09-30T19:22:09.444Z] #25: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a46) [0x3ff7d808a46]
[2024-09-30T19:22:09.444Z] #26: /lib64/libpthread.so.0(+0x8312) [0x3ff7e308312]
[2024-09-30T19:22:09.444Z] #27: /lib64/libc.so.6(+0x10e232) [0x3ff7e20e232]
[2024-09-30T19:22:09.444Z] Unhandled exception
[2024-09-30T19:22:09.444Z] Type=Segmentation error vmState=0x0005ff04
[2024-09-30T19:22:09.444Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=06b6fed5 Signal_Code=00000001
[2024-09-30T19:22:09.444Z] Handler1=000003FF7D9CBA48 Handler2=000003FF7D8B1A18 InaccessibleAddress=00001FCCCBA40000
[2024-09-30T19:22:09.444Z] gpr0=0000000000000000 gpr1=000003FF7C971908 gpr2=000003FF1CD562F0 gpr3=000003FF7D2993A0
[2024-09-30T19:22:09.444Z] gpr4=0000000000005E40 gpr5=0000000000000030 gpr6=000003FECB7D2BC0 gpr7=000003FEC9D00960
[2024-09-30T19:22:09.444Z] gpr8=00001FCCCBA40810 gpr9=000003FF36104CA0 gpr10=000003FF1CD562F0 gpr11=000003FEC9D007D0
[2024-09-30T19:22:09.444Z] gpr12=000003FF7D295000 gpr13=000003FF7D0EA4F8 gpr14=000003FF7CA0816E gpr15=000003FF559F6B68
[2024-09-30T19:22:09.444Z] psw=000003FF7CA0829C mask=0705200180000000 fpc=0008fe00 bea=000003FF7CA0817A
[2024-09-30T19:22:09.444Z] fpr0=401924407ab0e074 (f: 2058412160.000000, d: 6.285402e+00)
[2024-09-30T19:22:09.444Z] fpr1=3fb93d4bb7e327a9 (f: 3085117440.000000, d: 9.859155e-02)
[2024-09-30T19:22:09.444Z] fpr2=bfd24407ab0e073a (f: 2869823232.000000, d: -2.854022e-01)
[2024-09-30T19:22:09.444Z] fpr3=3fd999f1abd2dada (f: 2882722560.000000, d: 4.000210e-01)
[2024-09-30T19:22:09.444Z] fpr4=3f7ab2e26ca4c962 (f: 1822738816.000000, d: 6.518254e-03)
[2024-09-30T19:22:09.444Z] fpr5=3fcc723e7dcde3fb (f: 2110645248.000000, d: 2.222365e-01)
[2024-09-30T19:22:09.444Z] fpr6=3fcb2a68ec9ad9b5 (f: 3969571328.000000, d: 2.122317e-01)
[2024-09-30T19:22:09.444Z] fpr7=3f83e83bb8b32a19 (f: 3098749440.000000, d: 9.720294e-03)
[2024-09-30T19:22:09.444Z] fpr8=000003ff65a141a8 (f: 1705066880.000000, d: 2.171644e-311)
[2024-09-30T19:22:09.444Z] fpr9=00000000005cd458 (f: 6083672.000000, d: 3.005733e-317)
[2024-09-30T19:22:09.444Z] fpr10=000003ff65a28d50 (f: 1705151872.000000, d: 2.171644e-311)
[2024-09-30T19:22:09.444Z] fpr11=000003ffa5ffe3d8 (f: 2785010688.000000, d: 2.172178e-311)
[2024-09-30T19:22:09.454Z] fpr12=0006235ae58e315c (f: 3851301120.000000, d: 8.536087e-309)
[2024-09-30T19:22:09.454Z] fpr13=000003ff080537d8 (f: 134559712.000000, d: 2.170868e-311)
[2024-09-30T19:22:09.454Z] fpr14=000003ffdcd7aa94 (f: 3705121536.000000, d: 2.172632e-311)
[2024-09-30T19:22:09.454Z] fpr15=000003ff0804fe98 (f: 134545056.000000, d: 2.170868e-311)
[2024-09-30T19:22:09.454Z] Module=/home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
[2024-09-30T19:22:09.454Z] Module_base_address=000003FF7C380000
[2024-09-30T19:22:09.454Z] 
[2024-09-30T19:22:09.454Z] Method_being_compiled=java/util/stream/Gatherers$3State.integrate(Ljava/lang/Object;Ljava/util/stream/Gatherer$Downstream;)Z
[2024-09-30T19:22:09.454Z] Target=2_90_20240930_8 (Linux 3.10.0-1160.118.1.el7.s390x)
[2024-09-30T19:22:09.454Z] CPU=s390x (4 logical CPUs) (0x1ec1b1000 RAM)
[2024-09-30T19:22:09.454Z] ----------- Stack Backtrace -----------
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[2] Config[streamSize=10, parallel=true]'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[3] Config[streamSize=33, parallel=false]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[3] Config[streamSize=33, parallel=false]'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[4] Config[streamSize=33, parallel=true]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[4] Config[streamSize=33, parallel=true]'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[5] Config[streamSize=99, parallel=false]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[5] Config[streamSize=99, parallel=false]'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[6] Config[streamSize=99, parallel=true]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[6] Config[streamSize=99, parallel=true]'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[1] true'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[1] true'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[2] false'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[2] false'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[1] -2147483648'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[1] -2147483648'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[2] -999'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[2] -999'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[3] -1'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[3] -1'
[2024-09-30T19:22:09.454Z] STARTED    GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[4] 0'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[4] 0'
github-actions[bot] commented 1 month ago

Issue Number: 20267 Status: Open Recommended Components: comp:test, comp:gc, comp:vm Recommended Assignees: jasonfengj9, pshipton, llxia

pshipton commented 1 month ago

@hzongaro fyi

hzongaro commented 1 month ago

@r30shah, could someone on your team take an initial look at this? It looks like it's crashing in a call to OMR::CodeGenerator::decReferenceCount, but I didn't seem to be able to do much more than that with the core file. . . .

   0x3ff7ca0829c <_ZN3OMR13CodeGenerator17decReferenceCountEPN2TR4NodeE+332>:   l       %r1,48(%r8)
   0x3ff7ca082a0 <_ZN3OMR13CodeGenerator17decReferenceCountEPN2TR4NodeE+336>:   ahi     %r1,-1
   0x3ff7ca082a4 <_ZN3OMR13CodeGenerator17decReferenceCountEPN2TR4NodeE+340>:   st      %r1,48(%r8)
   0x3ff7ca082a8 <_ZN3OMR13CodeGenerator17decReferenceCountEPN2TR4NodeE+344>:   ltr     %r1,%r1
r30shah commented 1 month ago

@ehsankianifar Can you take a look at this one ? I have launched a grinder of 10x to see if it is easily reproducible. We should take a look at the core-dump (You can use the debug-images and see if we can find out which node it is complaining of).

r30shah commented 1 month ago

@ehsankianifar Link for the 10x grinder : https://openj9-jenkins.osuosl.org/job/Grinder/3861 - 1/10 failed with BBEnd node which is same as seen in https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/4. May be generated JITdump can show what it is complaining about.

pshipton commented 1 month ago

Another seg fault with 0x0005ff04 compiling test material. https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_OMR_testList_0/132

02:38:12      [javac] Assertion failed at /home/****/workspace/Build_JDK21_ppc64_aix_OMR/omr/compiler/codegen/OMRTreeEvaluator.cpp:590: false
02:38:12      [javac] VMState: 0x0005ff04
02:38:12      [javac]   badILOp BadILOp cannot be evaluated
02:38:12      [javac] 
02:38:12      [javac] compiling java/lang/ClassLoader.defineClass(Ljava/lang/String;Ljava/nio/ByteBuffer;Ljava/security/ProtectionDomain;)Ljava/lang/Class; at level: very-hot (profiling)
ehsankianifar commented 1 month ago

Thanks @pshipton for sharing it. I am investigating this issue and will provide the fix as soon as I find the root cause.

pshipton commented 5 days ago

A couple more here on xmac jdk21 https://github.com/eclipse-openj9/openj9/issues/20531#issuecomment-2477354906