eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 720 forks source link

cmdLineTester_jvmtitests emex001 crash vmState=0x00000000 in jvmtiHookMethodExit #17889

Closed pshipton closed 1 year ago

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/359/ cmdLineTester_jvmtitests_1 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/359/functional_test_output.tar.gz

Testing: emex001
Test start time: 2023/07/31 16:32:35 Coordinated Universal Time
Running command: "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:gencon -Xshareclasses:name=bcienabled,enableBCI -Xnocompressedrefs  -Xdump    -agentlib:jvmtitest=test:emex001 -cp "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 6 milliseconds
Time spent executing: 27078 milliseconds
Test result: FAILED
Output from test:
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=00007FFF8132FA40 Handler2=00007FFF81258AA0
 [ERR] R0=00007FFF811B1960 R1=00007FFF81A4C430 R2=00007FFF805B7B00 R3=00007FFF7C02C818
 [ERR] R4=0000000000000019 R5=0000000000000000 R6=00007FFF7C0BBCD8 R7=00000000000F4240
 [ERR] R8=0000000000000015 R9=00007FFEAD46F149 R10=0000000000000000 R11=00007FFF81A4C4F0
 [ERR] R12=00007FFF80545670 R13=00007FFF81A568E0 R14=00007FFF7C23E920 R15=00007FFF7C0DEE00
 [ERR] R16=00007FFEE2CB0038 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=00007FFF7B049960
 [ERR] R20=00007FFF7C064548 R21=00007FFF819C1F80 R22=00007FFF7C0BBCD8 R23=000601CAFA7303BC
 [ERR] R24=000000000000003B R25=00007FFF7C16A348 R26=0000000000000000 R27=00007FFEE2EAE4F0
 [ERR] R28=00007FFF7C0DEE00 R29=00007FFF805B1228 R30=0000000000000000 R31=00007FFF7C0BBCD8
 [ERR] NIP=00007FFF805458F0 MSR=800000000280F033 ORIG_GPR3=00007FFF80545898 CTR=00007FFF80545670
 [ERR] LINK=00007FFF811B1960 XER=0000000020040000 CCR=0000000024828248 SOFTE=0000000000000001
 [ERR] TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
 [ERR] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR1 4051e54a60000000 (f: 1610612736.000000, d: 7.158266e+01)
 [ERR] FPR2 3ff07a4c40000000 (f: 1073741824.000000, d: 1.029858e+00)
 [ERR] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
 [ERR] FPR4 4010bb2140000000 (f: 1073741824.000000, d: 4.182744e+00)
 [ERR] FPR5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [ERR] FPR7 bfba4e76ce8c0e5e (f: 3465285120.000000, d: -1.027598e-01)
 [ERR] FPR8 5900b62c00b42a2a (f: 11807274.000000, d: 5.394194e+120)
 [ERR] FPR9 be00801800000000 (f: 0.000000, d: -4.802239e-10)
 [ERR] FPR10 3880180a80000000 (f: 2147483648.000000, d: 1.513464e-36)
 [ERR] FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
 [ERR] FPR12 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
 [ERR] FPR13 00000000167e693e (f: 377383232.000000, d: 1.864521e-315)
 [ERR] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
 [ERR] Module_base_address=00007FFF80520000
 [ERR] Target=2_90_20230731_387 (Linux 4.18.0-408.el8.ppc64le)
 [ERR] CPU=ppc64le (4 logical CPUs) (0x1dbc30000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] jvmtiHookMethodExit+0x280 (0x00007FFF805458F0 [libj9jvmti29.so+0x258f0])
 [ERR] J9HookDispatch+0x1e0 (0x00007FFF811B1960 [libj9hookable29.so+0x1960])
 [ERR] old_slow_jitReportMethodExit+0x18c (0x00007FFF7BC9308C [libj9jit29.so+0xc4308c])
 [ERR]  (0x00007FFF7BCA7A90 [libj9jit29.so+0xc57a90])
 [ERR] sendResolveInvokeDynamic+0x26c (0x00007FFF81309B6C [libj9vm29.so+0x19b6c])
 [ERR] resolveInvokeDynamic+0x240 (0x00007FFF8137F0B0 [libj9vm29.so+0x8f0b0])
 [ERR] debugBytecodeLoopFull+0x2a438 (0x00007FFF814535D8 [libj9vm29.so+0x1635d8])
 [ERR]  (0x00007FFF8145E150 [libj9vm29.so+0x16e150])
 [ERR] runCallInMethod+0x258 (0x00007FFF81309F18 [libj9vm29.so+0x19f18])
 [ERR] gpProtectedRunCallInMethod+0x54 (0x00007FFF81333DC4 [libj9vm29.so+0x43dc4])
 [ERR] signalProtectAndRunGlue+0x28 (0x00007FFF81471468 [libj9vm29.so+0x181468])
 [ERR] omrsig_protect+0x3f4 (0x00007FFF81259F74 [libj9prt29.so+0x39f74])
 [ERR] gpProtectAndRun+0xa8 (0x00007FFF81471538 [libj9vm29.so+0x181538])
 [ERR] gpCheckCallin+0xc4 (0x00007FFF81336484 [libj9vm29.so+0x46484])
 [ERR] callStaticVoidMethod+0x48 (0x00007FFF813333C8 [libj9vm29.so+0x433c8])
 [ERR] JavaMain+0x11e4 (0x00007FFF81D27304 [libjli.so+0x7304])
 [ERR] ThreadJavaMain+0x18 (0x00007FFF81D2D258 [libjli.so+0xd258])
 [ERR] start_thread+0xf8 (0x00007FFF81CD9678 [libpthread-2.28.so+0x9678])
 [ERR] clone+0x74 (0x00007FFF81BC8738 [libc-2.28.so+0x138738])
 [ERR] ---------------------------------------
pshipton commented 1 year ago

@babsingh is this related to a recent change?

babsingh commented 1 year ago

is this related to a recent change?

No since the recent JVMTI changes should only impact JDK19+. Will look at the core file to determine the cause of the failure.

pshipton commented 1 year ago

A 5x grinder passed

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/517 cmdLineTester_jvmtitests_1 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/517/functional_test_output.tar.gz

Test start time: 2023/08/09 01:38:54 Coordinated Universal Time
Running command: "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:gencon -Xshareclasses:name=bcienabled,enableBCI -Xnocompressedrefs  -Xdump    -agentlib:jvmtitest=test:emex001 -cp "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 6 milliseconds
Time spent executing: 30289 milliseconds
Test result: FAILED
Output from test:
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=00007FFF9277FA40 Handler2=00007FFF926A8AA0
 [ERR] R0=00007FFF92601960 R1=00007FFF92E9C430 R2=00007FFF90A57B00 R3=00007FFF8C02C7B8
 [ERR] R4=0000000000000019 R5=0000000000000000 R6=00007FFF8C0BBD48 R7=00000000000F4240
 [ERR] R8=0000000000000015 R9=00007FFEC146F149 R10=0000000000000000 R11=00007FFF92E9C4F0
 [ERR] R12=00007FFF909E5670 R13=00007FFF92EA68E0 R14=00007FFF8C23F630 R15=00007FFF8C0DF000
 [ERR] R16=00007FFEF8130038 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=00007FFF8BFDED38
 [ERR] R20=00007FFF8C0645B8 R21=00007FFF92E11F80 R22=00007FFF8C0BBD48 R23=000602738AF24E03
 [ERR] R24=000000000000003B R25=00007FFF8C16A278 R26=0000000000000000 R27=00007FFEF832E4F0
 [ERR] R28=00007FFF8C0DF000 R29=00007FFF90A51228 R30=0000000000000000 R31=00007FFF8C0BBD48
 [ERR] NIP=00007FFF909E58F0 MSR=800000000280F033 ORIG_GPR3=00007FFF909E5898 CTR=00007FFF909E5670
 [ERR] LINK=00007FFF92601960 XER=0000000020040000 CCR=0000000024828248 SOFTE=0000000000000001
 [ERR] TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
 [ERR] FPR0 00007fff8c23f830 (f: 2351167488.000000, d: 6.953260e-310)
 [ERR] FPR1 40523530c0000000 (f: 3221225472.000000, d: 7.283110e+01)
 [ERR] FPR2 3ffd8456a0000000 (f: 2684354560.000000, d: 1.844809e+00)
 [ERR] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
 [ERR] FPR4 400b922980000000 (f: 2147483648.000000, d: 3.446368e+00)
 [ERR] FPR5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [ERR] FPR7 3fbc5e53aa362eb4 (f: 2855677696.000000, d: 1.108143e-01)
 [ERR] FPR8 00007fff8be71190 (f: 2347176448.000000, d: 6.953260e-310)
 [ERR] FPR9 00007fff8be71278 (f: 2347176448.000000, d: 6.953260e-310)
 [ERR] FPR10 00007fff8be71360 (f: 2347176704.000000, d: 6.953260e-310)
 [ERR] FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
 [ERR] FPR12 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
 [ERR] FPR13 00000000167e693e (f: 377383232.000000, d: 1.864521e-315)
 [ERR] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
 [ERR] Module_base_address=00007FFF909C0000
 [ERR] Target=2_90_20230808_548 (Linux 4.18.0-408.el8.ppc64le)
 [ERR] CPU=ppc64le (4 logical CPUs) (0x1dbc30000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] jvmtiHookMethodExit+0x280 (0x00007FFF909E58F0 [libj9jvmti29.so+0x258f0])
 [ERR] J9HookDispatch+0x1e0 (0x00007FFF92601960 [libj9hookable29.so+0x1960])
 [ERR] old_slow_jitReportMethodExit+0x18c (0x00007FFF91C131CC [libj9jit29.so+0xc431cc])
 [ERR]  (0x00007FFF91C27BD0 [libj9jit29.so+0xc57bd0])
 [ERR] sendResolveInvokeDynamic+0x26c (0x00007FFF92759B6C [libj9vm29.so+0x19b6c])
 [ERR] resolveInvokeDynamic+0x240 (0x00007FFF927CF0B0 [libj9vm29.so+0x8f0b0])
 [ERR] debugBytecodeLoopFull+0x30cec (0x00007FFF928A9BBC [libj9vm29.so+0x169bbc])
 [ERR]  (0x00007FFF928ADDC0 [libj9vm29.so+0x16ddc0])
 [ERR] runCallInMethod+0x258 (0x00007FFF92759F18 [libj9vm29.so+0x19f18])
 [ERR] gpProtectedRunCallInMethod+0x54 (0x00007FFF92783DC4 [libj9vm29.so+0x43dc4])
 [ERR] signalProtectAndRunGlue+0x28 (0x00007FFF928C10C8 [libj9vm29.so+0x1810c8])
 [ERR] omrsig_protect+0x3f4 (0x00007FFF926A9F74 [libj9prt29.so+0x39f74])
 [ERR] gpProtectAndRun+0xa8 (0x00007FFF928C1198 [libj9vm29.so+0x181198])
 [ERR] gpCheckCallin+0xc4 (0x00007FFF92786484 [libj9vm29.so+0x46484])
 [ERR] callStaticVoidMethod+0x48 (0x00007FFF927833C8 [libj9vm29.so+0x433c8])
 [ERR] JavaMain+0x11e4 (0x00007FFF93177304 [libjli.so+0x7304])
 [ERR] ThreadJavaMain+0x18 (0x00007FFF9317D258 [libjli.so+0xd258])
 [ERR] start_thread+0xf8 (0x00007FFF93129678 [libpthread-2.28.so+0x9678])
 [ERR] clone+0x74 (0x00007FFF93018738 [libc-2.28.so+0x138738])
 [ERR] ---------------------------------------
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR/364 cmdLineTester_jvmtitests_2

Testing: emex001
Test start time: 2023/08/13 01:37:39 Coordinated Universal Time
Running command: "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:gencon -Xshareclasses:cacheRetransformed,name=retransformenabled -Xnocompressedrefs  -Xdump    -agentlib:jvmtitest=test:emex001 -cp "/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 4 milliseconds
Time spent executing: 25381 milliseconds
Test result: FAILED
Output from test:
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=00007FFFBA36FA40 Handler2=00007FFFBA298AA0
 [ERR] R0=00007FFFBA1F1960 R1=00007FFFBAA8C430 R2=00007FFFB8647B00 R3=00007FFFB402C848
 [ERR] R4=0000000000000019 R5=0000000000000000 R6=00007FFFB40BBD18 R7=00000000000F4240
 [ERR] R8=0000000000000015 R9=00007FFEE946F149 R10=0000000000000000 R11=00007FFFBAA8C4F0
 [ERR] R12=00007FFFB85D5670 R13=00007FFFBAA968E0 R14=00007FFFB42397A0 R15=00007FFFB40DF000
 [ERR] R16=00007FFF03CF0038 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=00007FFFB3FD8240
 [ERR] R20=00007FFFB4064588 R21=00007FFFBAA01F80 R22=00007FFFB40BBD18 R23=000602C3FDD92A8A
 [ERR] R24=000000000000003B R25=00007FFFB416A578 R26=0000000000000000 R27=00007FFF03EEE4F0
 [ERR] R28=00007FFFB40DF000 R29=00007FFFB8641248 R30=0000000000000000 R31=00007FFFB40BBD18
 [ERR] NIP=00007FFFB85D58F0 MSR=800000000280F033 ORIG_GPR3=00007FFFB85D5898 CTR=00007FFFB85D5670
 [ERR] LINK=00007FFFBA1F1960 XER=0000000020040000 CCR=0000000024828248 SOFTE=0000000000000001
 [ERR] TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
 [ERR] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR1 40551ac3a0000000 (f: 2684354560.000000, d: 8.441819e+01)
 [ERR] FPR2 3ff4e91960000000 (f: 1610612736.000000, d: 1.306909e+00)
 [ERR] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
 [ERR] FPR4 4014591600000000 (f: 0.000000, d: 5.086998e+00)
 [ERR] FPR5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [ERR] FPR7 3fc526e57720db08 (f: 1998641920.000000, d: 1.652495e-01)
 [ERR] FPR8 6b6f766e692f676e (f: 1764714368.000000, d: 3.232376e+209)
 [ERR] FPR9 6164626d614c2f65 (f: 1632382848.000000, d: 1.432938e+161)
 [ERR] FPR10 484d42246d726f46 (f: 1836216192.000000, d: 1.991221e+40)
 [ERR] FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
 [ERR] FPR12 4008000000000000 (f: 0.000000, d: 3.000000e+00)
 [ERR] FPR13 000000005750dba2 (f: 1464916864.000000, d: 7.237651e-315)
 [ERR] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_ppc64le_linux_OMR_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
 [ERR] Module_base_address=00007FFFB85B0000
 [ERR] Target=2_90_20230812_390 (Linux 4.18.0-408.el8.ppc64le)
 [ERR] CPU=ppc64le (4 logical CPUs) (0x1dbc30000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] jvmtiHookMethodExit+0x280 (0x00007FFFB85D58F0 [libj9jvmti29.so+0x258f0])
 [ERR] J9HookDispatch+0x1e0 (0x00007FFFBA1F1960 [libj9hookable29.so+0x1960])
 [ERR] old_slow_jitReportMethodExit+0x18c (0x00007FFFB98037CC [libj9jit29.so+0xc437cc])
 [ERR]  (0x00007FFFB98181D0 [libj9jit29.so+0xc581d0])
 [ERR] sendResolveInvokeDynamic+0x26c (0x00007FFFBA349B6C [libj9vm29.so+0x19b6c])
 [ERR] resolveInvokeDynamic+0x240 (0x00007FFFBA3BF0B0 [libj9vm29.so+0x8f0b0])
 [ERR] debugBytecodeLoopFull+0x30cfc (0x00007FFFBA499BEC [libj9vm29.so+0x169bec])
 [ERR]  (0x00007FFFBA49DDF0 [libj9vm29.so+0x16ddf0])
 [ERR] runCallInMethod+0x258 (0x00007FFFBA349F18 [libj9vm29.so+0x19f18])
 [ERR] gpProtectedRunCallInMethod+0x54 (0x00007FFFBA373DC4 [libj9vm29.so+0x43dc4])
 [ERR] signalProtectAndRunGlue+0x28 (0x00007FFFBA4B1108 [libj9vm29.so+0x181108])
 [ERR] omrsig_protect+0x3f4 (0x00007FFFBA299F74 [libj9prt29.so+0x39f74])
 [ERR] gpProtectAndRun+0xa8 (0x00007FFFBA4B11D8 [libj9vm29.so+0x1811d8])
 [ERR] gpCheckCallin+0xc4 (0x00007FFFBA376484 [libj9vm29.so+0x46484])
 [ERR] callStaticVoidMethod+0x48 (0x00007FFFBA3733C8 [libj9vm29.so+0x433c8])
 [ERR] JavaMain+0x11e4 (0x00007FFFBAD67304 [libjli.so+0x7304])
 [ERR] ThreadJavaMain+0x18 (0x00007FFFBAD6D258 [libjli.so+0xd258])
 [ERR] start_thread+0xf8 (0x00007FFFBAD19678 [libpthread-2.28.so+0x9678])
 [ERR] clone+0x74 (0x00007FFFBAC08738 [libc-2.28.so+0x138738])
 [ERR] ---------------------------------------
pshipton commented 1 year ago

Setting as a blocker since it appears a regression.

babsingh commented 1 year ago

Various links and files used to read the core file

Details from gdb and jdmpview

The below output is from core.20230813.013741.1418275.0001.dmp.

(gdb) where

...
#13 <signal handler called>
#14 jvmtiHookMethodExit (hook=<optimized out>, eventNum=<optimized out>, eventData=<optimized out>, userData=0x7fffb40bbd18)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/jvmti/jvmtiHook.c:1623
#15 0x00007fffba1f1960 in J9HookDispatch (hookInterface=0x7fffb402c848, taggedEventNum=<optimized out>, eventData=0x7fffbaa8c628)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/omr/util/hookable/hookable.cpp:235
#16 0x00007fffb98037cc in old_slow_jitReportMethodExit (currentThread=0x7fffb40df000)
--Type <RET> for more, q to quit, c to continue without paging--
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/codert_vm/cnathelp.cpp:1976
#17 0x00007fffb98181d0 in jitReportMethodExit () at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/build/linux-ppc64le-server-release/vm/runtime/codert_vm/pnathelp.s:5755
#18 0x00007fff2248b6e4 in ?? ()
#19 0x00007fffba349b6c in sendResolveInvokeDynamic (currentThread=0x21, ramCP=0x1, callSiteIndex=<optimized out>, nameAndSig=0x21, bsmData=0x7fff3d109298)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/callin.cpp:1120
#20 0x00007fffba3bf0b0 in resolveInvokeDynamic (vmThread=0x7fffb40df000, ramCP=0x7fffb4999e50, callSiteIndex=2, resolveFlags=0)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/resolvesupport.cpp:2227
#21 0x00007fffba499bec in VM_DebugBytecodeInterpreterFull::invokedynamic (_pc=<optimized out>, _sp=<optimized out>, this=<optimized out>)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/BytecodeInterpreter.hpp:8867
#22 VM_DebugBytecodeInterpreterFull::run (vmThread=<optimized out>, this=0x7fffbaa8d3b0)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/BytecodeInterpreter.hpp:11693
#23 debugBytecodeLoopFull (currentThread=<optimized out>) at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/BytecodeInterpreter.inc:112
#24 0x00007fffba49ddf0 in c_cInterpreter () at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/build/linux-ppc64le-server-release/vm/runtime/vm/pcinterp.s:329
#25 0x00007fffba349f18 in runCallInMethod (env=0x7fffb40df000, receiver=<optimized out>, clazz=0x7fffb423a310, methodID=0x7fffb4a1a4a8, args=0x7fffbaa8e668)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/callin.cpp:1174
...

(gdb) p *(J9VMMethodReturnEvent *)0x7fffbaa8c628
$1 = {currentThread = 0x7fffb40df000, method = 0x7fffb416a578, poppedByException = 0, returnValuePtr = 0x0, methodType = 1}

// jvmtiHookMethodExit - Method being exited - method = 0x7fffb416a578
> !j9method 0x7fffb416a578
J9Method at 0x7fffb416a578 {
  Fields for J9Method:
    0x0: U8* bytecodes = !j9x 0x00007FFEE947DFD8 // "*��"
    0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x00007FFFB4165230 (flags = 0x0)
    0x10: void* methodRunAddress = !j9x 0x0000000000000018
    0x18: volatile void* extra = !j9x 0x00007FFF2248AA10
}
Signature: java/lang/String.substring(II)Ljava/lang/String; !bytecodes 0x00007FFFB416A578
ROM Method: !j9rommethod 0x00007FFEE947DFC4
Next Method: !j9method 0x00007FFFB416A598

// jvmtiHookMethodExit - Thread exiting the method - currentThread = 0x7fffb40df000
!stack 0x7fffb40df000   !j9vmthread 0x7fffb40df000  !j9thread 0x7fffb401d6e0    tid 0x15a427 (1418279) // (main)

> !stack 0x7fffb40df000
<7fffb40df000>                          known but unhandled frame type com.ibm.j9ddr.vm29.pointer.U8Pointer @ 0x00000005

 FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT

<7fffb40df000>  !j9method 0x00007FFFB416AD38   java/lang/String.checkBoundsBeginEnd(III)V
<7fffb40df000>  !j9method 0x00007FFFB416A578   java/lang/String.substring(II)Ljava/lang/String;
<7fffb40df000>  !j9method 0x00007FFFB4369FA8   jdk/internal/org/objectweb/asm/Type.getInternalName()Ljava/lang/String;
...

The below output is from core.20230813.013751.1418275.0003.dmp.

(gdb) where

...
#13 <signal handler called>
#14 jvmtiHookMethodExit (hook=<optimized out>, eventNum=<optimized out>, eventData=<optimized out>, userData=0x7fffb40bbd18)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/jvmti/jvmtiHook.c:1623
#15 0x00007fffba1f1960 in J9HookDispatch (hookInterface=0x7fffb402c848, taggedEventNum=<optimized out>, eventData=0x7fff0321b6a8)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/omr/util/hookable/hookable.cpp:235
#16 0x00007fffb98037cc in old_slow_jitReportMethodExit (currentThread=0x7ffec4174100)
--Type <RET> for more, q to quit, c to continue without paging--
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/codert_vm/cnathelp.cpp:1976
#17 0x00007fffb98181d0 in jitReportMethodExit () at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/build/linux-ppc64le-server-release/vm/runtime/codert_vm/pnathelp.s:5755
#18 0x00007fff2248b6e4 in ?? ()
#19 0x00007fffba346cf0 in runJavaThread (currentThread=0x24) at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/callin.cpp:682
#20 0x00007fffba3e3cc8 in javaProtectedThreadProc (portLibrary=<optimized out>, entryarg=0x7ffec4174100)
    at /home/jenkins/workspace/Build_JDK17_ppc64le_linux_OMR/openj9/runtime/vm/vmthread.cpp:2104
...

(gdb) p *(J9VMMethodReturnEvent *)0x7fff0321b6a8 // eventData
$1 = {currentThread = 0x7ffec4174100, method = 0x7fffb416a578, poppedByException = 0, returnValuePtr = 0x0, methodType = 1}

// jvmtiHookMethodExit - Method being exited - method = 0x7fffb416a578
> !j9method 0x7fffb416a578
J9Method at 0x7fffb416a578 {
  Fields for J9Method:
  0x0: U8* bytecodes = !j9x 0x00007FFEE947DFD8 // "*��"
  0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x00007FFFB4165230 (flags = 0x0)
  0x10: void* methodRunAddress = !j9x 0x0000000000000018
  0x18: volatile void* extra = !j9x 0x00007FFF2248AA10
}
Signature: java/lang/String.substring(II)Ljava/lang/String; !bytecodes 0x00007FFFB416A578
ROM Method: !j9rommethod 0x00007FFEE947DFC4
Next Method: !j9method 0x00007FFFB416A598

// jvmtiHookMethodExit - Thread exiting the method - currentThread = 0x7ffec4174100
!stack 0x7ffec4174100   !j9vmthread 0x7ffec4174100  !j9thread 0x7ffec4171630    tid 0x15a43b (1418299) // (Attach API wait loop)

> !stack 0x7ffec4174100
<7ffec4174100>                          known but unhandled frame type com.ibm.j9ddr.vm29.pointer.U8Pointer @ 0x00000005

 FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT  FAULT

<7ffec4174100>  !j9method 0x00007FFFB416AD38   java/lang/String.checkBoundsBeginEnd(III)V
<7ffec4174100>  !j9method 0x00007FFFB416A578   java/lang/String.substring(II)Ljava/lang/String;
<7ffec4174100>  !j9method 0x00007FFFB416A558   java/lang/String.substring(I)Ljava/lang/String;
...

Analysis

The crash happens when the JIT triggers J9HOOK_VM_METHOD_RETURN for the JVMTI MethodExit event. JVMTI MethodExit doc: https://download.java.net/java/early_access/jdk21/docs/specs/jvmti.html#MethodExit.

https://github.com/eclipse-openj9/openj9/blob/bac839b1e50b8fba68832342dbffb8cd201d5d6a/runtime/codert_vm/cnathelp.cpp#L1976

A NULL (valueAddress / returnValuePtr) is passed by the JIT. The segfault occurs when the NULL returnValuePtr is dereferenced in jvmtiHookMethodExit.

https://github.com/eclipse-openj9/openj9/blob/bac839b1e50b8fba68832342dbffb8cd201d5d6a/runtime/jvmti/jvmtiHook.c#L1623

Requesting the JIT team to look at this failure since the NULL returnValuePtr is passed from the JIT. fyi @0xdaryl.

pshipton commented 1 year ago

@hzongaro fyi

hzongaro commented 1 year ago

Devin @jdmpapin, may I ask you to look at this failure? It's currently marked as a blocker for the 0.41 release.

jdmpapin commented 1 year ago

jitReportMethodExit stores the JIT return address on the stack at r1+552:

Dump of assembler code for function jitReportMethodExit:
   0x00007fffb9817f48 <+0>:     std     r0,288(r1)
   0x00007fffb9817f4c <+4>:     mflr    r0
   0x00007fffb9817f50 <+8>:     std     r0,552(r1)
   [...]
(gdb) x/xg $r1+552
0x7fffbaa8c8a8: 0x00007fff2248b6e4

This return address is here within an AOT body for String.substring:

0x7fff2248b6b8    lis  r3, 0
0x7fff2248b6bc    ori  r3, r3, 0x7fff
0x7fff2248b6c0    sldi r3, r3, 0x20
0x7fff2248b6c4    oris r3, r3, 0xb416
0x7fff2248b6c8    ori  r3, r3, 0xa578    <--- r3 = 0x7fffb416a578 (J9Method of String.substring) **
0x7fff2248b6cc    lis  r4, 0
0x7fff2248b6d0    ori  r4, r4, 0
0x7fff2248b6d4    sldi r4, r4, 0x20
0x7fff2248b6d8    oris r4, r4, 0
0x7fff2248b6dc    ori  r4, r4, 0         <--- r4 = 0 (null)
0x7fff2248b6e0    bl   0x7fff2267e6e0    <--- helper call, bci 2:62
0x7fff2248b6e4    b    0x7fff2248b52c    <--- return address

From the inlining table of this body, inlined site 2 is String.checkBoundsBeginEnd(III)V (J9Method 0x7fffb416ad38), which agrees with the stacks shown in Babneet's comment above. At offset +62 its bytecode has a (void) return0 instruction

The JIT generates exit hooks here: https://github.com/eclipse-openj9/openj9/blob/b47a6d454b305feeb6c45b173715e5a9471aa254/runtime/compiler/ilgen/Walker.cpp#L6825-L6837 It looks like it's expected that we would pass null when exiting a method with void return type

So I think that the null is correct, and that the bug is instead that we passed the J9Method of substring when we should have passed checkBoundsBeginEnd (at ** above). I'm guessing that from the point of view of the hook, the null is unexpected because substring returns a value

I'm trying to figure out why we passed the wrong J9Method. Since this is an AOT body, it might be a relocation problem

JasonFengJ9 commented 1 year ago

JDK21 ppc64le_linux(ubu20le-rtp-rt7-1)

variation: Mode610
JVM_OPTIONS:  -Xcompressedrefs -Xjit -Xgcpolicy:gencon 

Testing: emex001
Test start time: 2023/08/21 13:43:06 Pacific Standard Time
Running command: "/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java"  -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump    -agentlib:jvmtitest=test:emex001 -cp "/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 14 milliseconds
Time spent executing: 4532 milliseconds
Test result: FAILED
Output from test:
 [OUT] *** Testing [1/2]:   testMethodExitFromJava
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=000073DF41BD0870 Handler2=000073DF41AF96C0
 [ERR] R0=000073DF41A51A00 R1=000073DF4238C420 R2=000073DF3BF17B00 R3=0000000000000000
 [ERR] R4=0000000000000019 R5=0000000000000000 R6=000073DF3C0B6108 R7=00000000000F4240
 [ERR] R8=0000000000000015 R9=000073DEE546F35F R10=0000000000000000 R11=000073DF4238C4E0
 [ERR] R12=000073DF3BEA9F70 R13=000073DF423968F0 R14=0000000000192F50 R15=0000000000095800
 [ERR] R16=000073DF1D6F0038 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=00000000FFFF8E40
 [ERR] R20=000073DF3C061318 R21=000073DF42301F90 R22=000073DF3C0B6108 R23=00060374ED06BB35
 [ERR] R24=000000000000003B R25=0000000000000000 R26=00000000000BEA40 R27=0000000000095800
 [ERR] R28=000073DF1D8EED90 R29=000073DF3BF11240 R30=0000000000000000 R31=000073DF3C0B6108
 [ERR] NIP=000073DF3BEAA230 MSR=800000000280F033 ORIG_GPR3=C0000000003F8A30 CTR=000073DF3BEA9F70
 [ERR] LINK=000073DF3BEAA044 XER=0000000000000000 CCR=0000000028828824 SOFTE=0000000000000001
 [ERR] TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
 [ERR] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR1 40555663e0000000 (f: 3758096384.000000, d: 8.534985e+01)
 [ERR] FPR2 3fec834f00000000 (f: 0.000000, d: 8.910289e-01)
 [ERR] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
 [ERR] FPR4 40140e2860000000 (f: 1610612736.000000, d: 5.013826e+00)
 [ERR] FPR5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [ERR] FPR7 3fd4043057b6ee09 (f: 1471606272.000000, d: 3.127557e-01)
 [ERR] FPR8 656b6f766e696572 (f: 1852401024.000000, d: 3.557613e+180)
 [ERR] FPR9 bfdffffef20a4123 (f: 4060758272.000000, d: -4.999997e-01)
 [ERR] FPR10 bfe002622ad9e5e6 (f: 718923264.000000, d: -5.002910e-01)
 [ERR] FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
 [ERR] FPR12 3f50624dd2f1a9fc (f: 3539053056.000000, d: 1.000000e-03)
 [ERR] FPR13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [ERR] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
 [ERR] Module_base_address=000073DF3BE80000
 [ERR] Target=2_90_20230821_42 (Linux 5.4.0-155-generic)
 [ERR] CPU=ppc64le (4 logical CPUs) (0x1fcc00000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] jvmtiHookMethodExit+0x2c0 (0x000073DF3BEAA230 [libj9jvmti29.so+0x2a230])
 [ERR] J9HookDispatch+0x1e0 (0x000073DF41A51A00 [libj9hookable29.so+0x1a00])
 [ERR] old_slow_jitReportMethodExit+0x18c (0x000073DF410FF18C [libj9jit29.so+0xc2f18c])
 [ERR]  (0x000073DF41113B10 [libj9jit29.so+0xc43b10])
 [ERR] sendResolveInvokeDynamic+0x27c (0x000073DF41BAA75C [libj9vm29.so+0x1a75c])
 [ERR] resolveInvokeDynamic+0x240 (0x000073DF41C20630 [libj9vm29.so+0x90630])
 [ERR] debugBytecodeLoopCompressed+0x2f158 (0x000073DF41CC1A28 [libj9vm29.so+0x131a28])
 [ERR]  (0x000073DF41CFE6D0 [libj9vm29.so+0x16e6d0])
 [ERR] runCallInMethod+0x264 (0x000073DF41BAAB14 [libj9vm29.so+0x1ab14])
 [ERR] gpProtectedRunCallInMethod+0x54 (0x000073DF41BD4BA4 [libj9vm29.so+0x44ba4])
 [ERR] signalProtectAndRunGlue+0x28 (0x000073DF41D124C8 [libj9vm29.so+0x1824c8])
 [ERR] omrsig_protect+0x3e4 (0x000073DF41AFAB74 [libj9prt29.so+0x3ab74])
 [ERR] gpProtectAndRun+0xa8 (0x000073DF41D12598 [libj9vm29.so+0x182598])
 [ERR] gpCheckCallin+0xc4 (0x000073DF41BD73E4 [libj9vm29.so+0x473e4])
 [ERR] callStaticVoidMethod+0x48 (0x000073DF41BD41A8 [libj9vm29.so+0x441a8])
 [ERR] JavaMain+0x1504 (0x000073DF42699664 [libjli.so+0x9664])
 [ERR] ThreadJavaMain+0x18 (0x000073DF4269DAA8 [libjli.so+0xdaa8])
 [ERR] start_thread+0xe8 (0x000073DF42648838 [libpthread.so.0+0x8838])
 [ERR] clone+0x74 (0x000073DF4251B884 [libc.so.6+0x14b884])
 [ERR] ---------------------------------------
 [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/08/21 13:43:08 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/core.20230821.134308.214227.0001.dmp' in response to an event
 [ERR] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/core.20230821.134308.214227.0001.dmp
 [ERR] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/javacore.20230821.134308.214227.0002.txt' in response to an event
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=000073DF41BD0870 Handler2=000073DF41AF96C0
 [ERR] R0=000073DF41A51A00 R1=000073DF1CCEB4C0 R2=000073DF3BF17B00 R3=0000000000000000
 [ERR] R4=0000000000000019 R5=0000000000000000 R6=000073DF3C0B6108 R7=00000000000F4240
 [ERR] R8=0000000000000015 R9=000073DEE546F35F R10=0000000000000000 R11=000073DF1CCEB580
 [ERR] R12=000073DF3BEA9F70 R13=000073DF1CCF68F0 R14=0000000000095420 R15=000000000023CD00
 [ERR] R16=000073DF1D6F0038 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=0000000000000000
 [ERR] R20=0000000000000000 R21=000073DF42301F90 R22=000073DF3C0B6108 R23=00060374ED1B5425
 [ERR] R24=000000000000003B R25=0000000000000000 R26=00000000000BEA40 R27=000000000023CD00
 [ERR] R28=000073DF1D8EED90 R29=000073DF3BF11240 R30=0000000000000000 R31=000073DF3C0B6108
 [ERR] NIP=000073DF3BEAA230 MSR=800000000280F033 ORIG_GPR3=000073DF1D8EEEE0 CTR=000073DF3BEA9F70
 [ERR] LINK=000073DF3BEAA044 XER=0000000000000000 CCR=0000000024228448 SOFTE=0000000000000001
 [ERR] TRAP=0000000000000300 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
 [ERR] FPR0 0000000000095570 (f: 611696.000000, d: 3.022180e-318)
 [ERR] FPR1 4055565b00000000 (f: 0.000000, d: 8.534930e+01)
 [ERR] FPR2 41d0000000000000 (f: 0.000000, d: 1.073742e+09)
 [ERR] FPR3 0000000000000020 (f: 32.000000, d: 1.581010e-322)
 [ERR] FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
 [ERR] FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
 [ERR] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [ERR] FPR7 3fd4043057b6ee09 (f: 1471606272.000000, d: 3.127557e-01)
 [ERR] FPR8 3e46862640000000 (f: 1073741824.000000, d: 1.048856e-08)
 [ERR] FPR9 6f745f6d62695f6d (f: 1651072896.000000, d: 7.721944e+228)
 [ERR] FPR10 617474615f736c6f (f: 1601399936.000000, d: 2.875735e+161)
 [ERR] FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
 [ERR] FPR12 3f60624dd2f1a9fc (f: 3539053056.000000, d: 2.000000e-03)
 [ERR] FPR13 0000000000118790 (f: 1148816.000000, d: 5.675905e-318)
 [ERR] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
 [ERR] Module_base_address=000073DF3BE80000
 [ERR] Target=2_90_20230821_42 (Linux 5.4.0-155-generic)
 [ERR] CPU=ppc64le (4 logical CPUs) (0x1fcc00000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/javacore.20230821.134308.214227.0002.txt
 [ERR] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/Snap.20230821.134308.214227.0003.trc' in response to an event
 [ERR] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/Snap.20230821.134308.214227.0003.trc
 [ERR] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/jitdump.20230821.134308.214227.0004.dmp' in response to an event
 [ERR] JVMDUMP051I JIT dump occurred in 'main' thread 0x0000000000095800
 [ERR] JVMDUMP053I JIT dump is recompiling java/lang/String.substring(II)Ljava/lang/String;
 [ERR] jvmtiHookMethodExit+0x2c0 (0x000073DF3BEAA230 [libj9jvmti29.so+0x2a230])
 [ERR] J9HookDispatch+0x1e0 (0x000073DF41A51A00 [libj9hookable29.so+0x1a00])
 [ERR] old_slow_jitReportMethodExit+0x18c (0x000073DF410FF18C [libj9jit29.so+0xc2f18c])
 [ERR]  (0x000073DF41113B10 [libj9jit29.so+0xc43b10])
 [ERR] runJavaThread+0x250 (0x000073DF41BA7810 [libj9vm29.so+0x17810])
 [ERR] javaProtectedThreadProc+0xc8 (0x000073DF41C45128 [libj9vm29.so+0xb5128])
 [ERR] omrsig_protect+0x3e4 (0x000073DF41AFAB74 [libj9prt29.so+0x3ab74])
 [ERR] javaThreadProc+0x60 (0x000073DF41C408C0 [libj9vm29.so+0xb08c0])
 [ERR] thread_wrapper+0x190 (0x000073DF41A8D820 [libj9thr29.so+0xd820])
 [ERR] start_thread+0xe8 (0x000073DF42648838 [libpthread.so.0+0x8838])
 [ERR] clone+0x74 (0x000073DF4251B884 [libc.so.6+0x14b884])
 [ERR] ---------------------------------------
 [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/08/21 13:43:09 - please wait.
 [ERR] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_Personal_testList_1/aqa-tests/TKG/output_16926489613226/cmdLineTester_jvmtitests_8/jitdump.20230821.134308.214227.0004.dmp
 [ERR] JVMDUMP013I Processed dump event "gpf", detail "".
>> Success condition was not found: [Return code: 0]

cmdLineTester_jvmtitests_8_FAILED
jdmpapin commented 1 year ago

@dsouzai found that TR_RamMethodSequence always patches in a pointer to the J9Method of the outermost method:

https://github.com/eclipse-openj9/openj9/blob/f4c120ed8cd92953cd9e74da7a8f32f9e55ce724/runtime/compiler/runtime/RelocationRecord.cpp#L1383-L1389

I'm attaching a repro that (at least on PPC64LE Linux) reliably passes the wrong J9Method to jitReportMethodExit, which then crashes. Backtrace:

fillInJValue+0x100 (0x00007FFFB254ED60 [libj9jvmti29.so+0x1ed60])                  
J9HookDispatch+0x1e0 (0x00007FFFB8181960 [libj9hookable29.so+0x1960])              
old_slow_jitReportMethodExit+0x18c (0x00007FFFB37837CC [libj9jit29.so+0xc437cc])   

It just inlines a void method into a non-void method in an AOT compilation with JVMTI method exit tracing enabled

Repro.java, testagent.c. To run (with a given $jdk):

$ gcc -fPIC -shared -I"$jdk/include" -I"$jdk/include/linux" -o libtestagent.so testagent.c
$ export LD_LIBRARY_PATH="$PWD${LD_LIBRARY_PATH+:}${LD_LIBRARY_PATH:-}"
$ "$jdk/bin/javac" -g Repro.java
$ "$jdk/bin/java" -Xshareclasses:destroyAll
$ "$jdk/bin/java" -Xshareclasses -agentlib:testagent -X{jit,aot}:'forceAOT,limit=go,count=1,disableAsyncCompilation,disableGuardedCountingRecompilation,disableDelayRelocationForAOTCompilations,tryToInline={*.inline*}' Repro

I can see from the core that the incorrect (non-void) J9Method is passed alongside null in the same way as I described in my previous comment. Additionally, logging shows that the trees specified the correct J9Method, and that the constant materialization sequence is indeed relocated using TR_RamMethodSequence, confirming that the bug is due to the problem that Irwin identified

To fix this, I'll generalize TR_RamMethodSequence relocations so that they can also deal with the J9Method of any method that has been inlined in the compilation

pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_1/531 cmdLineTester_jvmtitests_1 cmdLineTester_jvmtitests_8 https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_ppc64le_linux_Nightly_testList_1/531/functional_test_output.tar.gz

jdmpapin commented 1 year ago

I've opened a fix for this. Rather than generalize TR_RamMethodSequence to support inlined methods, I've removed the paths that would require such support

jdmpapin commented 1 year ago

Reopening because the fix also has to go into 0.41

jdmpapin commented 1 year ago

Fix has been merged into the 0.41 release branch