search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

ppc64le_linux SecurityTests_0_FAILED Illegal instruction vmState=0x00040000 Java_jdk_crypto_jniprovider_NativeCrypto_GCMDecrypt #16371

Open JasonFengJ9 opened 1 year ago

JasonFengJ9 commented 1 year ago

Failure link

From an internal build(rhel8le-rt1-5):

openjdk version "19.0.1" 2022-10-18
IBM Semeru Runtime Open Edition 19.0.1+10 (build 19.0.1+10)
Eclipse OpenJ9 VM 19.0.1+10 (build master-8206cebaa, JRE 19 Linux ppc64le-64-Bit Compressed References 20221018_81 (JIT enabled, AOT enabled)
OpenJ9   - 8206cebaa
OMR      - dc42afa38
JCL      - 720d535776 based on jdk-19.0.1+10)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2022-11-26T17:59:01.395Z] variation: NoOptions
[2022-11-26T17:59:01.395Z] JVM_OPTIONS:  

[2022-11-26T17:59:05.765Z] Unhandled exception
[2022-11-26T17:59:05.765Z] Type=Illegal instruction vmState=0x00040000
[2022-11-26T17:59:05.765Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
[2022-11-26T17:59:05.765Z] Handler1=00007FFF8AF6FCE0 Handler2=00007FFF8AE98AA0
[2022-11-26T17:59:05.765Z] R0=00007FFF63E07CEC R1=00007FFF8B68CBC0 R2=00007FFF63FC7F00 R3=00000000FFE943C0
[2022-11-26T17:59:05.765Z] R4=00000000FFE92290 R5=00000000000009A0 R6=00007FFF84310010 R7=00007FFF8430FE48
[2022-11-26T17:59:05.765Z] R8=00007FFF8430FE88 R9=00000000000001B0 R10=0000000000000002 R11=00007FFF63D37F60
[2022-11-26T17:59:05.765Z] R12=0000000000004400 R13=00007FFF8B6968E0 R14=00007FFF63F7A7B8 R15=0000000000000005
[2022-11-26T17:59:05.765Z] R16=0000000000000010 R17=000000000000000C R18=0000000000000000 R19=00000000FFE939D0
[2022-11-26T17:59:05.765Z] R20=00000000000009B4 R21=00007FFF8430FE88 R22=00007FFF84310010 R23=0000000100000000
[2022-11-26T17:59:05.765Z] R24=00000000FFE92290 R25=00000000FFE943C0 R26=00000000000009A0 R27=0000000000000000
[2022-11-26T17:59:05.765Z] R28=0000000000000000 R29=0000000000000000 R30=000000000000009A R31=00007FFF8430FE48
[2022-11-26T17:59:05.765Z] NIP=00007FFF63D31B0C MSR=800000000288F033 ORIG_GPR3=C0000000000027E0 CTR=00007FFF63D37F60
[2022-11-26T17:59:05.765Z] LINK=00007FFF63E07CEC XER=0000000020000000 CCR=0000000048242444 SOFTE=0000000000000001
[2022-11-26T17:59:05.765Z] TRAP=0000000000000700 DAR=00007FFF845F0598 dsisr=0000000042000000 RESULT=0000000000000000
[2022-11-26T17:59:05.765Z] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR1 40538d0480000000 (f: 2147483648.000000, d: 7.820340e+01)
[2022-11-26T17:59:05.765Z] FPR2 3fe8000000000000 (f: 0.000000, d: 7.500000e-01)
[2022-11-26T17:59:05.765Z] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
[2022-11-26T17:59:05.765Z] FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
[2022-11-26T17:59:05.765Z] FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
[2022-11-26T17:59:05.765Z] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2022-11-26T17:59:05.765Z] FPR7 bfd2bef0a7c06ddb (f: 2814406144.000000, d: -2.929041e-01)
[2022-11-26T17:59:05.765Z] FPR8 c17a91d30e3fe8e2 (f: 239069408.000000, d: -2.786027e+07)
[2022-11-26T17:59:05.765Z] FPR9 476930a4e0000000 (f: 3758096384.000000, d: 1.046352e+36)
[2022-11-26T17:59:05.765Z] FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR11 0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2022-11-26T17:59:05.765Z] FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T17:59:05.765Z] Module=/lib64/libcrypto.so.3
[2022-11-26T17:59:05.765Z] Module_base_address=00007FFF63AC0000
[2022-11-26T17:59:05.765Z] Target=2_90_20221018_81 (Linux 4.18.0-372.32.1.el8_6.ppc64le)
[2022-11-26T17:59:05.765Z] CPU=ppc64le (4 logical CPUs) (0x1dcce0000 RAM)
[2022-11-26T17:59:05.765Z] ----------- Stack Backtrace -----------
[2022-11-26T17:59:05.765Z]  (0x00007FFF63D31B0C [libcrypto.so.3+0x271b0c])
[2022-11-26T17:59:05.765Z]  (0x00007FFF63E07CEC [libcrypto.so.3+0x347cec])
[2022-11-26T17:59:05.765Z]  (0x00007FFF63E08210 [libcrypto.so.3+0x348210])
[2022-11-26T17:59:05.765Z]  (0x00007FFF63E63D34 [libcrypto.so.3+0x3a3d34])
[2022-11-26T17:59:05.765Z]  (0x00007FFF63E64DF4 [libcrypto.so.3+0x3a4df4])
[2022-11-26T17:59:05.765Z] EVP_DecryptUpdate+0xd4 (0x00007FFF63CDB9F4 [libcrypto.so.3+0x21b9f4])
[2022-11-26T17:59:05.765Z] Java_jdk_crypto_jniprovider_NativeCrypto_GCMDecrypt+0x28c (0x00007FFF63FD536C [libjncrypto.so+0x536c])
[2022-11-26T17:59:05.765Z]  (0x00007FFF8B129998 [libj9vm29.so+0x1f9998])
[2022-11-26T17:59:05.765Z] ffi_call_int+0xd4 (0x00007FFF8B129664 [libj9vm29.so+0x1f9664])
[2022-11-26T17:59:05.765Z] bytecodeLoopCompressed+0x116cc (0x00007FFF8AFF676C [libj9vm29.so+0xc676c])
[2022-11-26T17:59:05.765Z]  (0x00007FFF8B096360 [libj9vm29.so+0x166360])
[2022-11-26T17:59:05.766Z] runCallInMethod+0x264 (0x00007FFF8AF4A0A4 [libj9vm29.so+0x1a0a4])
[2022-11-26T17:59:05.766Z] gpProtectedRunCallInMethod+0x54 (0x00007FFF8AF74084 [libj9vm29.so+0x44084])
[2022-11-26T17:59:05.766Z] signalProtectAndRunGlue+0x28 (0x00007FFF8B0A95D8 [libj9vm29.so+0x1795d8])
[2022-11-26T17:59:05.766Z] omrsig_protect+0x3f4 (0x00007FFF8AE99F74 [libj9prt29.so+0x39f74])
[2022-11-26T17:59:05.766Z] gpProtectAndRun+0xa8 (0x00007FFF8B0A96A8 [libj9vm29.so+0x1796a8])
[2022-11-26T17:59:05.766Z] gpCheckCallin+0xc4 (0x00007FFF8AF768E4 [libj9vm29.so+0x468e4])
[2022-11-26T17:59:05.766Z] callStaticVoidMethod+0x48 (0x00007FFF8AF73698 [libj9vm29.so+0x43698])
[2022-11-26T17:59:05.766Z] JavaMain+0x11e4 (0x00007FFF8B977344 [libjli.so+0x7344])
[2022-11-26T17:59:05.766Z] ThreadJavaMain+0x18 (0x00007FFF8B97D298 [libjli.so+0xd298])
[2022-11-26T17:59:05.766Z] start_thread+0xf8 (0x00007FFF8B929718 [libpthread.so.0+0x9718])
[2022-11-26T17:59:05.766Z] clone+0x74 (0x00007FFF8B80B498 [libc.so.6+0x13b498])
[2022-11-26T17:59:05.766Z] ---------------------------------------
[2022-11-26T17:59:05.766Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/26 09:59:05 - please wait.

[2022-11-26T17:59:06.644Z] 
[2022-11-26T17:59:06.644Z] SecurityTests_0_FAILED

50x internal grinder - all passed 50x internal grinder -Djdk.nativeGCM=false - all passed

JasonFengJ9 commented 1 year ago

JDK11 ppc64le_linux(rhel8le-rt1-3)

java version "11.0.18" 2023-01-17
IBM Semeru Runtime Certified Edition 11.0.18+4 (build 11.0.18+4)
Eclipse OpenJ9 VM 11.0.18+4 (build master-8206cebaa, JRE 11 Linux ppc64le-64-Bit Compressed References 20221125_447 (JIT enabled, AOT enabled)
OpenJ9   - 8206cebaa
OMR      - dc42afa38
JCL      - 2c5043df7e based on jdk-11.0.18+4)

[2022-11-26T03:05:20.403Z] variation: NoOptions
[2022-11-26T03:05:20.403Z] JVM_OPTIONS:  

[2022-11-26T03:05:23.901Z] Unhandled exception
[2022-11-26T03:05:23.901Z] Type=Illegal instruction vmState=0x00040000
[2022-11-26T03:05:23.901Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
[2022-11-26T03:05:23.901Z] Handler1=00007FFF7DFB4960 Handler2=00007FFF7DEDD440
[2022-11-26T03:05:23.901Z] R0=00007FFF54707CEC R1=00007FFF7E69C030 R2=00007FFF548C7F00 R3=00000000FFFAF580
[2022-11-26T03:05:23.901Z] R4=00000000FFFAE1E8 R5=00000000000009A0 R6=00007FFF78172FC0 R7=00007FFF78172DF8
[2022-11-26T03:05:23.901Z] R8=00007FFF78172E38 R9=00000000000001B0 R10=0000000000000002 R11=00007FFF54637F60
[2022-11-26T03:05:23.901Z] R12=0000000000004400 R13=00007FFF7E6A68E0 R14=000000000017ABC0 R15=00007FFF549000C8
[2022-11-26T03:05:23.901Z] R16=00000000FFFAEBA0 R17=00000000FFFAF580 R18=00000000FFFAE1E8 R19=00000000FFF3F9F8
[2022-11-26T03:05:23.901Z] R20=0000000000000000 R21=00007FFF78172E38 R22=00007FFF78172FC0 R23=0000000100000000
[2022-11-26T03:05:23.901Z] R24=00000000FFFAE1E8 R25=00000000FFFAF580 R26=00000000000009A0 R27=0000000000000000
[2022-11-26T03:05:23.901Z] R28=0000000000000000 R29=0000000000000000 R30=000000000000009A R31=00007FFF78172DF8
[2022-11-26T03:05:23.901Z] NIP=00007FFF54631B0C MSR=800000000288F033 ORIG_GPR3=C0000000000027E0 CTR=00007FFF54637F60
[2022-11-26T03:05:23.901Z] LINK=00007FFF54707CEC XER=0000000020000000 CCR=0000000048242444 SOFTE=0000000000000001
[2022-11-26T03:05:23.901Z] TRAP=0000000000000700 DAR=00007FFF784D0298 dsisr=0000000042000000 RESULT=0000000000000000
[2022-11-26T03:05:23.901Z] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR1 4050dcb960000000 (f: 1610612736.000000, d: 6.744881e+01)
[2022-11-26T03:05:23.901Z] FPR2 41e0000000000000 (f: 0.000000, d: 2.147484e+09)
[2022-11-26T03:05:23.901Z] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
[2022-11-26T03:05:23.901Z] FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
[2022-11-26T03:05:23.901Z] FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
[2022-11-26T03:05:23.901Z] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2022-11-26T03:05:23.901Z] FPR7 bfc6574f0ac07758 (f: 180385632.000000, d: -1.745395e-01)
[2022-11-26T03:05:23.901Z] FPR8 f6a7c4eec27af487 (f: 3262838016.000000, d: -3.742330e+263)
[2022-11-26T03:05:23.901Z] FPR9 f6a7c4eec27af487 (f: 3262838016.000000, d: -3.742330e+263)
[2022-11-26T03:05:23.901Z] FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR13 0000000000095700 (f: 612096.000000, d: 3.024156e-318)
[2022-11-26T03:05:23.901Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-11-26T03:05:23.901Z] Module=/lib64/libcrypto.so.3
[2022-11-26T03:05:23.901Z] Module_base_address=00007FFF543C0000
[2022-11-26T03:05:23.901Z] Target=2_90_20221125_447 (Linux 4.18.0-425.3.1.el8.ppc64le)
[2022-11-26T03:05:23.901Z] CPU=ppc64le (4 logical CPUs) (0x1dccb0000 RAM)
[2022-11-26T03:05:23.901Z] ----------- Stack Backtrace -----------
[2022-11-26T03:05:23.901Z]  (0x00007FFF54631B0C [libcrypto.so.3+0x271b0c])
[2022-11-26T03:05:23.901Z]  (0x00007FFF54707CEC [libcrypto.so.3+0x347cec])
[2022-11-26T03:05:23.901Z]  (0x00007FFF54708210 [libcrypto.so.3+0x348210])
[2022-11-26T03:05:23.901Z]  (0x00007FFF54763D34 [libcrypto.so.3+0x3a3d34])
[2022-11-26T03:05:23.901Z]  (0x00007FFF54764DF4 [libcrypto.so.3+0x3a4df4])
[2022-11-26T03:05:23.901Z] EVP_DecryptUpdate+0xd4 (0x00007FFF545DB9F4 [libcrypto.so.3+0x21b9f4])
[2022-11-26T03:05:23.901Z] Java_jdk_crypto_jniprovider_NativeCrypto_GCMDecrypt+0x410 (0x00007FFF548E59E0 [libjncrypto.so+0x59e0])
[2022-11-26T03:05:23.901Z]  (0x00007FFF7E17CDA8 [libj9vm29.so+0x20cda8])
[2022-11-26T03:05:23.901Z] ffi_call_int+0xd4 (0x00007FFF7E17CA74 [libj9vm29.so+0x20ca74])
[2022-11-26T03:05:23.901Z] bytecodeLoopCompressed+0x129b8 (0x00007FFF7E039808 [libj9vm29.so+0xc9808])
[2022-11-26T03:05:23.901Z]  (0x00007FFF7E0E7750 [libj9vm29.so+0x177750])
[2022-11-26T03:05:23.901Z] sidecarInvokeReflectMethodImpl+0x3c0 (0x00007FFF7DF8E640 [libj9vm29.so+0x1e640])
[2022-11-26T03:05:23.901Z] sidecarInvokeReflectMethod+0x38 (0x00007FFF7DF8FAD8 [libj9vm29.so+0x1fad8])
[2022-11-26T03:05:23.901Z] JVM_InvokeMethod_Impl+0xb4 (0x00007FFF7C291A84 [libjclse29.so+0x81a84])
[2022-11-26T03:05:23.901Z] JVM_InvokeMethod+0x40 (0x00007FFF7E5D4350 [libjvm.so+0x24350])
[2022-11-26T03:05:23.901Z] JVM_InvokeMethod+0x34 (0x00007FFF7E6A6C04 [libjvm.so+0x6c04])
[2022-11-26T03:05:23.901Z] Java_jdk_internal_reflect_NativeMethodAccessorImpl_invoke0+0x24 (0x00007FFF5C22D174 [libjava.so+0xd174])
[2022-11-26T03:05:23.901Z]  (0x00007FFF5D0A52A4 [<unknown>+0x0])
[2022-11-26T03:05:23.901Z] runCallInMethod+0x270 (0x00007FFF7DF8DF30 [libj9vm29.so+0x1df30])
[2022-11-26T03:05:23.901Z] gpProtectedRunCallInMethod+0x50 (0x00007FFF7DFB9CE0 [libj9vm29.so+0x49ce0])
[2022-11-26T03:05:23.901Z] signalProtectAndRunGlue+0x28 (0x00007FFF7E0F7C98 [libj9vm29.so+0x187c98])
[2022-11-26T03:05:23.901Z] omrsig_protect+0x358 (0x00007FFF7DEDE848 [libj9prt29.so+0x3e848])
[2022-11-26T03:05:23.901Z] gpProtectAndRun+0xac (0x00007FFF7E0F7D6C [libj9vm29.so+0x187d6c])
[2022-11-26T03:05:23.901Z] gpCheckCallin+0xc4 (0x00007FFF7DFBC7B4 [libj9vm29.so+0x4c7b4])
[2022-11-26T03:05:23.901Z] callStaticVoidMethod+0x48 (0x00007FFF7DFB92E8 [libj9vm29.so+0x492e8])
[2022-11-26T03:05:23.901Z] JavaMain+0x11c0 (0x00007FFF7E93C240 [libjli.so+0xc240])
[2022-11-26T03:05:23.901Z] ThreadJavaMain+0x18 (0x00007FFF7E942128 [libjli.so+0x12128])
[2022-11-26T03:05:23.901Z] start_thread+0xf8 (0x00007FFF7E979718 [libpthread.so.0+0x9718])
[2022-11-26T03:05:23.901Z] clone+0x74 (0x00007FFF7E81B498 [libc.so.6+0x13b498])
[2022-11-26T03:05:23.901Z] ---------------------------------------
[2022-11-26T03:05:23.901Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/25 19:05:23 - please wait.
[2022-11-26T03:05:24.738Z] 
[2022-11-26T03:05:24.738Z] SecurityTests_0_FAILED

50x grinder - all passed 50x grinder -Djdk.nativeGCM=false - no failure (some infra related issues)

JasonFengJ9 commented 1 year ago

JDK11 0.36.0 milestone 1 build(rhel8le-rt1-3)

openjdk version "11.0.18" 2023-01-17
IBM Semeru Runtime Open Edition 11.0.18.0-m1 (build 11.0.18+4)
Eclipse OpenJ9 VM 11.0.18.0-m1 (build v0.36.0-release-f98e55973, JRE 11 Linux ppc64le-64-Bit Compressed References 20221130_553 (JIT enabled, AOT enabled)
OpenJ9   - f98e55973
OMR      - acdbafc4a
JCL      - acdae91f88 based on jdk-11.0.18+4)

[2022-12-01T05:35:56.469Z] variation: NoOptions
[2022-12-01T05:35:56.469Z] JVM_OPTIONS:  

[2022-12-01T05:35:59.979Z] Unhandled exception
[2022-12-01T05:35:59.979Z] Type=Illegal instruction vmState=0x00040000
[2022-12-01T05:35:59.979Z] J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
[2022-12-01T05:35:59.979Z] Handler1=00007FFFACF04960 Handler2=00007FFFACE2D440
[2022-12-01T05:35:59.979Z] R0=00007FFF88657CEC R1=00007FFFAD5EC030 R2=00007FFF88817F00 R3=00000000FFF226D8
[2022-12-01T05:35:59.979Z] R4=00000000FFF21340 R5=00000000000009A0 R6=00007FFFA8173110 R7=00007FFFA8172F48
[2022-12-01T05:35:59.979Z] R8=00007FFFA8172F88 R9=00000000000001B0 R10=0000000000000002 R11=00007FFF88587F60
[2022-12-01T05:35:59.979Z] R12=0000000000004400 R13=00007FFFAD5F68E0 R14=000000000017ABC0 R15=00007FFF888400C8
[2022-12-01T05:35:59.979Z] R16=00000000FFF21CF8 R17=00000000FFF226D8 R18=00000000FFF21340 R19=00000000FFF20868
[2022-12-01T05:35:59.979Z] R20=0000000000000000 R21=00007FFFA8172F88 R22=00007FFFA8173110 R23=0000000100000000
[2022-12-01T05:35:59.979Z] R24=00000000FFF21340 R25=00000000FFF226D8 R26=00000000000009A0 R27=0000000000000000
[2022-12-01T05:35:59.979Z] R28=0000000000000000 R29=0000000000000000 R30=000000000000009A R31=00007FFFA8172F48
[2022-12-01T05:35:59.979Z] NIP=00007FFF88581B0C MSR=800000000288F033 ORIG_GPR3=C0000000000027E0 CTR=00007FFF88587F60
[2022-12-01T05:35:59.979Z] LINK=00007FFF88657CEC XER=0000000020000000 CCR=0000000048242444 SOFTE=0000000000000001
[2022-12-01T05:35:59.979Z] TRAP=0000000000000700 DAR=00007FFFA84D0228 dsisr=0000000042000000 RESULT=0000000000000000
[2022-12-01T05:35:59.979Z] FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR1 4053af9a60000000 (f: 1610612736.000000, d: 7.874380e+01)
[2022-12-01T05:35:59.979Z] FPR2 41e0000000000000 (f: 0.000000, d: 2.147484e+09)
[2022-12-01T05:35:59.979Z] FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
[2022-12-01T05:35:59.979Z] FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
[2022-12-01T05:35:59.979Z] FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
[2022-12-01T05:35:59.979Z] FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2022-12-01T05:35:59.979Z] FPR7 bfcb31d8a68224e9 (f: 2793547008.000000, d: -2.124587e-01)
[2022-12-01T05:35:59.979Z] FPR8 744c2dcfd8ac1f15 (f: 3635158784.000000, d: 1.614029e+252)
[2022-12-01T05:35:59.979Z] FPR9 1e67b456a3efa5ff (f: 2750391808.000000, d: 3.293079e-162)
[2022-12-01T05:35:59.979Z] FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR13 0000000000095700 (f: 612096.000000, d: 3.024156e-318)
[2022-12-01T05:35:59.979Z] FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2022-12-01T05:35:59.979Z] Module=/lib64/libcrypto.so.3
[2022-12-01T05:35:59.979Z] Module_base_address=00007FFF88310000
[2022-12-01T05:35:59.979Z] Target=2_90_20221130_553 (Linux 4.18.0-425.3.1.el8.ppc64le)
[2022-12-01T05:35:59.980Z] CPU=ppc64le (4 logical CPUs) (0x1dccb0000 RAM)
[2022-12-01T05:35:59.980Z] ----------- Stack Backtrace -----------
[2022-12-01T05:35:59.980Z]  (0x00007FFF88581B0C [libcrypto.so.3+0x271b0c])
[2022-12-01T05:35:59.980Z]  (0x00007FFF88657CEC [libcrypto.so.3+0x347cec])
[2022-12-01T05:35:59.980Z]  (0x00007FFF88658210 [libcrypto.so.3+0x348210])
[2022-12-01T05:35:59.980Z]  (0x00007FFF886B3D34 [libcrypto.so.3+0x3a3d34])
[2022-12-01T05:35:59.980Z]  (0x00007FFF886B4DF4 [libcrypto.so.3+0x3a4df4])
[2022-12-01T05:35:59.980Z] EVP_DecryptUpdate+0xd4 (0x00007FFF8852B9F4 [libcrypto.so.3+0x21b9f4])
[2022-12-01T05:35:59.980Z] Java_jdk_crypto_jniprovider_NativeCrypto_GCMDecrypt+0x410 (0x00007FFF888259E0 [libjncrypto.so+0x59e0])
[2022-12-01T05:35:59.980Z]  (0x00007FFFAD0CCDA8 [libj9vm29.so+0x20cda8])
[2022-12-01T05:35:59.980Z] ffi_call_int+0xd4 (0x00007FFFAD0CCA74 [libj9vm29.so+0x20ca74])
[2022-12-01T05:35:59.980Z] bytecodeLoopCompressed+0x129b8 (0x00007FFFACF89808 [libj9vm29.so+0xc9808])
[2022-12-01T05:35:59.980Z]  (0x00007FFFAD037750 [libj9vm29.so+0x177750])
[2022-12-01T05:35:59.980Z] sidecarInvokeReflectMethodImpl+0x3c0 (0x00007FFFACEDE640 [libj9vm29.so+0x1e640])
[2022-12-01T05:35:59.980Z] sidecarInvokeReflectMethod+0x38 (0x00007FFFACEDFAD8 [libj9vm29.so+0x1fad8])
[2022-12-01T05:35:59.980Z] JVM_InvokeMethod_Impl+0xb4 (0x00007FFFAC141A84 [libjclse29.so+0x81a84])
[2022-12-01T05:35:59.980Z] JVM_InvokeMethod+0x40 (0x00007FFFAD524350 [libjvm.so+0x24350])
[2022-12-01T05:35:59.980Z] JVM_InvokeMethod+0x34 (0x00007FFFAD5F6C04 [libjvm.so+0x6c04])
[2022-12-01T05:35:59.980Z] Java_jdk_internal_reflect_NativeMethodAccessorImpl_invoke0+0x24 (0x00007FFF8A16D174 [libjava.so+0xd174])
[2022-12-01T05:35:59.980Z]  (0x00007FFF8BFE3E24 [<unknown>+0x0])
[2022-12-01T05:35:59.980Z] runCallInMethod+0x270 (0x00007FFFACEDDF30 [libj9vm29.so+0x1df30])
[2022-12-01T05:35:59.980Z] gpProtectedRunCallInMethod+0x50 (0x00007FFFACF09CE0 [libj9vm29.so+0x49ce0])
[2022-12-01T05:35:59.980Z] signalProtectAndRunGlue+0x28 (0x00007FFFAD047C98 [libj9vm29.so+0x187c98])
[2022-12-01T05:35:59.980Z] omrsig_protect+0x358 (0x00007FFFACE2E848 [libj9prt29.so+0x3e848])
[2022-12-01T05:35:59.980Z] gpProtectAndRun+0xac (0x00007FFFAD047D6C [libj9vm29.so+0x187d6c])
[2022-12-01T05:35:59.980Z] gpCheckCallin+0xc4 (0x00007FFFACF0C7B4 [libj9vm29.so+0x4c7b4])
[2022-12-01T05:35:59.980Z] callStaticVoidMethod+0x48 (0x00007FFFACF092E8 [libj9vm29.so+0x492e8])
[2022-12-01T05:35:59.980Z] JavaMain+0x11c0 (0x00007FFFAD88C240 [libjli.so+0xc240])
[2022-12-01T05:35:59.980Z] ThreadJavaMain+0x18 (0x00007FFFAD892128 [libjli.so+0x12128])
[2022-12-01T05:35:59.980Z] start_thread+0xf8 (0x00007FFFAD8C9718 [libpthread.so.0+0x9718])
[2022-12-01T05:35:59.980Z] clone+0x74 (0x00007FFFAD76B498 [libc.so.6+0x13b498])
[2022-12-01T05:35:59.980Z] ---------------------------------------
[2022-12-01T05:35:59.980Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/30 21:35:59 - please wait.

[2022-12-01T05:36:00.827Z] SecurityTests_0_FAILED

500x grinder - one infra failure, other passed

pshipton commented 1 year ago

The jenkins agent is crashing in libcrypto.so.3 (OpenSSL 3) on plinux rhel 8 trying to run Semeru testing. Details below are from rhel8le-rt1-4, but it occurs often on multiple machines. Signal 4 is SIGILL, illegal instruction.

Copied diags16371.zip with core, javacore, etc. to team/pshipton

openssl atm is 3.0.1-43.el8.1 (latest version) on rhel8le-rt1-4.

1CIJAVAVERSION JRE 17 Linux ppc64le-64 (build 17.0.5+8)
1CIVMVERSION   20221018_307
1CIJ9VMTAG     openj9-0.35.0
1CIJ9VMVERSION e04a7f6c1
1CIJITVERSION  j9jit_20221031_1741_jenkins
1CIOMRVERSION  85a21674f_CMPRSS
1CIJCLVERSION  32d2c409a33 based on jdk-17.0.5+8
1TISIGINFO     Dump Event "gpf" (00002000) received

0SECTION       GPINFO subcomponent dump routine
NULL           ================================
2XHOSLEVEL     OS Level         : Linux 4.18.0-425.10.1.el8_7.ppc64le
2XHCPUS        Processors -
3XHCPUARCH       Architecture   : ppc64le
3XHNUMCPUS       How Many       : 4
3XHNUMASUP       NUMA is either not supported or has been disabled by user
NULL
1XHEXCPCODE    J9Generic_Signal_Number: 00000048
1XHEXCPCODE    Signal_Number: 00000004
1XHEXCPCODE    Error_Value: 00000000
1XHEXCPCODE    Signal_Code: 00000001
1XHEXCPCODE    Handler1: 00007FFFB8FCF270
1XHEXCPCODE    Handler2: 00007FFFB8EF87E0
NULL
1XHEXCPMODULE  Module: /lib64/libcrypto.so.3
1XHEXCPMODULE  Module_base_address: 00007FFF944D0000
NULL
1XHREGISTERS   Registers:
2XHREGISTER      R0: 00007FFF94817CEC
2XHREGISTER      R1: 00007FFF9431AB10
2XHREGISTER      R2: 00007FFF949D7F00
2XHREGISTER      R3: 00000000FFCC4688
2XHREGISTER      R4: 00000000FFCC2290
2XHREGISTER      R5: 00000000000010F0
2XHREGISTER      R6: 00007FFF3C2428E0
2XHREGISTER      R7: 00007FFF3C242718
2XHREGISTER      R8: 00007FFF3C242758
2XHREGISTER      R9: 00000000000001B0
2XHREGISTER      R10: 0000000000000002
2XHREGISTER      R11: 00007FFF94747F60
2XHREGISTER      R12: 0000000000004400
2XHREGISTER      R13: 00007FFF943268E0
2XHREGISTER      R14: 00007FFF9498A7B8
2XHREGISTER      R15: 0000000000000005
2XHREGISTER      R16: 0000000000000010
2XHREGISTER      R17: 000000000000000C
2XHREGISTER      R18: 0000000000000000
2XHREGISTER      R19: 00000000FFCC3548
2XHREGISTER      R20: 0000000000001101
2XHREGISTER      R21: 00007FFF3C242758
2XHREGISTER      R22: 00007FFF3C2428E0
2XHREGISTER      R23: 0000000100000000
2XHREGISTER      R24: 00000000FFCC2290
2XHREGISTER      R25: 00000000FFCC4688
2XHREGISTER      R26: 00000000000010F0
2XHREGISTER      R27: 0000000000000000
2XHREGISTER      R28: 0000000000000000
2XHREGISTER      R29: 0000000000000000
2XHREGISTER      R30: 000000000000010F
2XHREGISTER      R31: 00007FFF3C242718
2XHREGISTER      FPR0: 000000000000000E
2XHREGISTER      FPR1: 40518CBD40000000
2XHREGISTER      FPR2: 3FF009E8AB742AEB
2XHREGISTER      FPR3: 3FEE666660000000
2XHREGISTER      FPR4: 3FCE840B4AC4E4D2
2XHREGISTER      FPR5: BFE7154748BEF6C8
2XHREGISTER      FPR6: 3FE62E42FEFA39EF
2XHREGISTER      FPR7: 3FD4043057B6EE09
2XHREGISTER      FPR8: 3A206062E0000000
2XHREGISTER      FPR9: BFD00EA348B88334
2XHREGISTER      FPR10: 0000000000000000
2XHREGISTER      FPR11: 0000000000000000
2XHREGISTER      FPR12: 0000000000000000
2XHREGISTER      FPR13: 4031400000000000
2XHREGISTER      FPR14: 0000000000000000
2XHREGISTER      FPR15: 0000000000000000
2XHREGISTER      FPR16: 0000000000000000
2XHREGISTER      FPR17: 0000000000000000
2XHREGISTER      FPR18: 0000000000000000
2XHREGISTER      FPR19: 0000000000000000
2XHREGISTER      FPR20: 0000000000000000
2XHREGISTER      FPR21: 0000000000000000
2XHREGISTER      FPR22: 0000000000000000
2XHREGISTER      FPR23: 0000000000000000
2XHREGISTER      FPR24: 0000000000000000
2XHREGISTER      FPR25: 0000000000000000
2XHREGISTER      FPR26: 0000000000000000
2XHREGISTER      FPR27: 0000000000000000
2XHREGISTER      FPR28: 0000000000000000
2XHREGISTER      FPR29: 0000000000000000
2XHREGISTER      FPR30: 0000000000000000
2XHREGISTER      FPR31: 0000000000000000
2XHREGISTER      NIP: 00007FFF94741B0C
2XHREGISTER      MSR: 800000000288F033
2XHREGISTER      ORIG_GPR3: C0000000000087D0
2XHREGISTER      CTR: 00007FFF94747F60
2XHREGISTER      LINK: 00007FFF94817CEC
2XHREGISTER      XER: 0000000020000000
2XHREGISTER      CCR: 0000000048242444
2XHREGISTER      SOFTE: 0000000000000001
2XHREGISTER      TRAP: 0000000000000700
2XHREGISTER      DAR: 0000000000CA7C20
2XHREGISTER      dsisr: 0000000042000000
2XHREGISTER      RESULT: 0000000000000000
NULL
1XHFLAGS       VM flags:0000000000040000
1XMCURTHDINFO  Current thread
3XMTHREADINFO      "consumer_0" J9VMThread:0x00000000004E0400, omrthread_t:0x00007FFF2013C878, java/lang/Thread:0x0000000089AF6708, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x14E, isDaemon:false)
3XMJAVALTHRCCL            hudson/remoting/RemoteClassLoader(0x00000000898FA640)
3XMTHREADINFO1            (native thread ID:0x27EE39, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x40400020)
3XMTHREADINFO2            (native stack address range from:0x00007FFF942A0000, to:0x00007FFF94320000, size:0x80000)
3XMCPUTIME               CPU usage total: 1.606200068 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=3430280 (0x345788)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at jdk/crypto/jniprovider/NativeCrypto.GCMDecrypt(Native Method)
4XESTACKTRACE                at com/sun/crypto/provider/NativeGaloisCounterMode$GCMDecrypt.doFinal(NativeGaloisCounterMode.java:963)
5XESTACKTRACE                   (entered lock: com/sun/crypto/provider/NativeGaloisCounterMode$GCMDecrypt@0x00000000FFCC3498, entry count: 1)
4XESTACKTRACE                at com/sun/crypto/provider/NativeGaloisCounterMode$GCMDecrypt.doFinal(NativeGaloisCounterMode.java:1001)
4XESTACKTRACE                at com/sun/crypto/provider/NativeGaloisCounterMode.engineDoFinal(NativeGaloisCounterMode.java:454)
4XESTACKTRACE                at javax/crypto/Cipher.doFinal(Cipher.java:2500)
4XESTACKTRACE                at sun/security/ssl/SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1933)
4XESTACKTRACE                at sun/security/ssl/SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:260)
4XESTACKTRACE                at sun/security/ssl/SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181)
4XESTACKTRACE                at sun/security/ssl/SSLTransport.decode(SSLTransport.java:111)
4XESTACKTRACE                at sun/security/ssl/SSLSocketImpl.decode(SSLSocketImpl.java:1505)
4XESTACKTRACE                at sun/security/ssl/SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420)
4XESTACKTRACE                at sun/security/ssl/SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
4XESTACKTRACE                at sun/security/ssl/SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
4XESTACKTRACE                at org/apache/http/conn/ssl/SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
4XESTACKTRACE                at org/apache/http/conn/ssl/SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
4XESTACKTRACE                at org/apache/http/impl/conn/DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
4XESTACKTRACE                at org/apache/http/impl/conn/PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
4XESTACKTRACE                at org/apache/http/impl/execchain/MainClientExec.establishRoute(MainClientExec.java:393)
4XESTACKTRACE                at org/apache/http/impl/execchain/MainClientExec.execute(MainClientExec.java:236)
4XESTACKTRACE                at org/apache/http/impl/execchain/ProtocolExec.execute(ProtocolExec.java:186)
4XESTACKTRACE                at org/apache/http/impl/execchain/RetryExec.execute(RetryExec.java:89)
4XESTACKTRACE                at org/apache/http/impl/execchain/ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
4XESTACKTRACE                at org/apache/http/impl/execchain/RedirectExec.execute(RedirectExec.java:110)
4XESTACKTRACE                at org/apache/http/impl/client/InternalHttpClient.doExecute(InternalHttpClient.java:185)
4XESTACKTRACE                at org/apache/http/impl/client/CloseableHttpClient.execute(CloseableHttpClient.java:83)
4XESTACKTRACE                at org/jfrog/build/client/PreemptiveHttpClient.execute(PreemptiveHttpClient.java:72)
4XESTACKTRACE                at org/jfrog/build/client/JFrogHttpClient.sendRequest(JFrogHttpClient.java:133)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/JFrogService.execute(JFrogService.java:112)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/artifactory/services/Upload$TryChecksumUpload.ensureRequirements(Upload.java:122)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/JFrogService.execute(JFrogService.java:111)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/artifactory/services/Upload.tryChecksumUpload(Upload.java:83)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/artifactory/services/Upload.execute(Upload.java:72)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/artifactory/ArtifactoryManager.upload(ArtifactoryManager.java:262)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/client/artifactory/ArtifactoryManager.upload(ArtifactoryManager.java:257)
4XESTACKTRACE                at org/jfrog/build/extractor/clientConfiguration/util/spec/SpecDeploymentConsumer.consumerRun(SpecDeploymentConsumer.java:39)
4XESTACKTRACE                at org/jfrog/build/extractor/producerConsumer/ConsumerRunnableBase.run(ConsumerRunnableBase.java:11)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:857)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               protectedBacktrace+0x24 (0x00007FFFB8EF2C74 [libj9prt29.so+0x32c74])
4XENATIVESTACK               omrsig_protect+0x3f4 (0x00007FFFB8EF9CB4 [libj9prt29.so+0x39cb4])
4XENATIVESTACK               omrintrospect_backtrace_thread_raw+0xe0 (0x00007FFFB8EF3290 [libj9prt29.so+0x33290])
4XENATIVESTACK               protectedIntrospectBacktraceThread+0x24 (0x00007FFFB8EF28B4 [libj9prt29.so+0x328b4])
4XENATIVESTACK               omrsig_protect+0x3f4 (0x00007FFFB8EF9CB4 [libj9prt29.so+0x39cb4])
4XENATIVESTACK               omrintrospect_backtrace_thread+0x94 (0x00007FFFB8EF29C4 [libj9prt29.so+0x329c4])
4XENATIVESTACK               setup_native_thread+0x234 (0x00007FFFB8EF47F4 [libj9prt29.so+0x347f4])
4XENATIVESTACK               omrintrospect_threads_startDo_with_signal+0x484 (0x00007FFFB8EF5704 [libj9prt29.so+0x35704])
4XENATIVESTACK               protectedStartDoWithSignal+0x30 (0x00007FFFB88D8CC0 [libj9dmp29.so+0x18cc0])
4XENATIVESTACK               omrsig_protect+0x3f4 (0x00007FFFB8EF9CB4 [libj9prt29.so+0x39cb4])
4XENATIVESTACK               _ZN18JavaCoreDumpWriter28writeThreadsWithNativeStacksEv+0x3a8 (0x00007FFFB88E3D48 [libj9dmp29.so+0x23d48])
4XENATIVESTACK               protectedWriteThreadsWithNativeStacks+0x1c (0x00007FFFB88E455C [libj9dmp29.so+0x2455c])
4XENATIVESTACK               omrsig_protect+0x3f4 (0x00007FFFB8EF9CB4 [libj9prt29.so+0x39cb4])
4XENATIVESTACK               _ZN18JavaCoreDumpWriter18writeThreadSectionEv+0x1a4 (0x00007FFFB88DFAA4 [libj9dmp29.so+0x1faa4])
4XENATIVESTACK               protectedWriteSection+0x48 (0x00007FFFB88D8C68 [libj9dmp29.so+0x18c68])
4XENATIVESTACK               omrsig_protect+0x3f4 (0x00007FFFB8EF9CB4 [libj9prt29.so+0x39cb4])
4XENATIVESTACK               _ZN18JavaCoreDumpWriterC2EPKcP16J9RASdumpContextP14J9RASdumpAgent+0x440 (0x00007FFFB88DA9E0 [libj9dmp29.so+0x1a9e0])
4XENATIVESTACK               runJavadump+0x34 (0x00007FFFB88E7EF4 [libj9dmp29.so+0x27ef4])
4XENATIVESTACK               doJavaDump+0x6c (0x00007FFFB88C5D5C [libj9dmp29.so+0x5d5c])
4XENATIVESTACK               protectedDumpFunction+0x30 (0x00007FFFB88C4ED0 [libj9dmp29.so+0x4ed0])
4XENATIVESTACK               omrsig_protect+0x3f4 (0x00007FFFB8EF9CB4 [libj9prt29.so+0x39cb4])
4XENATIVESTACK               runDumpFunction+0x6c (0x00007FFFB88CA63C [libj9dmp29.so+0xa63c])
4XENATIVESTACK               runDumpAgent+0x198 (0x00007FFFB88CA838 [libj9dmp29.so+0xa838])
4XENATIVESTACK               0x00007FFFB88EB7E0
4XENATIVESTACK               0x00007FFFB8FCF238
4XENATIVESTACK               0x00007FFFB8EF9CB4
4XENATIVESTACK               0x00007FFFB8FCF408
4XENATIVESTACK               0x00007FFFB8EF8A74
4XENATIVESTACK               0x00007FFFB9A604D8
4XENATIVESTACK                (0x00007FFF94741B0C [libcrypto.so.3+0x271b0c])
4XENATIVESTACK               0x00007FFF94817CEC
4XENATIVESTACK               0x00007FFF94818210
4XENATIVESTACK               0x00007FFF94873D34
4XENATIVESTACK               0x00007FFF94874DF4
4XENATIVESTACK               0x00007FFF946EB9F4
4XENATIVESTACK               0x00007FFF949F52DC
4XENATIVESTACK               0x00007FFFB9181AF8
4XENATIVESTACK               0x00007FFFB91817C4
4XENATIVESTACK               0x00007FFFB905151C
4XENATIVESTACK               0x00007FFFB90F1C98
4XENATIVESTACK               0x00007FFFB8FA68F0
4XENATIVESTACK               0x00007FFFB903E3A8
4XENATIVESTACK               0x00007FFFB8EF9CB4
4XENATIVESTACK               0x00007FFFB9039CC0
4XENATIVESTACK               0x00007FFFB8E8CBC0
4XENATIVESTACK               0x00007FFFB9989718
4XENATIVESTACK               0x00007FFFB986B498
jasonkatonica commented 1 year ago

I added additional debugging information to a custom Java 11 ppcle build that is based upon the latest Java 11 code. I then attempted to recreate this issue in two ways:

  1. I ran the above reported Grinder failing test SecurityTests_0 against my build. I ran Grinder with 500 iterations without failure. This job and its debug output can be seen here https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/30976/
  2. I then created a testcase to closely simulate the calls being done in this test. I based my test upon the debug messages printed in the recreate attempt number 1 above. I performed AES/GCM in a loop for a few hours without failure.

For number 2 above, I ran my test on a ppcle environment as follows:

I built openssl 3.0.1 which reports the following:

# openssl version
OpenSSL 3.0.1 14 Dec 2021 (Library: OpenSSL 3.0.1 14 Dec 2021)
# uname -a
Linux katonicabuildjdk11j9c1.fyre.ibm.com 4.18.0-372.32.1.el8_6.ppc64le #1 SMP Fri Oct 7 11:37:39 EDT 2022 ppc64le ppc64le ppc64le GNU/Linux
# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

I am unsure why I cant recreate this in grinder at the moment as originally reported.

Perhaps this could have something to do with the rhel8le-rt1-5 OS vs the vanilla RHEL 8.6. I am unsure what Grinder is running on for my test job run above. My local system does not have the real time kernel installed so perhaps this is a factor here??

pshipton commented 1 year ago

We couldn't recreate the test suite failure either, but there didn't seem to be a problem recreating the problem using the last release (0.35) JVM as the jenkins agent, as per https://github.com/eclipse-openj9/openj9/issues/16371#issuecomment-1409235060. Assuming it can be recreated, what is the next step? We are currently running the jenkins agent using -Djdk.nativeGCM=false but that can be removed to recreate the crash.

pshipton commented 1 year ago

I am unsure what Grinder is running on for my test job run above.

The grinder shows rhel7lert-11, it's near the top of the grinder page.

jasonkatonica commented 1 year ago

Hello, If this can be recreated through the Jenkins agent could we run the Jenkins agent with -Djdk.nativeGCM=false removed and the agent running with my debug Java 11 SDK from the following location?

https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Build_JDK11_ppc64le_linux_Personal/1453/OpenJ9-JDK11-ppc64le_linux-20230210-074121.tar.gz

If we do observe a crash with this SDK, I should be able to see all the debug values sent to stdout for the GCM operation. Once I have those I could attempt to recreate this outside of the Jenkins agent.

In the meantime I will also try to install the real time kernel from RH described here to see if that makes any difference on my system:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/8/html-single/installing_rhel_8_for_real_time/index

pshipton commented 1 year ago

@AdamBrousseau can we make the previous comment happen?

@jasonkatonica In the meantime, try updating your test machine to 8.7, which is where the jenkins crash occurred. That may get you to the same kernel level Linux 4.18.0-425.10.1.el8_7.ppc64le where we saw the crash.

jasonkatonica commented 1 year ago

The grinder shows rhel7lert-11, it's near the top of the grinder page.

OK thank you I see it now.

jasonkatonica commented 1 year ago

@jasonkatonica In the meantime, try updating your test machine to 8.7, which is where the jenkins crash occurred. That may get you to the same kernel level Linux 4.18.0-425.10.1.el8_7.ppc64le where we saw the crash.

Sure ill give this a try and see what happens.

JasonFengJ9 commented 1 year ago

We couldn't recreate the test suite failure either, but there didn't seem to be a problem recreating the problem using the last release (0.35) JVM

Just launched a 100x grinder against 0.35 release.

pshipton commented 1 year ago

@JasonFengJ9 pls direct the grinder to sw.os.rhel.8

JasonFengJ9 commented 1 year ago

50x grinder at sw.os.rhel.8

JasonFengJ9 commented 1 year ago

50x grinder at sw.os.rhel.8 w/ JDK11 0.35 release reproduced Illegal instruction vmState=0x00040000

16:19:44  openjdk version "11.0.17" 2022-10-18
16:19:44  IBM Semeru Runtime Open Edition 11.0.17.0 (build 11.0.17+8)
16:19:44  Eclipse OpenJ9 VM 11.0.17.0 (build openj9-0.35.0, JRE 11 Linux ppc64le-64-Bit Compressed References 20221031_530 (JIT enabled, AOT enabled)
16:19:44  OpenJ9   - e04a7f6c1
16:19:44  OMR      - 85a21674f
16:19:44  JCL      - a94c231303 based on jdk-11.0.17+8)

variation: NoOptions
JVM_OPTIONS: 

Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
Handler1=00007FFF912848F0 Handler2=00007FFF911AD180
R0=00007FFF13C27CEC R1=00007FFF9196C030 R2=00007FFF13DE7F00 R3=00000000FFFDFE58
R4=00000000FFFDE880 R5=0000000000000AC0 R6=00007FFF8C171370 R7=00007FFF8C1711A8
R8=00007FFF8C1711E8 R9=00000000000001B0 R10=0000000000000002 R11=00007FFF13B57F60
R12=0000000000004400 R13=00007FFF919768E0 R14=000000000017AB60 R15=00007FFF680500C0
R16=00000000FFFDF358 R17=00000000FFFDFE58 R18=00000000FFFDE880 R19=00000000FFFDDC88
R20=0000000000000000 R21=00007FFF8C1711E8 R22=00007FFF8C171370 R23=0000000100000000
R24=00000000FFFDE880 R25=00000000FFFDFE58 R26=0000000000000AC0 R27=0000000000000000
R28=0000000000000000 R29=0000000000000000 R30=00000000000000AC R31=00007FFF8C1711A8
NIP=00007FFF13B51B0C MSR=800000000288F033 ORIG_GPR3=C0000000000027E0 CTR=00007FFF13B57F60
LINK=00007FFF13C27CEC XER=0000000020000000 CCR=0000000048242444 SOFTE=0000000000000001
TRAP=0000000000000700 DAR=00007FFF8C4D00A8 dsisr=0000000042000000 RESULT=0000000000000000
FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR1 4053a15e00000000 (f: 0.000000, d: 7.852136e+01)
FPR2 41e0000000000000 (f: 0.000000, d: 2.147484e+09)
FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
FPR7 bfd01eae7f513a67 (f: 2136029824.000000, d: -2.518727e-01)
FPR8 e995cdd0e3441bbd (f: 3812891648.000000, d: -4.172465e+200)
FPR9 00905441e85fbf9e (f: 3898589184.000000, d: 5.813363e-306)
FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR13 0000000000095700 (f: 612096.000000, d: 3.024156e-318)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib64/libcrypto.so.3
Module_base_address=00007FFF138E0000
Target=2_90_20221031_530 (Linux 4.18.0-425.10.1.el8_7.ppc64le)
CPU=ppc64le (4 logical CPUs) (0x1dccb0000 RAM)
----------- Stack Backtrace -----------
 (0x00007FFF13B51B0C [libcrypto.so.3+0x271b0c])
 (0x00007FFF13C27CEC [libcrypto.so.3+0x347cec])
 (0x00007FFF13C28210 [libcrypto.so.3+0x348210])
 (0x00007FFF13C83D34 [libcrypto.so.3+0x3a3d34])
 (0x00007FFF13C84DF4 [libcrypto.so.3+0x3a4df4])
EVP_DecryptUpdate+0xd4 (0x00007FFF13AFB9F4 [libcrypto.so.3+0x21b9f4])
Java_jdk_crypto_jniprovider_NativeCrypto_GCMDecrypt+0x410 (0x00007FFF68035970 [libjncrypto.so+0x5970])
 (0x00007FFF9144C868 [libj9vm29.so+0x20c868])
ffi_call_int+0xd4 (0x00007FFF9144C534 [libj9vm29.so+0x20c534])
bytecodeLoopCompressed+0x129b8 (0x00007FFF913093C8 [libj9vm29.so+0xc93c8])
 (0x00007FFF913B7310 [libj9vm29.so+0x177310])
sidecarInvokeReflectMethodImpl+0x3c0 (0x00007FFF9125E5E0 [libj9vm29.so+0x1e5e0])
sidecarInvokeReflectMethod+0x38 (0x00007FFF9125FA78 [libj9vm29.so+0x1fa78])
JVM_InvokeMethod_Impl+0xb4 (0x00007FFF904D1664 [libjclse29.so+0x81664])
JVM_InvokeMethod+0x40 (0x00007FFF918A42F0 [libjvm.so+0x242f0])
JVM_InvokeMethod+0x34 (0x00007FFF91976C04 [libjvm.so+0x6c04])
Java_jdk_internal_reflect_NativeMethodAccessorImpl_invoke0+0x24 (0x00007FFF6A52CFD4 [libjava.so+0xcfd4])
 (0x00007FFF703D3F24 [<unknown>+0x0])
runCallInMethod+0x270 (0x00007FFF9125DED0 [libj9vm29.so+0x1ded0])
gpProtectedRunCallInMethod+0x50 (0x00007FFF91289C60 [libj9vm29.so+0x49c60])
signalProtectAndRunGlue+0x28 (0x00007FFF913C7758 [libj9vm29.so+0x187758])
omrsig_protect+0x358 (0x00007FFF911AE588 [libj9prt29.so+0x3e588])
gpProtectAndRun+0xac (0x00007FFF913C782C [libj9vm29.so+0x18782c])
gpCheckCallin+0xc4 (0x00007FFF9128C734 [libj9vm29.so+0x4c734])
callStaticVoidMethod+0x48 (0x00007FFF91289268 [libj9vm29.so+0x49268])
JavaMain+0x11c0 (0x00007FFF91C0C240 [libjli.so+0xc240])
ThreadJavaMain+0x18 (0x00007FFF91C12128 [libjli.so+0x12128])
start_thread+0xf8 (0x00007FFF91C49718 [libpthread.so.0+0x9718])
clone+0x74 (0x00007FFF91AEB498 [libc.so.6+0x13b498])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/02/10 13:23:51 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/core.20230210.132351.148681.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/core.20230210.132351.148681.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/javacore.20230210.132351.148681.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/javacore.20230210.132351.148681.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/Snap.20230210.132351.148681.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/Snap.20230210.132351.148681.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/jitdump.20230210.132351.148681.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'main' thread 0x0000000000095700
JVMDUMP010I JIT dump written to /home/jenkins/workspace/Grinder_iteration_4/aqa-tests/TKG/output_1676064226331/SecurityTests_0/jitdump.20230210.132351.148681.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".
-----------------------------------
SecurityTests_0_FAILED

FYI @jasonkatonica

pshipton commented 1 year ago

FYI there are a couple of optimizations you can use in a grinder now. Set TKG_ITERATIONS instead of ITERATIONS, and it should run a bit faster. You can also set EXIT_FAILURE in order to stop after the first failure if that is all you need, rather than running them all.

jasonkatonica commented 1 year ago

It seems like this fails every time when running Grinder against the label ci.role.test&&hw.arch.ppc64le&&sw.os.rhel.8&&rhel8le-rt1-2.fyre.ibm.com , this always runs the job on the rhel8le-rt1-2.fyre.ibm.com machine so there must be something specific going on with that env.

I attempted to run the test on that machine with my own debug SDK with: https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/30997/

This ended up failing as reported above but did not give me any debug messages which should be sent to stdout as I had hoped.

I also managed to update my RHEL system to the same level as above which is kernel 4.18.0-425.10.1.el8_7.ppc64le and version 8.7, yet I did not encounter any problems with my stand alone test so far:

# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.7 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"
# uname -a
Linux katonicabuildjdk11j9d1.fyre.ibm.com 4.18.0-425.10.1.el8_7.ppc64le #1 SMP Wed Dec 14 15:45:58 EST 2022 ppc64le ppc64le ppc64le GNU/Linux

I am still trying to figure out how to update the system to the real time kernel to see if that makes a difference.

pshipton commented 1 year ago

Does the stand alone test fail on rhel8le-rt1-2.fyre.ibm.com? You can reserve the machine and try it. You can also try running the test that does fail on your machine.

Does rhel8le-rt1-2.fyre.ibm.com use a real time kernel? You just mentioned you already have the same kernel version.

jasonkatonica commented 1 year ago

Hi @pshipton do you happen to have a link to the process to sign out rhel8le-rt1-2.fyre.ibm.com ? I can certainly give my stand alone test a try on the system once I am able to login.

My personal system has this kernel:

# uname -r
4.18.0-425.10.1.el8_7.ppc64le

Which indeed does match the value that is in my failing jobs when I run on rhel8le-rt1-2.fyre.ibm.com.

Given that this value matches what is failing in the Jenkins job, the real time kernel perhaps is not configured there, since Id expect a rt value in the kernel name.

pshipton commented 1 year ago

See this internal link

jasonkatonica commented 1 year ago

@pshipton Thanks for the link this allowed me to reserve that system and run a simple test case.

My simple test case which exercises AES/GCM cipher also crashes similar to the Jenkins job the first call to AES/GCM services in openssl. I used the default version of Java for the jenkins user which is:

[jenkins@rhel8le-rt1-2] java -version
openjdk version "17.0.5" 2022-10-18
IBM Semeru Runtime Open Edition 17.0.5.0 (build 17.0.5+8)
Eclipse OpenJ9 VM 17.0.5.0 (build openj9-0.35.0, JRE 17 Linux ppc64le-64-Bit Compressed References 20221018_307 (JIT enabled, AOT enabled)
OpenJ9   - e04a7f6c1
OMR      - 85a21674f
JCL      - 32d2c409a33 based on jdk-17.0.5+8)

My test code: AESGCMCrank.zip

The results:

[jenkins@rhel8le-rt1-2]  java AESGCMCrank
Iteration: 0 with length: 4332
Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
Handler1=00007FFF989AF270 Handler2=00007FFF988D87E0
R0=00007FFF745F7C60 R1=00007FFF990CCBE0 R2=00007FFF747B7F00 R3=00000000FFE16020
R4=00000000FFE13E00 R5=00000000000010E0 R6=00007FFF94360AA0 R7=00007FFF943608D8
R8=00007FFF94360918 R9=00000000000001B0 R10=0000000000000002 R11=00007FFF74654DC0
R12=0000000000004400 R13=00007FFF990D68E0 R14=00007FFF990CD0DC R15=000000000000000D
R16=0000000000000010 R17=00000000000010EC R18=0000000000000000 R19=0000000000000000
R20=00000000FFE16020 R21=00007FFF94360918 R22=00007FFF94360AA0 R23=0000000100000000
R24=00000000FFE13E00 R25=00000000FFE16020 R26=00000000000010E0 R27=0000000000000000
R28=0000000000000000 R29=0000000000000000 R30=000000000000010E R31=00007FFF943608D8
NIP=00007FFF7452122C MSR=800000000288F033 ORIG_GPR3=C0000000000027E0 CTR=00007FFF74527F60
LINK=00007FFF745F7C60 XER=0000000020000000 CCR=0000000048224444 SOFTE=0000000000000001
TRAP=0000000000000700 DAR=00007FFF94360018 dsisr=0000000042000000 RESULT=0000000000000000
FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR1 40550fdc80000000 (f: 2147483648.000000, d: 8.424783e+01)
FPR2 40085621e0000000 (f: 3758096384.000000, d: 3.042057e+00)
FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
FPR4 40130ed080000000 (f: 2147483648.000000, d: 4.764467e+00)
FPR5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
FPR7 3fbc5e53aa362eb4 (f: 2855677696.000000, d: 1.108143e-01)
FPR8 b7008733169a6723 (f: 379217696.000000, d: -9.264334e-44)
FPR9 3bc1b67d80000000 (f: 2147483648.000000, d: 7.501685e-21)
FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR11 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR13 00007fff9401d0d0 (f: 2483147008.000000, d: 6.953266e-310)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib64/libcrypto.so.3
Module_base_address=00007FFF742B0000
Target=2_90_20221018_307 (Linux 4.18.0-425.10.1.el8_7.ppc64le)
CPU=ppc64le (4 logical CPUs) (0x1dccb0000 RAM)
----------- Stack Backtrace -----------
 (0x00007FFF7452122C [libcrypto.so.3+0x27122c])
 (0x00007FFF745F7C60 [libcrypto.so.3+0x347c60])
 (0x00007FFF745F8180 [libcrypto.so.3+0x348180])
 (0x00007FFF74653D34 [libcrypto.so.3+0x3a3d34])
 (0x00007FFF74654DF4 [libcrypto.so.3+0x3a4df4])
EVP_EncryptUpdate+0xb0 (0x00007FFF744CB2B0 [libcrypto.so.3+0x21b2b0])
EVP_CipherUpdate+0x24 (0x00007FFF744CBF04 [libcrypto.so.3+0x21bf04])
Java_jdk_crypto_jniprovider_NativeCrypto_GCMEncrypt+0x280 (0x00007FFF747D4AA0 [libjncrypto.so+0x4aa0])
 (0x00007FFF98B61AF8 [libj9vm29.so+0x1f1af8])
ffi_call_int+0xd4 (0x00007FFF98B617C4 [libj9vm29.so+0x1f17c4])
bytecodeLoopCompressed+0x126dc (0x00007FFF98A3151C [libj9vm29.so+0xc151c])
 (0x00007FFF98AD1C98 [libj9vm29.so+0x161c98])
runCallInMethod+0x258 (0x00007FFF98989B18 [libj9vm29.so+0x19b18])
gpProtectedRunCallInMethod+0x54 (0x00007FFF989B3604 [libj9vm29.so+0x43604])
signalProtectAndRunGlue+0x28 (0x00007FFF98AE1818 [libj9vm29.so+0x171818])
omrsig_protect+0x3f4 (0x00007FFF988D9CB4 [libj9prt29.so+0x39cb4])
gpProtectAndRun+0xa8 (0x00007FFF98AE18E8 [libj9vm29.so+0x1718e8])
gpCheckCallin+0xc4 (0x00007FFF989B5CC4 [libj9vm29.so+0x45cc4])
callStaticVoidMethod+0x48 (0x00007FFF989B2C08 [libj9vm29.so+0x42c08])
JavaMain+0x11e4 (0x00007FFF993B7304 [libjli.so+0x7304])
ThreadJavaMain+0x18 (0x00007FFF993BD258 [libjli.so+0xd258])
start_thread+0xf8 (0x00007FFF99369718 [libpthread.so.0+0x9718])
clone+0x74 (0x00007FFF9924B498 [libc.so.6+0x13b498])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/02/13 08:28:34 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/core.20230213.082834.515043.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/core.20230213.082834.515043.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/javacore.20230213.082834.515043.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/javacore.20230213.082834.515043.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/Snap.20230213.082834.515043.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/Snap.20230213.082834.515043.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/jitdump.20230213.082834.515043.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'main' thread 0x0000000000095700
JVMDUMP010I JIT dump written to /home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/jitdump.20230213.082834.515043.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".

I did notice that the openssl version is the following by default on the system:

[jenkins@rhel8le-rt1-2]  openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021

So unless this is overwritten someplace in an environment variable when Jenkins runs it seems that the system openssl version is 1.1.1k. I will attempt to switch some of my other working systems ( ppc + rhel 8.7 ) to this version of openssl to see if it crashes in the same way.

jasonkatonica commented 1 year ago

Turning on debug traces ( -Djdk.nativeCryptoTrace=true and -Djava.security.auth.debug=all ) with this simple AESGCMCrank test indicates that 3.0.1 of openssl is really being loaded. The code is indeed loading version 3.0.1 when it crashes on the system rhel8le-rt1-2.fyre.ibm.com. It is NOT loading version 1.1.1k.

provider: NativePRNG.MIXED seedFile: /dev/random nextFile: /dev/urandom
Attempt to load OpenSSL /lib64/libcrypto.so.3
Using OpenSSL version: OpenSSL 3.0.1 14 Dec 2021 (/lib64)

In any case I attempted to make use of both 3.0.1 and 1.1.1k on another working ppc system and did not observe any crashes there yet.

jasonkatonica commented 1 year ago

I was able to get my AESGCMCrank test working on the system rhel8le-rt1-2.fyre.ibm.com by installing a version of the openssl 3.0.1 library that I built myself from source. I then configured the failing system rhel8le-rt1-2.fyre.ibm.com for my login as follows:

export LD_LIBRARY_PATH=/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/openssl/openssl-3.0.1/lib
export LDFLAGS="-L /home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/openssl/openssl-3.0.1/lib/ -Wl,-rpath,/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/openssl/openssl-3.0.1/lib/"
export PATH=/home/jenkins/jasonkatonica/issue_gcmcrash_72_16371/openssl/openssl-3.0.1/apps/:$PATH

After making this change to use a library that I built from source, I no longer received any crashes. I am not exactly sure why the 3.0.1 library present on the system at /lib64/libcrypto.so.3 is working differently and causing crashes. I did notice that the library came from this package on the failing system.:

rpm -qf /lib64/libcrypto.so.3
openssl3-libs-3.0.1-43.el8.1.ppc64le

I am not sure how this package was installed on the system.

jasonkatonica commented 1 year ago

I was able to recreate this issue on my system. I needed to install that openssl package from the EPL repository. I did that as follows:

To install the EPEL repository:

yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

To install openssl:

yum install openssl3

I then ran my test on the system and it crashed on the first call to the openssl library.

There is certainly an issue with this version of the EPL library.

jasonkatonica commented 1 year ago

I tried to create a minimum size test as possible in a C program that is able to replicate the problem. It does appear that the epel library on ppc has some sort of illegal instruction:

Test Code:

gcmEncrypt2.zip

To Compile:

 gcc gcmEncrypt2.c -o crashgcm -I /usr/include/openssl3 -ldl

Execution output against the epel library on ppc. Notice it fails at the bottom with Illegal instruction (core dumped):

# ./crashgcm /lib64/libcrypto.so.3
OpenSSL version: OpenSSL 3.0.1 14 Dec 2021
Symbol EVP_CIPHER_CTX_new found in library: /lib64/libcrypto.so.3
Symbol EVP_EncryptInit found in library: /lib64/libcrypto.so.3
Symbol EVP_EncryptUpdate found in library: /lib64/libcrypto.so.3
Symbol EVP_aes_256_gcm found in library: /lib64/libcrypto.so.3
Initialize Context.
Initialize Context Done.
Initialize done.
Encrypt.
Illegal instruction (core dumped)

Executing output against my own openssl 3.0.1 level of the code that I built on ppc. Notice that it works and completes encrypting:

# ./crashgcm /root/openssl/openssl-3.0.1/lib/libcrypto.so.3
OpenSSL version: OpenSSL 3.0.1 14 Dec 2021
Symbol EVP_CIPHER_CTX_new found in library: /root/openssl/openssl-3.0.1/lib/libcrypto.so.3
Symbol EVP_EncryptInit found in library: /root/openssl/openssl-3.0.1/lib/libcrypto.so.3
Symbol EVP_EncryptUpdate found in library: /root/openssl/openssl-3.0.1/lib/libcrypto.so.3
Symbol EVP_aes_256_gcm found in library: /root/openssl/openssl-3.0.1/lib/libcrypto.so.3
Initialize Context.
Initialize Context Done.
Initialize done.
Encrypt.
Encrypt done.

I should also note that this test works correctly against the elel library installed at /lib64/libcrypto.so.3 on a x86 machine that I have.

jasonkatonica commented 1 year ago

Wanted to also state the exact version of the failing EPEL library on PPC is this one:

# rpm -qf /lib64/libcrypto.so.3
openssl3-libs-3.0.1-43.el8.1.ppc64le

This version works on x86:

# rpm -qf /lib64/libcrypto.so.3
openssl3-libs-3.0.1-43.el8.1.x86_64
JasonFengJ9 commented 1 year ago

JDK17 ppc64le_linux(rhel8le-rt1-1)

java version "17.0.7-beta" 2023-04-18
IBM Semeru Runtime Certified Edition 17.0.7+5-202303232332 (build 17.0.7-beta+5-202303232332)
Eclipse OpenJ9 VM 17.0.7+5-202303232332 (build master-7e14589c2, JRE 17 Linux ppc64le-64-Bit Compressed References 20230323_366 (JIT enabled, AOT enabled)
OpenJ9   - 7e14589c2
OMR      - c766c4155
JCL      - a87ab94e85 based on jdk-17.0.7+5)

[2023-03-24T00:47:15.570Z] variation: Mode150
[2023-03-24T00:47:15.570Z] JVM_OPTIONS:  -XX:+UseCompressedOops 

[2023-03-24T00:47:37.279Z] STF 17:47:36.813 - Overall result: **FAILED**
[2023-03-24T00:47:37.279Z] -----------------------------------
[2023-03-24T00:47:37.279Z] TestJlmRemoteThreadAuth_0_FAILED

aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/3.LT1.stderr

Unhandled exception
Type=Illegal instruction vmState=0x00040000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=00000001
Handler1=00007FFF867B01C0 Handler2=00007FFF866D8B00
R0=00007FFEAFE3C530 R1=00007FFE57AEAB20 R2=00007FFEAFFF7F00 R3=00007FFF7FB7A748
R4=00007FFF7FB7A08D R5=00000000000001D0 R6=00007FFED008E4A0 R7=00007FFED008E2D8
R8=00007FFED008E318 R9=00000000000001B0 R10=0000000000000002 R11=00007FFEAFE9A030
R12=0000000000004400 R13=00007FFE57AF68E0 R14=00007FFE57AEB01C R15=0000000000000005
R16=0000000000000010 R17=00000000000001D8 R18=0000000000000005 R19=0000000000000000
R20=00007FFF7FB7A748 R21=00007FFED008E318 R22=00007FFED008E4A0 R23=0000000100000000
R24=00007FFF7FB7A08D R25=00007FFF7FB7A748 R26=00000000000001D0 R27=0000000000000000
R28=0000000000000000 R29=0000000000000000 R30=000000000000001D R31=00007FFED008E2D8
NIP=00007FFEAFD6452C MSR=800000000288F033 ORIG_GPR3=00007FFEAFE3C52C CTR=00007FFEAFD6B260
LINK=00007FFEAFE3C530 XER=0000000020000000 CCR=0000000048224444 SOFTE=0000000000000001
TRAP=0000000000000700 DAR=00007FFE55A50048 dsisr=0000000042000000 RESULT=0000000000000000
FPR0 000000000000000e (f: 14.000000, d: 6.916919e-323)
FPR1 4046d0e8c0000000 (f: 3221225472.000000, d: 4.563210e+01)
FPR2 3fefe64b9fc82b3a (f: 2680695552.000000, d: 9.968622e-01)
FPR3 3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
FPR4 3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
FPR5 bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
FPR6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR8 38206062e0000000 (f: 3758096384.000000, d: 2.406312e-38)
FPR9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR13 00007fff800e3b00 (f: 2148416256.000000, d: 6.953250e-310)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib64/libcrypto.so.3
Module_base_address=00007FFEAFAF0000
Target=2_90_20230323_366 (Linux 4.18.0-425.13.1.el8_7.ppc64le)
CPU=ppc64le (4 logical CPUs) (0x1dcca0000 RAM)
----------- Stack Backtrace -----------
 (0x00007FFEAFD6452C [libcrypto.so.3+0x27452c])
 (0x00007FFEAFE3C530 [libcrypto.so.3+0x34c530])
 (0x00007FFEAFE3CA50 [libcrypto.so.3+0x34ca50])
 (0x00007FFEAFE98F94 [libcrypto.so.3+0x3a8f94])
 (0x00007FFEAFE9A064 [libcrypto.so.3+0x3aa064])
EVP_EncryptUpdate+0xac (0x00007FFEAFD0DB0C [libcrypto.so.3+0x21db0c])
EVP_CipherUpdate+0x24 (0x00007FFEAFD0E754 [libcrypto.so.3+0x21e754])
Java_jdk_crypto_jniprovider_NativeCrypto_GCMEncrypt+0x280 (0x00007FFEB4834AE0 [libjncrypto.so+0x4ae0])
 (0x00007FFF86970938 [libj9vm29.so+0x200938])
ffi_call_int+0xd4 (0x00007FFF86970604 [libj9vm29.so+0x200604])
bytecodeLoopFull+0x13c28 (0x00007FFF86862CC8 [libj9vm29.so+0xf2cc8])
 (0x00007FFF868DD208 [libj9vm29.so+0x16d208])
runJavaThread+0x240 (0x00007FFF86786F30 [libj9vm29.so+0x16f30])
javaProtectedThreadProc+0x138 (0x00007FFF86823FC8 [libj9vm29.so+0xb3fc8])
omrsig_protect+0x3f4 (0x00007FFF866D9FD4 [libj9prt29.so+0x39fd4])
javaThreadProc+0x60 (0x00007FFF8681F8A0 [libj9vm29.so+0xaf8a0])
thread_wrapper+0x190 (0x00007FFF8666CBC0 [libj9thr29.so+0xcbc0])
start_thread+0xf8 (0x00007FFF87169718 [libpthread.so.0+0x9718])
clone+0x74 (0x00007FFF8704B498 [libc.so.6+0x13b498])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/03/23 18:04:12 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/core.20230323.180412.1280052.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/core.20230323.180412.1280052.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/javacore.20230323.180412.1280052.0002.txt' in response to an event
javacore file generated - /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/javacore.20230323.180412.1280052.0002.txt
core file generated - /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/core.20230323.180412.1280052.0001.dmp
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/javacore.20230323.180412.1280052.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/Snap.20230323.180412.1280052.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/Snap.20230323.180412.1280052.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/jitdump.20230323.180412.1280052.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'RMI TCP Connection(2)-9.30.182.222' thread 0x00007FFEA40BFA00
JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/jitdump.20230323.180412.1280052.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".
Snap file generated - /home/jenkins/workspace/Test_openjdk17_j9_sanity.system_ppc64le_linux_testList_2/aqa-tests/TKG/output_16796174769767/TestIBMJlmRemoteMemoryAuth_0/20230323-180352-TestIBMJlmRemoteMemoryAuth/results/Snap.20230323.180412.1280052.0003.trc
JasonFengJ9 commented 5 months ago

JDK17 ppc64le_linux(rhel8le-rtp-rt3-1)

[2024-04-09T20:20:34.694Z] variation: Mode650
[2024-04-09T20:20:34.694Z] JVM_OPTIONS:  -XX:-UseCompressedOops -Xverbosegclog 

[2024-04-09T20:23:42.931Z] TEST: sun/security/krb5/auto/HttpsCB.java

[2024-04-09T20:23:42.934Z] Unhandled exception
[2024-04-09T20:23:42.934Z] Type=Illegal instruction vmState=0x00040000

[2024-04-09T20:28:18.489Z] jdk_security4_1_FAILED

Infra issue