Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0

[c-koell]

Java -version output

openjdk version "21.0.3" 2024-04-16 LTS IBM Semeru Runtime Open Edition 21.0.3.0 (build 21.0.3+9-LTS) Eclipse OpenJ9 VM 21.0.3.0 (build openj9-0.44.0, JRE 21 Windows Server 2022 amd64-64-Bit Compressed References 20240416_179 (JIT enabled, AOT enabled) OpenJ9 - b0699311c7 OMR - 254af5a04 JCL - ac8f341bc20 based on jdk-21.0.3+9)

Summary of problem

We have a openliberty application running on ibm-semeru-open-jdk_x64_windows_21.0.3_09_openj9-0.44.0. Today we had jvm crashes with following error in console.log

Unhandled exception
Type=Segmentation error vmState=0x00000000
Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFFD85D42F0 ContextFlags=0010005f
Handler1=00007FFFD856EAA0 Handler2=00007FFFD827AC50 InaccessibleReadAddress=0000000000000003
RDI=0000000000000000 RSI=000000000000000B RAX=0000000000000005 RBX=000001DA0C6058D8
RCX=0000000000000002 RDX=0000000000000000 R8=0000000000000000 R9=0000000000000000
R10=000001DA0C6058D8 R11=00000000000000EB R12=0000000000000008 R13=000001DA0987B9F0
R14=000001DA0C6058D0 R15=00000077E1C7F150
RIP=00007FFFD85D42F0 RSP=00000077E1C7EF28 RBP=0000000000000008 EFLAGS=0000000000010246
FS=0053 ES=002B DS=002B
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM2 000001da7b4972b4 (f: 2068411008.000000, d: 1.006848e-311)
XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 00000cb741000000 (f: 1090519040.000000, d: 6.907635e-311)
XMM9 000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=C:\openjdk\bin\default\j9vm29.dll
Module_base_address=00007FFFD84A0000 Offset_in_DLL=00000000001342f0
Target=2_90_20240416_179 (Windows Server 2022 10.0 build 20348)
CPU=amd64 (4 logical CPUs) (0x5fff0a000 RAM)

JavaVMInitArgs.nOptions=69:
    -Xoptionsfile=C:\openjdk\lib\options.default
    -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/PrimitiveHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple
    -XX:+ensurehashed:java/lang/Class,java/lang/Thread
    -Xjcl:jclse29
    -Dcom.ibm.oti.vm.bootstrap.library.path=C:\openjdk\bin\default;C:\openjdk\bin
    -Dsun.boot.library.path=C:\openjdk\bin\default;C:\openjdk\bin
    -Djava.library.path=C:\openjdk\bin\default;C:\openjdk\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ElakWebServerP\AppData\Local\Microsoft\WindowsApps;.
    -Djava.home=C:\openjdk

Generated system dump: C:\Users\ElakWebServerP\j9.dmp
----------- Stack Backtrace -----------
J9_GetInterface+0x57ea0 (0x00007FFFD85D42F0 [j9vm29+0x1342f0])
(0x00007FFFD8569F81 [j9vm29+0xc9f81])
JVM_GetExtendedNPEMessage+0x177 (0x00007FFFDB5B8947 [jvm+0x18947])
(0x00007FFFC6BD04A8)
(0x00000007FB82FC80)
(0x00000007FB82E618)
(0x00000007FB82FCA0)
(0x00007FFFC5BE5550)
(0x0000000000000003)
(0x000000000098BD00)
(0x00000007FB6D38E8)
(0x000000007DB0A948)
(0x00000007FB7D8A50)
(0x0000100200000003)
(0x000000000098BD00)
(0x00000077E1C7F430)
(0x00000007FB82E618)
(0x00000007FB82FC80)

Now we are searching for the reason ... We switched back to to ibm-semeru-open-jdk_x64_windows_21.0.2_13_openj9-0.43.0. I'm not sure the failure comes from the latest jdk but we will see. At the moment the application runs without any crashes :-)

We had the crashes on windows and also on Linux. I will attach the dump file from a windows server.

Diagnostic files

I will attach a j9.dmp file ...

[c-koell]

You can find the dump under following cloud storage https://tbox.tirol.gv.at/index.php/s/GPrM3xrF98pTDWX with "jEzmabtqTR" as password

[pshipton]

@tajila @JasonFengJ9 fyi

[pshipton]

Perhaps related to https://github.com/eclipse-openj9/openj9/issues/19163 ?

[pshipton]

Here is a preliminary Windows jdk21 build of 0.46 you can try, in case it's fixed. https://ibm.box.com/s/d9jcgx4ggb46mycq2duzvz5hw497kzka

[pshipton]

and xLinux as well https://ibm.box.com/s/cgxoi65pjrustf8f8wi6qyhukg0fqd6b

[pshipton]

There is some confusion with the title and description, but the core is from openj9-0.44.0

JRE 21 Windows Server 2022 amd64-64-Bit Compressed References 20240416_179 (JIT enabled, AOT enabled)
OpenJ9   - b0699311c7
OMR      - 254af5a04
JCL      - ac8f341bc20 based on jdk-21.0.3+9

Also I zipped the 3GB file and transferred it to Box. https://ibm.box.com/s/u56fke31u817a943x2rvsmpx63ocvg7m

[c-koell]

There is some confusion with the title and description, but the core is from openj9-0.44.0

JRE 21 Windows Server 2022 amd64-64-Bit Compressed References 20240416_179 (JIT enabled, AOT enabled)
OpenJ9   - b0699311c7
OMR      - 254af5a04
JCL      - ac8f341bc20 based on jdk-21.0.3+9

Also I zipped the 3GB file and transferred it to Box. https://ibm.box.com/s/u56fke31u817a943x2rvsmpx63ocvg7m

Sorry for the confusion. I have changed the title and description.

[JasonFengJ9]

@c-koell Is this an intermittent failure? Do you happen to have a standalone test case?

[c-koell]

Yes it is an intermittent failure.

[c-koell]

FYI .. as a fast fix we have tried to set the flag -XX:-ShowCodeDetailsInExceptionMessages as mentioned in https://github.com/eclipse-openj9/openj9/issues/19163

Since that we havn't seen any crashes.

[c-koell]

@JasonFengJ9 dou you think our problem is the same as https://github.com/eclipse-openj9/openj9/issues/19163

[JasonFengJ9]

dou you think our problem is the same as https://github.com/eclipse-openj9/openj9/issues/19163 Here is a preliminary Windows jdk21 build of 0.46 you can try, in case it's fixed. https://ibm.box.com/s/d9jcgx4ggb46mycq2duzvz5hw497kzka

@c-koell did you try the preliminary 0.46 build?

[c-koell]

Hi @JasonFengJ9. Unfortionally not. We have set the flag -XX:-ShowCodeDetailsInExceptionMessages and sinse that we had no crashes. The application where the crash has occured is a key application. We where happy since that it runs now.

When the official release is out we have planned to test it with that application.

[JasonFengJ9]

When the official release is out we have planned to test it with that application.

@c-koell Thanks for the update. In the meantime, is there any standalone test case we can try?

[c-koell]

@JasonFengJ9 unfortunately not. We have no idea why this happens only in our key application :-) We have a lot of other applications running on the latest release without any problems.

[JasonFengJ9]

@c-koell could you re-try with OpenJ9 latest 0.46 release https://github.com/ibmruntimes/semeru21-binaries/releases?

[c-koell]

Yes of course! But it will take some time because I'm on vacation