Openliberty JIT Compiler Crashes with segmenation error while trying to get a ConnectionRequestInfo from PoolManager

timmalich commented 3 weeks ago

Describe the bug
Our started open-liberty crashes after about 3 Minutes with the dump below. We use the latest full version from docker (library/open-liberty:full) which resolves into: Open Liberty 24.0.0.8

Investigation status:

The issues suddenly occurred last week. Of course we first assumed it's due to some changes in our code base. That seems not to be the case. We've gone back in time step by step for about two years, but the crash remained.
As workaround we temporary added jvm.options Parameter -Xnojit. This obviously slows down the performance significantly, but open-liberty was otherwise running just fine.
After that we did the same for the underlining open-liberty image. We figured out, that issue starts with version: Open Liberty 24.0.0.8 / Version Open Liberty 24.0.0.7 just runs fine.

Exception / Dump:

Unhandled exception
Type=Segmentation error vmState=0x000501ff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000076B4DFE37F50 Handler2=000076B4E40D1740 InaccessibleAddress=0000000099669966
RDI=0000000001F64278 RSI=0000000001F64278 RAX=0000000099669966 RBX=000000000220D500
RCX=0000000001F64700 RDX=0000000000000200 R8=0000000000000200 R9=000076B3F8650A00
R10=0000000001F64700 R11=0000000001F64700 R12=0000000000000156 R13=000076B4DF9C5AC0
R14=0000000001F6F040 R15=000000000220D500
RIP=000076B4DF6EFBF7 GS=0000 FS=0000 RSP=000076B4D3FF4C20
EFlags=0000000000010206 CS=0033 RBP=000076B4CC008050 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000099669966
xmm0=0000000000005800 (f: 22528.000000, d: 1.113031e-319)
xmm1=0000000001f64700 (f: 32917248.000000, d: 1.626328e-316)
xmm2=0000000000000156 (f: 342.000000, d: 1.689705e-321)
xmm3=6176616a00136f66 (f: 1273702.000000, d: 3.146501e+161)
xmm4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm5=0000000000000017 (f: 23.000000, d: 1.136351e-322)
xmm6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm7=000076b4b316e430 (f: 3004621824.000000, d: 6.448469e-310)
xmm8=ffffff0000000000 (f: 0.000000, d: -nan)
xmm9=6c6c6c686c6c6c63 (f: 1819044992.000000, d: 1.913745e+214)
xmm10=6c6c6c254557030a (f: 1163330304.000000, d: 1.913676e+214)
xmm11=0000015200000151 (f: 337.000000, d: 7.172346e-312)
xmm12=0000013d00000140 (f: 320.000000, d: 6.726727e-312)
xmm13=000001380000013f (f: 319.000000, d: 6.620627e-312)
xmm14=0000000008001800 (f: 134223872.000000, d: 6.631540e-316)
xmm15=000001420000013b (f: 315.000000, d: 6.832826e-312)
Module=/opt/java/openjdk/lib/amd64/default/libj9jit29.so
Module_base_address=000076B4DECF2000

Method_being_compiled=com/ibm/ejs/j2c/PoolManager.reserve(Ljavax/resource/spi/ManagedConnectionFactory;Ljavax/security/auth/Subject;Ljavax/resource/spi/ConnectionRequestInfo;Ljava/lang/Object;ZZII)Lcom/ibm/ws/j2c/MCWrapper;
Target=2_90_20240802_1004 (Linux 6.8.0-38-generic)
CPU=amd64 (8 logical CPUs) (0x5dca6e000 RAM)
----------- Stack Backtrace -----------
jitGetInterfaceMethodFromCP+0x17 (0x000076B4DF6EFBF7 [libj9jit29.so+0x9fdbf7])
_ZN11TR_J9VMBase26getResolvedInterfaceMethodEP14J9ConstantPoolP19TR_OpaqueClassBlocki+0x8b (0x000076B4DEF1021B [libj9jit29.so+0x21e21b])
_ZN19TR_ResolvedJ9Method26getResolvedInterfaceMethodEPN2TR11CompilationEP19TR_OpaqueClassBlocki+0x4c (0x000076B4DEF0735C [libj9jit29.so+0x21535c])
_ZN19CollectImplementors13visitSubclassEP22TR_PersistentClassInfo+0xd9 (0x000076B4DEEF3AD9 [libj9jit29.so+0x201ad9])
_ZN18TR_SubclassVisitor15visitSubclassesEP22TR_PersistentClassInfoRN10TR_CHTable12VisitTrackerE.localalias+0x5e (0x000076B4DEEF4F6E [libj9jit29.so+0x202f6e])
_ZN18TR_SubclassVisitor5visitEP19TR_OpaqueClassBlockb+0x118 (0x000076B4DEEF5148 [libj9jit29.so+0x203148])
_ZN15TR_ClassQueries25collectImplementorsCappedEP22TR_PersistentClassInfoPP17TR_ResolvedMethodiiS3_PN2TR11CompilationEb13TR_YesNoMaybe+0x146 (0x000076B4DEEF5336 [libj9jit29.so+0x203336])
_ZN20TR_PersistentCHTable21findSingleImplementerEP19TR_OpaqueClassBlockiP17TR_ResolvedMethodPN2TR11CompilationEb13TR_YesNoMaybeb+0x91 (0x000076B4DEF2CDD1 [libj9jit29.so+0x23add1])
_ZN22TR_J9InterfaceCallSite22findCallSiteTargetImplEP12TR_CallStackP14TR_InlinerBaseP19TR_OpaqueClassBlock+0x146 (0x000076B4DF0B36B6 [libj9jit29.so+0x3c16b6])
_ZN22TR_J9InterfaceCallSite18findCallSiteTargetEP12TR_CallStackP14TR_InlinerBase+0x3a (0x000076B4DF0B3DFA [libj9jit29.so+0x3c1dfa])
_ZN14TR_InlinerBase29getSymbolAndFindInlineTargetsEP12TR_CallStackP11TR_CallSiteb+0x46d (0x000076B4DF37251D [libj9jit29.so+0x68051d])
_ZN28TR_MultipleCallTargetInliner17inlineCallTargetsEPN2TR20ResolvedMethodSymbolEP12TR_CallStackP24TR_InnerPreexistenceInfo+0x4dd (0x000076B4DF09381D [libj9jit29.so+0x3a181d])
_ZN14TR_InlinerBase15performInliningEPN2TR20ResolvedMethodSymbolE+0xae (0x000076B4DF377DDE [libj9jit29.so+0x685dde])
_ZN10TR_Inliner7performEv+0x142 (0x000076B4DF08BAF2 [libj9jit29.so+0x399af2])
_ZN3OMR9Optimizer19performOptimizationEPK20OptimizationStrategyiii.localalias+0x855 (0x000076B4DF48E195 [libj9jit29.so+0x79c195])
_ZN3OMR9Optimizer8optimizeEv+0x1b3 (0x000076B4DF48FEB3 [libj9jit29.so+0x79deb3])
_ZN3OMR11Compilation7compileEv+0xa25 (0x000076B4DF27FE45 [libj9jit29.so+0x58de45])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x4bf (0x000076B4DEE697AF [libj9jit29.so+0x1777af])
_ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x381 (0x000076B4DEE6A7E1 [libj9jit29.so+0x1787e1])
omrsig_protect+0x239 (0x000076B4E40D23C9 [libj9prt29.so+0x2a3c9])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x372 (0x000076B4DEE68322 [libj9jit29.so+0x176322])
_ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x128 (0x000076B4DEE68668 [libj9jit29.so+0x176668])
_ZN2TR24CompilationInfoPerThread14processEntriesEv+0x35b (0x000076B4DEE6758B [libj9jit29.so+0x17558b])
_ZN2TR24CompilationInfoPerThread3runEv+0x42 (0x000076B4DEE678F2 [libj9jit29.so+0x1758f2])
_Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x82 (0x000076B4DEE679A2 [libj9jit29.so+0x1759a2])
omrsig_protect+0x239 (0x000076B4E40D23C9 [libj9prt29.so+0x2a3c9])
_Z21compilationThreadProcPv+0x17b (0x000076B4DEE67D6B [libj9jit29.so+0x175d6b])
thread_wrapper+0x163 (0x000076B4E409A3A3 [libj9thr29.so+0xb3a3])
(0x000076B4E45A6AC3 [libc.so.6+0x94ac3])
clone+0x44 (0x000076B4E4637A04 [libc.so.6+0x125a04])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2024/08/19 13:03:53 - please wait.

Steps to Reproduce
Currently not that easy, because the code base quite large and it's hard to reproduce a small reproducer.

Expected behavior
Open-liberty does not crash.

Diagnostic information:

OpenLiberty Version: 24.0.0.8
Affected feature(s) n/a
Java Version: IBM Semeru Runtime Open Edition (build 1.8.0_422-b05)

server.xml configuration:

<?xml version="1.0" encoding="UTF-8"?>
<server description="obfuscated1">

<!-- Enable features -->
<featureManager>
    <feature>jdbc-4.2</feature>
    <feature>jndi-1.0</feature>
    <feature>ejb-3.2</feature>
    <feature>concurrent-1.0</feature>
    <feature>transportSecurity-1.0</feature>
    <feature>beanValidation-2.0</feature>
    <feature>cdi-2.0</feature>
    <feature>el-3.0</feature>
    <feature>jaxrs-2.1</feature>
    <feature>appSecurity-2.0</feature>
    <feature>jpaContainer-2.2</feature>
    <feature>jsf-2.3</feature>
    <feature>jsonp-1.1</feature>
    <feature>jsonb-1.0</feature>
    <feature>jsp-2.3</feature>
    <feature>managedBeans-1.0</feature>
    <feature>servlet-4.0</feature>
    <feature>websocket-1.1</feature>
    <feature>appClientSupport-1.0</feature>
    <feature>batch-1.0</feature>
    <feature>j2eeManagement-1.1</feature>
    <feature>jacc-1.5</feature>
    <feature>jaspic-1.1</feature>
    <feature>javaMail-1.5</feature>
    <feature>jaxws-2.2</feature>
    <feature>jca-1.7</feature>
    <feature>jcaInboundSecurity-1.0</feature>
    <feature>jms-2.0</feature>
    <feature>wasJmsClient-2.0</feature>
    <feature>wasJmsSecurity-1.0</feature>
    <feature>wasJmsServer-1.0</feature>
    <feature>passwordUtilities-1.0</feature>
    <feature>mpMetrics-3.0</feature>
    <feature>openidConnectClient-1.0</feature>
    <feature>localConnector-1.0</feature>
    <feature>mpOpenAPI-2.0</feature>
    <feature>bells-1.0</feature>
</featureManager>

<logging hideMessage="SRVE9967W, HHH90000022, HHH90000012"/>

<bell libraryRef="hibernate"/>

<!--<include location="oidc_config.xml" />-->
<include location="ldap_config.xml" />

<!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
<httpEndpoint id="defaultHttpEndpoint"
              httpPort="9081"
              httpsPort="9443" />

<application id="obfuscated1" location="obfuscated1/obfuscated1.ear">
    <classloader commonLibraryRef="obfuscated1-library, hibernate">
        <privateLibrary>
            <folder dir="lib/logback/obfuscated1" />
        </privateLibrary>
    </classloader>
</application>

<application id="obfuscated2-listener" location="obfuscated1/obfuscated2-listener.ear" startAfter="obfuscated1">
    <classloader commonLibraryRef="obfuscated1-library">
        <privateLibrary>
            <folder dir="lib/logback/obfuscated2-listener" />
        </privateLibrary>
    </classloader>
</application>

<application id="obfuscated3-adapter" location="obfuscated1/obfuscated3-adapter.ear" startAfter="obfuscated1">
    <classloader commonLibraryRef="obfuscated1-library">
        <privateLibrary>
            <folder dir="lib/logback/obfuscated3-adapter" />
        </privateLibrary>
    </classloader>
</application>

<!-- Default SSL configuration enables trust for default certificates from the Java runtime --> 
<dataSource id="obfuscated1 Datasource"
            jndiName="jdbc/obfuscated1db"
            jdbcDriverRef="postgresql-driver"
            type="javax.sql.XADataSource"
            transactional="true">

    <properties serverName="obfuscated1-postgres"
                portNumber="64000"
                databaseName="obfuscated1db"
                user="obfuscated4"
                password="obfuscated5"/>
</dataSource>

<dataSource id="obfuscated3 Adapter Datasource"
            jndiName="jdbc/obfuscated3adapterdb"
            jdbcDriverRef="postgresql-driver"
            type="javax.sql.XADataSource"
            transactional="true">

    <properties serverName="obfuscated1-postgres"
                portNumber="64000"
                databaseName="obfuscated1db"
                user="obfuscated6"
                password="obfuscated7"/>
</dataSource>

<dataSource id="obfuscated1 Audit Datasource"
            jndiName="jdbc/obfuscated1auditdb"
            jdbcDriverRef="postgresql-driver"
            type="javax.sql.XADataSource"
            transactional="true">

    <properties serverName="obfuscated1-postgres"
                portNumber="64000"
                databaseName="obfuscated1db"
                user="obfuscated8"
                password="obfuscated9"/>
</dataSource>

<dataSource id="DefaultDataSource"
            jdbcDriverRef="derby-driver">
    <properties.derby.embedded createDatabase="create"
                               databaseName="${server.output.dir}/data/EJBTimerDB"/>
</dataSource>

<jdbcDriver id="derby-driver" libraryRef="derby-library"/>

<jdbcDriver id="postgresql-driver"
            libraryRef="postgresql-library"/>

<library id="postgresql-library">
    <fileset id="PostgresFileset" dir="lib/jdbc" includes="postgresql-*.jar"/>
</library>

<library id="derby-library">
    <fileset id="DerbyFileset" dir="lib/jdbc" includes="derby-*.jar"/>
</library>

<library id="obfuscated1-library">
    <folder dir="lib/config" />
    <fileset id="OtherFileset" dir="lib/other" includes="*.jar"/>
    <fileset id="LogFileset" dir="lib/log" includes="*.jar"/>
</library>

<library id="hibernate">
    <fileset id="hibernateFileset" dir="lib/hibernate" includes="*.jar"/>
</library>

<jndiURLEntry id="obfuscated3 WebService" jndiName="url/obfuscated3webservice" value="https://jcocker:start@obfuscated3.obfuscated10.obfuscated11.net" />

<managedScheduledExecutorService jndiName="wm/monitoringWM">
    <contextService>
        <jeeMetadataContext/>
        <securityContext/>
    </contextService>
</managedScheduledExecutorService>

<ssl id="defaultSSLConfig"
     trustStoreRef="obfuscated1TrustStore"
     sslProtocol="TLSv1.3,TLSv1.2" />

<keyStore id="emptyKeyStore" location="empty.jks" type="JKS" password="WebAS1" />
<keyStore id="obfuscated1TrustStore" location="obfuscated1_trust.p12" type="PKCS12" password="obfuscated11" />

<mpMetrics authentication="false"/>

<mailSession mailSessionID="obfuscated1MailSession"
             jndiName="mail/obfuscated1mail"
             description="obfuscated1 Mail Session"
             transportProtocol="smtp"
             host="obfuscated1-smtp"
             user="obfuscated14"
             password="obfuscated15"
             from="obfuscated1-ops-obfuscated13@obfuscated12.com" >
    <property name="mail.smtp.port" value="1025" />
</mailSession>

<distributedMap id="obfuscated16"
                jndiName="dmap/services/exports/roletranslations/cache"
                memorySizeInEntries="100" />
</server>

Additional context
Add any other context about the problem knödeln here.

DevBoxFanBoy commented 3 weeks ago

What means "knödeln"?

timmalich commented 3 weeks ago

DO NOT DOWNLOAD EXTRACT AND RUN THE STUPID fix.zip FROM THE MALICIOUS COMMENTS ABOVE! Hope they will be deleted soon. All Users have been reported already. I downloaded and encrypted the file with the given password on spare PC. It contains a x86_64-w64-ranlib.exe and a msvcp140.dll

dgeissl commented 2 weeks ago

We could narrow down very similar issues to an official image openliberty/open-liberty:24.0.0.6-full-java8-openj9-ubi update on 21st of August 7:40PM CEST. With that the OpenJ9 version changed in our case from 1.8.0_412-b08 to 1.8.0_422-b05 (see https://github.com/eclipse-openj9/openj9/milestone/53)

We have tried many other compositions since:

openliberty/open-liberty/4.0.0.8-full-java11-openj9-ubi
openliberty/open-liberty:24.0.0.8-full-java17-openj9-ubi
openliberty/open-liberty:24.0.0.6-full-java17-openj9-ubi
open-liberty:24.0.0.6-full-java17-openj9 (ubuntu community image)

to no success. For now, we hope https://github.com/eclipse-openj9/openj9/issues/20012 will be released soon, so we can double check if the issue is resolved for us.

evia-blank commented 2 weeks ago

The problem seems to affect all images that have been updated recently. 24.0.0.7 has not been updated, so fortunately it still works

OpenLiberty / open-liberty

Openliberty JIT Compiler Crashes with segmenation error while trying to get a ConnectionRequestInfo from PoolManager #29486