search

helidon-io / helidon

Java libraries for writing microservices
https://helidon.io
Apache License 2.0
3.52k stars 564 forks source link

Platform pinning observed, potentially, indirectly caused by Helidon MP framework code (JTA) #8769

Closed jvissers closed 4 months ago

jvissers commented 5 months ago

Environment Details

Problem Description

We are seeing "platform pinning", for various reasons. One of which could be attribute to through a Helidon JTA specific code path. @danielkec asked me to raise an issue for this to investigate further.

I started observing this after applying this parameter -Djdk.tracePinnedThreads I now know that I should probably use JFR events for pinning detection. By that as it may, with the above, one of the observed pinned threads showed this stack:


Thread[#119,ForkJoinPool-1-worker-1,5,CarrierThreads]
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:185)
    java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
    java.base/java.lang.VirtualThread.park(VirtualThread.java:592)
    java.base/java.lang.System$2.parkVirtualThread(System.java:2639)
    java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
    java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:369)
    java.base/sun.nio.ch.Poller.pollIndirect(Poller.java:139)
    java.base/sun.nio.ch.Poller.poll(Poller.java:102)
    java.base/sun.nio.ch.Poller.poll(Poller.java:87)
    java.base/sun.nio.ch.SelChImpl.park(SelChImpl.java:88)
    java.base/sun.nio.ch.SelChImpl.park(SelChImpl.java:116)
    java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:428)
    oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:415)
    oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:1127)
    oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:266)
    oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:198)
    oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:140)
    oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:113)
    oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:97)
    oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:833)
    oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:486)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:621)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:298)
    oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:93)
    oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:2039)
    oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2651)
    oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2692)
    oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1ConnectionProxy$2oracle$1jdbc$1internal$1OracleConnection$$$Proxy.commit(Unknown Source)
    io.helidon.integrations.jdbc.DelegatingConnection.commit(DelegatingConnection.java:113)
    io.helidon.integrations.jta.jdbc.LocalXAResource$Association.runAndReset(LocalXAResource.java:925)
    io.helidon.integrations.jta.jdbc.LocalXAResource$Association.commitAndReset(LocalXAResource.java:908)
    io.helidon.integrations.jta.jdbc.LocalXAResource.commitAndReset(LocalXAResource.java:573)
    io.helidon.integrations.jta.jdbc.LocalXAResource.lambda$commit$0(LocalXAResource.java:259)
    io.helidon.integrations.jta.jdbc.LocalXAResource.remap(LocalXAResource.java:487)
    io.helidon.integrations.jta.jdbc.LocalXAResource.lambda$computeAssociation$1(LocalXAResource.java:379)
    java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940) <== monitors:1
    io.helidon.integrations.jta.jdbc.LocalXAResource.computeAssociation(LocalXAResource.java:379)
    io.helidon.integrations.jta.jdbc.LocalXAResource.commit(LocalXAResource.java:255)
    com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:678)
    com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2457)
    com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1520)
    com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:74)
    com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:138)
    com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1271)
    com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:104)
    com.arjuna.ats.jta.cdi.DelegatingTransactionManager.commit(DelegatingTransactionManager.java:110)
    com.arjuna.ats.jta.cdi.NarayanaTransactionManager.commit(NarayanaTransactionManager.java:319)
    com.arjuna.ats.jta.cdi.NarayanaTransactionManager$Proxy$_$$_WeldClientProxy.commit(Unknown Source)
    com.arjuna.ats.jta.cdi.TransactionHandler.endTransaction(TransactionHandler.java:67)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:192)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:169)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:37)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:74)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:31)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
    com.oracle.decs.designer.workbench.admin.ConfigExportService$Proxy$_$$_WeldSubclass.exportWorkspace(Unknown Source)
    com.oracle.decs.designer.workbench.admin.ConfigExportService$Proxy$_$$_WeldClientProxy.exportWorkspace(Unknown Source)
    com.oracle.decs.designer.workbench.admin.WorkspaceResource.exportWorkspace(WorkspaceResource.java:72)
    com.oracle.decs.designer.workbench.admin.WorkspaceResource$Proxy$_$$_WeldSubclass.exportWorkspace$$super(Unknown Source)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51)
    org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78)
    com.oracle.decs.support.rest.AroundResourceInterceptor.execOperation(AroundResourceInterceptor.java:40)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
    com.oracle.decs.designer.workbench.admin.WorkspaceResource$Proxy$_$$_WeldSubclass.exportWorkspace(Unknown Source)
    com.oracle.decs.designer.workbench.admin.WorkspaceResource$Proxy$_$$_WeldClientProxy.exportWorkspace(Unknown Source)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
    org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
    org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
    org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
    org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
    org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
    org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
    org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
    org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
    org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
    io.helidon.microprofile.server.JaxRsService.doHandle(JaxRsService.java:228)
    io.helidon.microprofile.server.JaxRsService.lambda$handle$2(JaxRsService.java:185)
    io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
    io.helidon.microprofile.server.JaxRsService.handle(JaxRsService.java:185)
    io.helidon.webserver.http.HttpRoutingImpl$RoutingExecutor.doRoute(HttpRoutingImpl.java:165)
    io.helidon.webserver.http.HttpRoutingImpl$RoutingExecutor.call(HttpRoutingImpl.java:124)
    io.helidon.webserver.http.HttpRoutingImpl$RoutingExecutor.call(HttpRoutingImpl.java:102)
    io.helidon.webserver.http.ErrorHandlers.runWithErrorHandling(ErrorHandlers.java:75)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:121)
    io.helidon.webserver.observe.metrics.MetricsFeature.lambda$configureVendorMetrics$2(MetricsFeature.java:90)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:119)
    io.helidon.webserver.security.SecurityContextFilter.filter(SecurityContextFilter.java:88)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:119)
    io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
    io.helidon.webserver.context.ContextRoutingFeature.filter(ContextRoutingFeature.java:50)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:119)
    io.helidon.webserver.http.Filters.executeFilters(Filters.java:87)
    io.helidon.webserver.http.Filters.lambda$filter$0(Filters.java:83)
    io.helidon.webserver.http.ErrorHandlers.runWithErrorHandling(ErrorHandlers.java:75)
    io.helidon.webserver.http.Filters.filter(Filters.java:83)
    io.helidon.webserver.http.HttpRoutingImpl.route(HttpRoutingImpl.java:73)
    io.helidon.webserver.http1.Http1Connection.route(Http1Connection.java:357)
    io.helidon.webserver.http1.Http1Connection.handle(Http1Connection.java:194)
    io.helidon.webserver.ConnectionHandler.run(ConnectionHandler.java:165)
    io.helidon.common.task.InterruptableTask.call(InterruptableTask.java:47)
    io.helidon.webserver.ThreadPerTaskExecutor$ThreadBoundFuture.run(ThreadPerTaskExecutor.java:239)
    java.base/java.lang.VirtualThread.run(VirtualThread.java:311)
Thread[#119,ForkJoinPool-1-worker-1,5,CarrierThreads]
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:185)
    java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
    java.base/java.lang.VirtualThread.park(VirtualThread.java:592)
    java.base/java.lang.System$2.parkVirtualThread(System.java:2639)
    java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
    java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:369)
    java.base/sun.nio.ch.Poller.pollIndirect(Poller.java:139)
    java.base/sun.nio.ch.Poller.poll(Poller.java:102)
    java.base/sun.nio.ch.Poller.poll(Poller.java:87)
    java.base/sun.nio.ch.SelChImpl.park(SelChImpl.java:88)
    java.base/sun.nio.ch.SelChImpl.park(SelChImpl.java:116)
    java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:428)
    oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:415)
    oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:1127)
    oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:266)
    oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:198)
    oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:140)
    oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:113)
    oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:97)
    oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:833)
    oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:486)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:621)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:298)
    oracle.jdbc.driver.T4C7Ocommoncall.doOCOMMIT(T4C7Ocommoncall.java:93)
    oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:2039)
    oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2651)
    oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:2692)
    oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1ConnectionProxy$2oracle$1jdbc$1internal$1OracleConnection$$$Proxy.commit(Unknown Source)
    io.helidon.integrations.jdbc.DelegatingConnection.commit(DelegatingConnection.java:113)
    io.helidon.integrations.jta.jdbc.LocalXAResource$Association.runAndReset(LocalXAResource.java:925)
    io.helidon.integrations.jta.jdbc.LocalXAResource$Association.commitAndReset(LocalXAResource.java:908)
    io.helidon.integrations.jta.jdbc.LocalXAResource.commitAndReset(LocalXAResource.java:573)
    io.helidon.integrations.jta.jdbc.LocalXAResource.lambda$commit$0(LocalXAResource.java:259)
    io.helidon.integrations.jta.jdbc.LocalXAResource.remap(LocalXAResource.java:487)
    io.helidon.integrations.jta.jdbc.LocalXAResource.lambda$computeAssociation$1(LocalXAResource.java:379)
    java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940) <== monitors:1
    io.helidon.integrations.jta.jdbc.LocalXAResource.computeAssociation(LocalXAResource.java:379)
    io.helidon.integrations.jta.jdbc.LocalXAResource.commit(LocalXAResource.java:255)
    com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:678)
    com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2457)
    com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1520)
    com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:74)
    com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:138)
    com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1271)
    com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:104)
    com.arjuna.ats.jta.cdi.DelegatingTransactionManager.commit(DelegatingTransactionManager.java:110)
    com.arjuna.ats.jta.cdi.NarayanaTransactionManager.commit(NarayanaTransactionManager.java:319)
    com.arjuna.ats.jta.cdi.NarayanaTransactionManager$Proxy$_$$_WeldClientProxy.commit(Unknown Source)
    com.arjuna.ats.jta.cdi.TransactionHandler.endTransaction(TransactionHandler.java:67)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:192)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:169)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:37)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:74)
    com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:31)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
    com.oracle.decs.designer.workbench.admin.ConfigExportService$Proxy$_$$_WeldSubclass.get(Unknown Source)
    com.oracle.decs.designer.workbench.admin.ConfigExportService$Proxy$_$$_WeldClientProxy.get(Unknown Source)
    com.oracle.decs.support.rest.AbstractBaseResource.get(AbstractBaseResource.java:51)
    com.oracle.decs.designer.workbench.admin.ConfigExportResource$Proxy$_$$_WeldSubclass.get(Unknown Source)
    com.oracle.decs.designer.workbench.admin.ConfigExportResource.getConfigExportById(ConfigExportResource.java:46)
    com.oracle.decs.designer.workbench.admin.ConfigExportResource$Proxy$_$$_WeldSubclass.getConfigExportById$$super(Unknown Source)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51)
    org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78)
    com.oracle.decs.support.rest.AroundResourceInterceptor.execOperation(AroundResourceInterceptor.java:40)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
    org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
    org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
    com.oracle.decs.designer.workbench.admin.ConfigExportResource$Proxy$_$$_WeldSubclass.getConfigExportById(Unknown Source)
    com.oracle.decs.designer.workbench.admin.ConfigExportResource$Proxy$_$$_WeldClientProxy.getConfigExportById(Unknown Source)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
    org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
    org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
    org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
    org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
    org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
    org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
    org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
    org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
    org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
    io.helidon.microprofile.server.JaxRsService.doHandle(JaxRsService.java:228)
    io.helidon.microprofile.server.JaxRsService.lambda$handle$2(JaxRsService.java:185)
    io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
    io.helidon.microprofile.server.JaxRsService.handle(JaxRsService.java:185)
    io.helidon.webserver.http.HttpRoutingImpl$RoutingExecutor.doRoute(HttpRoutingImpl.java:165)
    io.helidon.webserver.http.HttpRoutingImpl$RoutingExecutor.call(HttpRoutingImpl.java:124)
    io.helidon.webserver.http.HttpRoutingImpl$RoutingExecutor.call(HttpRoutingImpl.java:102)
    io.helidon.webserver.http.ErrorHandlers.runWithErrorHandling(ErrorHandlers.java:75)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:121)
    io.helidon.webserver.observe.metrics.MetricsFeature.lambda$configureVendorMetrics$2(MetricsFeature.java:90)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:119)
    io.helidon.webserver.security.SecurityContextFilter.filter(SecurityContextFilter.java:88)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:119)
    io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
    io.helidon.webserver.context.ContextRoutingFeature.filter(ContextRoutingFeature.java:50)
    io.helidon.webserver.http.Filters$FilterChainImpl.proceed(Filters.java:119)
    io.helidon.webserver.http.Filters.executeFilters(Filters.java:87)
    io.helidon.webserver.http.Filters.lambda$filter$0(Filters.java:83)
    io.helidon.webserver.http.ErrorHandlers.runWithErrorHandling(ErrorHandlers.java:75)
    io.helidon.webserver.http.Filters.filter(Filters.java:83)
    io.helidon.webserver.http.HttpRoutingImpl.route(HttpRoutingImpl.java:73)
    io.helidon.webserver.http1.Http1Connection.route(Http1Connection.java:357)
    io.helidon.webserver.http1.Http1Connection.handle(Http1Connection.java:194)
    io.helidon.webserver.ConnectionHandler.run(ConnectionHandler.java:165)
    io.helidon.common.task.InterruptableTask.call(InterruptableTask.java:47)
    io.helidon.webserver.ThreadPerTaskExecutor$ThreadBoundFuture.run(ThreadPerTaskExecutor.java:239)
    java.base/java.lang.VirtualThread.run(VirtualThread.java:311)
`
ljnelson commented 5 months ago

Culprit: https://github.com/helidon-io/helidon/blob/4.0.8/integrations/jta/jdbc/src/main/java/io/helidon/integrations/jta/jdbc/LocalXAResource.java#L379

ljnelson commented 5 months ago

Ultimate culprit: https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java#L1931