search

eclipse-ee4j / jersey

Eclipse Jersey Project - Read our Wiki:
https://github.com/eclipse-ee4j/jersey/wiki
Other
692 stars 355 forks source link

IllegalStateException: ServiceLocatorImpl(__HK2_Generated_0,1,1151450492) has been shut down #3889

Open dlaprade opened 6 years ago

dlaprade commented 6 years ago

Issue:

Noticed in logs during high load a stack trace that referred to the class ServiceLocatorImpl(HK2). Not sure what is causing the issue to occur. Any assistance is appreciated.

Software:

Received an error in our logs right before containers were restarted: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_0,1,1151450492) has been shut down at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2371) at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandleImpl(ServiceLocatorImpl.java:632) at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandle(ServiceLocatorImpl.java:625) at org.jvnet.hk2.internal.ServiceLocatorImpl.getServiceHandle(ServiceLocatorImpl.java:643) at org.jvnet.hk2.internal.FactoryCreator.getFactoryHandle(FactoryCreator.java:103) at org.jvnet.hk2.internal.FactoryCreator.dispose(FactoryCreator.java:173) at org.jvnet.hk2.internal.SystemDescriptor.dispose(SystemDescriptor.java:542) at org.glassfish.jersey.process.internal.RequestScope$Instance.remove(RequestScope.java:532) at org.glassfish.jersey.process.internal.RequestScope$Instance.release(RequestScope.java:549) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

jansupol commented 6 years ago

Jersey 2.26 has the injection reworked very significantly. It still uses HK2, so it is possible your application can see similar logs, but udated HK2 version. I'd recommend updating Jersey and see whether the issue persists

jcalcote commented 4 years ago

I'm seeing this error on jersey 2.26:

2020-10-21 18:54:27.841 +0000,6891989020377 {} ERROR c.p.h.r.ReplicationService [hmdb-replicator-2] replication request for site=1bdee95a-5905-4914-8552-e0f0d7ad1e22, osv=268435472 failed with exception javax.ws.rs.ProcessingException: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_5,5,337374537) has been shut down
        at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:231)
        at org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:189)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312)
        at org.glassfish.jersey.client.ClientRuntime.lambda$createRunnableForAsyncProcessing$4(ClientRuntime.java:163)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_5,5,337374537) has been shut down
        at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2393)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:746)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:718)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:174)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54)
        at org.glassfish.jersey.client.ClientRuntime.lambda$new$0(ClientRuntime.java:115)
        at org.glassfish.jersey.client.RequestProcessingInitializationStage.apply(RequestProcessingInitializationStage.java:102)
        at org.glassfish.jersey.client.RequestProcessingInitializationStage.apply(RequestProcessingInitializationStage.java:66)
        at org.glassfish.jersey.process.internal.Stages$LinkedStage.apply(Stages.java:308)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:171)
        at org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:167)
        ... 14 more
, retrying...

This log line is, of course, mostly about our service, but the exception is the same as the OP's.

Question @jansupol : Is it possible that I'm somehow manually shutting down the ServiceLocatorImpl? That is, could I be doing something that would cause this?

jcalcote commented 4 years ago

I set a breakpoint in ServiceLocatorImpl.shutdown(). I hit the breakpoint in this call stack:

image

This tells me that I'm not allowed to close my clients because if I have only one client open and I close it, jersey terminates the client runtime and service locator in the process. Is this how it should work @jansupol ?

jcalcote commented 4 years ago

@jansupol please ignore my previous comments. I was using an async client and literally closing it immediately after starting the request. :/

sdyura commented 3 months ago

i see this error at the end of every run of a @HelidonTest

20:08:26.319 [main] DEBUG io.helidon.microprofile.server.JaxRsService - Exception during shutdown of Jersey
 java.lang.IllegalStateException: ServiceLocatorImpl(__HK2_Generated_0,0,488646216) has been shut down
    at org.jvnet.hk2.internal.ServiceLocatorImpl.checkState(ServiceLocatorImpl.java:2399) ~[hk2-locator-3.0.6.jar:?]
    at org.jvnet.hk2.internal.ServiceLocatorImpl.preDestroy(ServiceLocatorImpl.java:1060) ~[hk2-locator-3.0.6.jar:?]
    at org.jvnet.hk2.internal.ServiceLocatorImpl.preDestroy(ServiceLocatorImpl.java:1052) ~[hk2-locator-3.0.6.jar:?]
    at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.preDestroy(AbstractHk2InjectionManager.java:175) ~[jersey-hk2-3.1.7.jar:?]
    at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.preDestroy(ImmediateHk2InjectionManager.java:30) ~[jersey-hk2-3.1.7.jar:?]
    at io.helidon.microprofile.server.HelidonHK2InjectionManagerFactory$HelidonInjectionManager.preDestroy(HelidonHK2InjectionManagerFactory.java:287) ~[helidon-microprofile-server-4.0.10.jar:4.0.10]
    at org.glassfish.jersey.model.internal.ManagedObjectsFinalizer.preDestroy(ManagedObjectsFinalizer.java:70) ~[jersey-common-3.1.7.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.onShutdown(ApplicationHandler.java:747) ~[jersey-server-3.1.7.jar:?]
    at io.helidon.microprofile.server.JaxRsService.afterStop(JaxRsService.java:149) ~[helidon-microprofile-server-4.0.10.jar:4.0.10]
    at io.helidon.webserver.http.ServiceRoute.afterStop(ServiceRoute.java:52) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
    at io.helidon.webserver.http.ServiceRoute.afterStop(ServiceRoute.java:53) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at io.helidon.webserver.http.HttpRoutingImpl.afterStop(HttpRoutingImpl.java:86) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at io.helidon.webserver.RouterImpl.afterStop(RouterImpl.java:54) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at io.helidon.webserver.ServerListener.stop(ServerListener.java:247) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at io.helidon.webserver.LoomServer.stopIt(LoomServer.java:194) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at io.helidon.webserver.LoomServer.stop(LoomServer.java:146) ~[helidon-webserver-4.0.10.jar:4.0.10]
    at io.helidon.microprofile.server.ServerCdiExtension.doStop(ServerCdiExtension.java:637) ~[helidon-microprofile-server-4.0.10.jar:4.0.10]
    at io.helidon.microprofile.server.ServerCdiExtension.stopServer(ServerCdiExtension.java:620) ~[helidon-microprofile-server-4.0.10.jar:4.0.10]
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
    at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:85) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.injection.MethodInvocationStrategy$SimpleMethodInvocationStrategy.invoke(MethodInvocationStrategy.java:168) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserverMethodImpl.java:126) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at jakarta.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:142) ~[jakarta.enterprise.cdi-api-4.0.1.jar:?]
    at org.jboss.weld.util.Observers.notify(Observers.java:166) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at org.jboss.weld.event.EventImpl.fire(EventImpl.java:96) ~[weld-core-impl-4.0.10.jar:4.0.10]
    at io.helidon.microprofile.cdi.HelidonContainerImpl$HelidonCdi.close(HelidonContainerImpl.java:354) ~[helidon-microprofile-cdi-4.0.10.jar:4.0.10]
    at io.helidon.microprofile.testing.junit5.HelidonJunitExtension.stopContainer(HelidonJunitExtension.java:376) ~[helidon-microprofile-testing-junit5-4.0.10.jar:4.0.10]
    at io.helidon.microprofile.testing.junit5.HelidonJunitExtension.afterAll(HelidonJunitExtension.java:383) ~[helidon-microprofile-testing-junit5-4.0.10.jar:4.0.10]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeAfterAllCallbacks$18(ClassBasedTestDescriptor.java:462) ~[junit-jupiter-engine-5.10.2.jar:5.10.2]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeAfterAllCallbacks$19(ClassBasedTestDescriptor.java:462) ~[junit-jupiter-engine-5.10.2.jar:5.10.2]
    at org.junit.platform.commons.util.CollectionUtils.forEachInReverseOrder(CollectionUtils.java:221) ~[junit-platform-commons-1.10.2.jar:1.10.2]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeAfterAllCallbacks(ClassBasedTestDescriptor.java:461) ~[junit-jupiter-engine-5.10.2.jar:5.10.2]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.after(ClassBasedTestDescriptor.java:236) ~[junit-jupiter-engine-5.10.2.jar:5.10.2]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.after(ClassBasedTestDescriptor.java:85) ~[junit-jupiter-engine-5.10.2.jar:5.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:161) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:161) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) ~[?:?]
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) ~[junit-platform-engine-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:198) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:169) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:93) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:58) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:141) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:57) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:103) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:85) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47) ~[junit-platform-launcher-1.10.2.jar:1.10.2]
    at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:56) ~[surefire-junit-platform-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:184) ~[surefire-junit-platform-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:148) ~[surefire-junit-platform-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:122) ~[surefire-junit-platform-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385) ~[surefire-booter-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) ~[surefire-booter-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507) ~[surefire-booter-3.3.1.jar:3.3.1]
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495) ~[surefire-booter-3.3.1.jar:3.3.1]