Deadlock with SelectorRunner in Grizzly 2.4.2

[arinban]

Now and then (after a few weeks of uptime) our Grizzyl-based web server stops working properly. 100% CPU consumption but no requests are executed. The thread dump contains the following message:

Found one Java-level deadlock:

"Thread-245961": waiting to lock monitor 0x00007f99a00778a8 (object 0x00000000c807f460, a java.util.Collections$SynchronizedList), which is held by "grizzly-nio-kernel(2) SelectorRunner"

"grizzly-nio-kernel(2) SelectorRunner": waiting to lock monitor 0x00007f99901fbd58 (object 0x00000000c807b3a8, a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl), which is held by "Thread-245961"

Java stack information for the threads listed above:

"Thread-245961": at java.util.Collections$SynchronizedCollection.remove(Collections.java:2038)

• waiting to lock <0x00000000c807f460> (a java.util.Collections$SynchronizedList) at org.glassfish.grizzly.nio.NIOConnection.removeCloseListener(NIOConnection.java:695) at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl.markResumed(Response.java:2038)
• locked <0x00000000c807b3a8> (a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl) at org.glassfish.grizzly.http.server.Response.resume(Response.java:1976) at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer$ResponseWriter.commit(GrizzlyHttpContainer.java:212) at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:414) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:734) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:884) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:916) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:872) at com.eyo.cloudinary.util.Util.lambda$resume$0(Util.java:88) at com.eyo.cloudinary.util.Util$$Lambda$187/1029390762.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at com.eyo.cloudinary.endpoints.FetchFileEndpoint.buildSuccessResponse(FetchFileEndpoint.java:281) at com.eyo.cloudinary.endpoints.FetchFileEndpoint.buildResponse(FetchFileEndpoint.java:246) at com.eyo.cloudinary.endpoints.FetchFileEndpoint.lambda$null$10(FetchFileEndpoint.java:227) at com.eyo.cloudinary.endpoints.FetchFileEndpoint$$Lambda$206/1247595975.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.lang.Thread.run(Thread.java:748)

"grizzly-nio-kernel(2) SelectorRunner": at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl.markCancelled(Response.java:2060)

• waiting to lock <0x00000000c807b3a8> (a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl) at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl$SuspendCloseListener.onClosed(Response.java:2164) at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl$SuspendCloseListener.onClosed(Response.java:2151) at org.glassfish.grizzly.nio.NIOConnection.invokeCloseListener(NIOConnection.java:1023) at org.glassfish.grizzly.nio.NIOConnection.notifyCloseListeners(NIOConnection.java:890)
• locked <0x00000000c807f460> (a java.util.Collections$SynchronizedList) at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:622) at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:136) at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:108) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

Any ideas? Further information needed? Any help is highly appreciated.

Thanks in advance André

[arinban]

• Issue Imported From: https://github.com/javaee/grizzly/issues/1983
• Original Issue Raised By:@andrelanka
• Original Issue Assigned To:Unassigned
• Original Issue Closed By:@rlubke