Writing scalable server applications in the Java™ programming language has always been difficult. Before the advent of the Java New I/O API (NIO), thread management issues made it impossible for a server to scale to thousands of users. The Grizzly NIO framework has been designed to help developers to take advantage of the Java™ NIO API.
Now and then (after a few weeks of uptime) our Grizzyl-based web server stops working properly. 100% CPU consumption but no requests are executed. The thread dump contains the following message:
Found one Java-level deadlock:
"Thread-245961":
waiting to lock monitor 0x00007f99a00778a8 (object 0x00000000c807f460, a java.util.Collections$SynchronizedList),
which is held by "grizzly-nio-kernel(2) SelectorRunner"
"grizzly-nio-kernel(2) SelectorRunner":
waiting to lock monitor 0x00007f99901fbd58 (object 0x00000000c807b3a8, a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl),
which is held by "Thread-245961"
Java stack information for the threads listed above:
"Thread-245961":
at java.util.Collections$SynchronizedCollection.remove(Collections.java:2038)
waiting to lock <0x00000000c807f460> (a java.util.Collections$SynchronizedList)
at org.glassfish.grizzly.nio.NIOConnection.removeCloseListener(NIOConnection.java:695)
at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl.markResumed(Response.java:2038)
locked <0x00000000c807b3a8> (a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl)
at org.glassfish.grizzly.http.server.Response.resume(Response.java:1976)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer$ResponseWriter.commit(GrizzlyHttpContainer.java:212)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:414)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:734)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:884)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:916)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:872)
at com.eyo.cloudinary.util.Util.lambda$resume$0(Util.java:88)
at com.eyo.cloudinary.util.Util$$Lambda$187/1029390762.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at com.eyo.cloudinary.endpoints.FetchFileEndpoint.buildSuccessResponse(FetchFileEndpoint.java:281)
at com.eyo.cloudinary.endpoints.FetchFileEndpoint.buildResponse(FetchFileEndpoint.java:246)
at com.eyo.cloudinary.endpoints.FetchFileEndpoint.lambda$null$10(FetchFileEndpoint.java:227)
at com.eyo.cloudinary.endpoints.FetchFileEndpoint$$Lambda$206/1247595975.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at java.lang.Thread.run(Thread.java:748)
"grizzly-nio-kernel(2) SelectorRunner":
at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl.markCancelled(Response.java:2060)
waiting to lock <0x00000000c807b3a8> (a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl)
at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl$SuspendCloseListener.onClosed(Response.java:2164)
at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl$SuspendCloseListener.onClosed(Response.java:2151)
at org.glassfish.grizzly.nio.NIOConnection.invokeCloseListener(NIOConnection.java:1023)
at org.glassfish.grizzly.nio.NIOConnection.notifyCloseListeners(NIOConnection.java:890)
locked <0x00000000c807f460> (a java.util.Collections$SynchronizedList)
at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:622)
at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:136)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106)
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:108)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
Any ideas? Further information needed? Any help is highly appreciated.
Now and then (after a few weeks of uptime) our Grizzyl-based web server stops working properly. 100% CPU consumption but no requests are executed. The thread dump contains the following message:
Found one Java-level deadlock:
"Thread-245961": waiting to lock monitor 0x00007f99a00778a8 (object 0x00000000c807f460, a java.util.Collections$SynchronizedList), which is held by "grizzly-nio-kernel(2) SelectorRunner"
"grizzly-nio-kernel(2) SelectorRunner": waiting to lock monitor 0x00007f99901fbd58 (object 0x00000000c807b3a8, a org.glassfish.grizzly.http.server.Response$SuspendedContextImpl), which is held by "Thread-245961"
Java stack information for the threads listed above:
"Thread-245961": at java.util.Collections$SynchronizedCollection.remove(Collections.java:2038)
"grizzly-nio-kernel(2) SelectorRunner": at org.glassfish.grizzly.http.server.Response$SuspendedContextImpl.markCancelled(Response.java:2060)
Found 1 deadlock.
Any ideas? Further information needed? Any help is highly appreciated.
Thanks in advance André