Created attachment 30512full thread stack dumpI have a websocket test which performs some stress testing over an echo example server running on Tomcat 8. My test is creating up to 200 websocket connections and perform some message sending/receiving. At some point of time after some test restarts, the server stops to respond. I have restarted the test with a simple single socket connection and the server was still not responding. I checked the threads stacks at the server side and it seems all of the Tomcat worker therads were blocked. Attaching the full stack trace of all threads. In the server's logs file I can see only several exceptions like this:Jul 01, 2013 11:34:52 AM org.apache.tomcat.websocket.server.WsHttpUpgradeHandler destroySEVERE: Failed to close WebConnection while destroying the WebSocket HttpUpgradeHandlerjava.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:50) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94) at sun.nio.ch.IOUtil.write(IOUtil.java:51) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:135) at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:385) at org.apache.coyote.http11.upgrade.NioServletInputStream.doClose(NioServletInputStream.java:107) at org.apache.coyote.http11.upgrade.AbstractServletInputStream.close(AbstractServletInputStream.java:128) at org.apache.coyote.http11.upgrade.AbstractProcessor.close(AbstractProcessor.java:59) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.destroy(WsHttpUpgradeHandler.java:137) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1584) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1543) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:789)Note: My test setup includes proxy and loadbalancer in the middle. In some cases the loadbalancer closes websocket connections which is seen as connection reset by the server and the client.
Created attachment 30512full thread stack dumpI have a websocket test which performs some stress testing over an echo example server running on Tomcat 8. My test is creating up to 200 websocket connections and perform some message sending/receiving. At some point of time after some test restarts, the server stops to respond. I have restarted the test with a simple single socket connection and the server was still not responding. I checked the threads stacks at the server side and it seems all of the Tomcat worker therads were blocked. Attaching the full stack trace of all threads. In the server's logs file I can see only several exceptions like this:Jul 01, 2013 11:34:52 AM org.apache.tomcat.websocket.server.WsHttpUpgradeHandler destroySEVERE: Failed to close WebConnection while destroying the WebSocket HttpUpgradeHandlerjava.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:50) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94) at sun.nio.ch.IOUtil.write(IOUtil.java:51) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450) at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:135) at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:385) at org.apache.coyote.http11.upgrade.NioServletInputStream.doClose(NioServletInputStream.java:107) at org.apache.coyote.http11.upgrade.AbstractServletInputStream.close(AbstractServletInputStream.java:128) at org.apache.coyote.http11.upgrade.AbstractProcessor.close(AbstractProcessor.java:59) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.destroy(WsHttpUpgradeHandler.java:137) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1584) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1543) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:789)Note: My test setup includes proxy and loadbalancer in the middle. In some cases the loadbalancer closes websocket connections which is seen as connection reset by the server and the client.