Tyrus send partial message more than buffer size will cause exception in catalina OutputBuffer. In original glassfish 4.1 (not nightly build) works fine. Tyrus 1.10 with original glassfish 4.1 works fine too.

HighTower1991 commented 9 years ago

java.lang.IllegalStateException: Cannot set WriteListener for non-async or non-upgrade request at org.apache.catalina.connector.OutputBuffer.setWriteListener(OutputBuffer.java:536) at org.apache.catalina.connector.CoyoteOutputStream.setWriteListener(CoyoteOutputStream.java:223) at org.glassfish.tyrus.servlet.TyrusServletWriter.write(TyrusServletWriter.java:140) at org.glassfish.tyrus.core.ProtocolHandler.write(ProtocolHandler.java:486) at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:274) at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:269) at org.glassfish.tyrus.core.ProtocolHandler.stream(ProtocolHandler.java:402) at org.glassfish.tyrus.core.TyrusWebSocket.sendBinary(TyrusWebSocket.java:406) at org.glassfish.tyrus.core.TyrusWebSocket.sendBinary(TyrusWebSocket.java:392) at org.glassfish.tyrus.core.TyrusRemoteEndpoint$Basic.sendBinary(TyrusRemoteEndpoint.java:141)

HighTower1991 commented 9 years ago

Test case may be simple. Even if send not patial message but more big than buffer size from server endpoint then client session close with buffer overflow and on server will be the same exception. I use annotated endpoint on client and server and send response message to client by session.getAsyncEndpoint().sendBinary and the same exception occur.

peteral commented 9 years ago

I am getting the same exception, however from my investigations it looks to be happening when sending an object message through Async when the Session is currently occupied delivering a previous telegram to client (OutputStream still busy).

Stack trace:

Caused by: java.lang.IllegalStateException: Cannot set WriteListener for non-async or non-upgrade request
    at org.apache.catalina.connector.OutputBuffer.setWriteListener(OutputBuffer.java:536)
    at org.apache.catalina.connector.CoyoteOutputStream.setWriteListener(CoyoteOutputStream.java:223)
    at org.glassfish.tyrus.servlet.TyrusServletWriter.write(TyrusServletWriter.java:140)
    at org.glassfish.tyrus.core.ProtocolHandler.write(ProtocolHandler.java:486)
    at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:274)
    at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:269)
    at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:320)
    at org.glassfish.tyrus.core.TyrusWebSocket.sendText(TyrusWebSocket.java:306)
    at org.glassfish.tyrus.core.TyrusRemoteEndpoint.sendSyncObject(TyrusRemoteEndpoint.java:400)
    at org.glassfish.tyrus.core.TyrusRemoteEndpoint$Async.sendAsync(TyrusRemoteEndpoint.java:295)
    at org.glassfish.tyrus.core.TyrusRemoteEndpoint$Async.sendObject(TyrusRemoteEndpoint.java:256)
    at eisenmann.logic.vstore.hmi.server.SessionRepositoryBean.sendObject(SessionRepositoryBean.java:212)

Tried to work around the problem by using Basic and Locks in EJB Thread pool to only use a Session when idle, but sometimes tyrus keeps waiting for a future forewer and this blocks the whole EJB thread pool and completely blocks the application server:

__ejb-thread-pool47" thread-id: 393 thread-state: WAITING Waiting on lock: java.util.concurrent.CountDownLatch$Sync@1b5a87c2
     at: sun.misc.Unsafe.park(Native Method)
     at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at: java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
     at: java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
     at: java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
     at: java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
     at: org.glassfish.tyrus.core.TyrusFuture.get(TyrusFuture.java:77)
     at: org.glassfish.tyrus.core.TyrusRemoteEndpoint$Basic.sendObject(TyrusRemoteEndpoint.java:177)
     at: eisenmann.logic.vstore.hmi.server.SessionRepositoryBean.sendObject(SessionRepositoryBean.java:212)

Not sure how to reproduce the problem for I only get it in production environment. Was not able to simulate it yet in the stress tests. Probably because they only run via loopback and don't cover the diversity of the real production environment. Will try to make a stress test including real network communication, maybe this helps.

We have following configuration:

HMI system visualising production plants, JEE7 / HTML5 + GWT
Clients Chrome and Firefox around 100
Observed machines causing transactions in the system around 250
There is a relatively high amount of data transferred via the websocket (current status of the installation, client registers to "variables" and gets informed when they change)
also having problems with glassfish in this area although they manifest a little differenty, that's why I started evaluating payara (commercial support also looks interesting after oracle gave it up for glassfish)
currently I have the most stable environment in this area with glassfish 4.0 but this version has tons of other issues

Not sure what to try next, any idea any1?

HighTower1991 commented 9 years ago

Make simple test server and client: https://github.com/HighTower1991/PayaraTest