SockJS websocket timeout on Websphere

zanikolov commented 7 years ago

Hello,

We have a very interesting problem in the Websocket communication which depends on whether we use SockJS or not. Our application uses Websocket protocol for communication between the backend and frontend. In the backend side we use Spring Websocket with SockJS registry.addEndpoint("/ws") .withSockJS() .setClientLibraryUrl("https://cdn.jsdelivr.net/sockjs/1.1.0/sockjs.min.js");

In the frontend app we use a Stomp protocol over SockJS. var socket = new SockJS(backendUrl + '/ws'); stompClient = Stomp.over(socket);

We had ran our backend application on Tomcat and the frontend application on Websphere Application Server (with IHS in front of it. The IBM HTTP Server is Apache HTTP Server under the hood). In this scenario, all the Websocket communication ran smoothly and without issues. Afterwards, we got a requirement to move our backend application on Websphere Application Server (version 9, which supports Websockets). In front of the WAS, we also have IHS. Here, we found a problem - the SockJS websocket communication times out and SockJS switches to XHR. In the SockJS logs, we get this message: _sockjs-client:browser:xhr cleanup +2ms browser.js:120 sockjs-client:main transportTimeout +8s browser.js:120 sockjs-client:main transportClose +0ms undefined 2007 Transport timed out

In the /websocket request, we have the following headers: _Accept-Encoding:gzip, deflate, sdch, br Accept-Language:en-US,en;q=0.8,bg;q=0.6 Cache-Control:no-cache Connection:Upgrade Cookie: {some cookies} Host:{backend host} Origin:{frontend host} Pragma:no-cache Sec-WebSocket-Extensions:permessage-deflate; client_max_windowbits Sec-WebSocket-Key:krf8DAkLQhO3mJJpKuRF9A== Sec-WebSocket-Version:13 Upgrade:websocket User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36

And in the response: Access-Control-Allow-Credentials:true Access-Control-Allow-Origin:{frontend host} Connection:Upgrade Content-Language:en-US Content-Length:0 Date:Wed, 23 Nov 2016 06:26:16 GMT Sec-WebSocket-Accept:VarVcM0dY/2koYDQ3G3rpy6XMYo= Strict-Transport-Security:max-age=31536000; includeSubDomains; preload Upgrade:websocket Vary:Origin X-Content-Type-Options:nosniff X-Powered-By:Servlet/3.1 X-XSS-Protection:1; mode=block

The most interesting part of the problem is that when we turn off SockJS - the websockets run normally. We are fighting with this problem in the last two weeks and really don't know how to continue. Every tip or help will be highly appreciated.

Thank you, Zahari

brycekahle commented 7 years ago

Chrome will allow you to inspect the Websocket connection. Do you ever see any data sent over it?

rstoyanchev commented 7 years ago

hi, it's not very clear what the frontend and what the backend is in this case and the scenarios with Tomcat vs WebSphere. Can you clarify whether you meant that WebSocket without SockJS works when WebSphere is the WebSocket server or did you mean that it works when Tomcat is the WebSocket server? Also if there is an HTTP server in each case?

For a timeout issue it could be that the HTTP server might need a configuration tweak to support WebSocket traffic or perhaps the WebSocket response is too slow to arrive before SockJS falls back.

grudir commented 7 years ago

Hi folks, Thanks for helping. Here are the details:

@brycekahle There is no data sent over to server when we use SockJS. I checked in Chrome but there are no frames. However in FF (with a WebSocket addon) I do see some frames received. Unfortunately no CONNECT command is sent from the front-end. The server (WAS9) however sends a few heartbeats and eventually closes the connection. I debugged the back-end and figured out that this is due to a timeout in Spring which kicks in if the client has not initiated the connection (probably with the CONNECT command). It seems that SockJS is not sending anything to the server. Then timeouts and switches to the next transport.

@rstoyanchev At the moment both the front-end and back-end are JEE web applications running on the same WAS9 with an IHS in front of it. WAS9 is the WebSocket server. In the past when everything was working fine the back-end was deployed on Tomcat (the WebSocket server) on the same domain and machine as WAS, but Tomcat traffic was not going through IHS. At that time WAS was at version 8.5.5.

I also tried disabling SockJS in the WAS9 scenario. The WebSocket connection works for a short period. CONNECT and CONNECTED commands along with a few others are exchanged and then Stomp closes the connection. I'm not sure why. But at least the connection works for a while.

We use stomp-websocket 2.3.4 and sockjs-client 1.0.2.

rstoyanchev commented 7 years ago

I suspect the issue is related to the IBM HTTP server. The easiest way to isolate the issue is to try running without it? Then you can figure out what's wrong exactly. I found the following on WebSocket support for it so you can check that you have that version at least.

Clearly though if even a basic WebSocket connection is not working something is wrong that's not related to the use of SockJS. If the above does not prove the issue, try running a really plain WebSocket example, possibly something that IBM provides. If that doesn't work you know who to talk to.

nlebas commented 7 years ago

I've encountered a similar issue and in my case, it is not related to the server side of the websocket, but with the network load.

In most cases everything works fine, but sometimes hell breaks loose. Here is the sequence of events:

the website loads and fires a bunch of HTTP requests.
new SockJS(url)
the info request flies by quickly (~3ms)
the websocket request fires, but it is stuck for 5 seconds for some reason, probably the limit of concurrent connections per website stands in the way.
before the browser comes around to actually sending out the websocket request, SockJS' timeout expires and closes it.
the server receives and accepts the request but finds the websocket already closed.
SockJS moves on to try out other protocols if they are enabled (I disabled them in my case)

Workaround for me I've disabled SockJS' internal timeout with:

var sockJSConn = new SockJS(url);
sockJSConn._transportTimeout = function() { console.log('gotcha!!!'); };

Works for me so far, but it feels a bit like trespassing.

Hope this helps,

vadakr commented 4 years ago

I am experiencing a similar issue with Spring, Stomp over SockJS and WebSphere.

Spring = 5.1.8 Stomp-websocket = 2.3.3 SockJS client = 1.1.5 WebSphere = 9.0.5 (traditional, NOT Liberty)

Here's the websocket request in WireShark -

I then see this suspicious HTTP 'Continuation' -

And then the connection upgrade confirmation -

About a second later, the SockJS client wants to close the connection -

Finally, the server closes the connection -

Here are Spring debug logs from the server. Note that it's +4 hrs compared to the Wireshark time.

2020-06-14 20:14:37,881 WebContainer : 5 GET "/app/msg/endpoint/039/uoboylhr/websocket", parameters={}, headers={masked} in DispatcherServlet 'SpringMVC' 2020-06-14 20:14:37,932 WebContainer : 5 Mapped to HandlerExecutionChain with [org.springframework.web.socket.sockjs.support.SockJsHttpRequestHandler@69740d0c] and 1 interceptors 2020-06-14 20:14:37,933 WebContainer : 5 Skip: request is from same origin 2020-06-14 20:14:37,934 WebContainer : 5 Processing transport request: GET http://localhost:9080/app/msg/endpoint/039/uoboylhr/websocket 2020-06-14 20:14:38,103 WebContainer : 5 No view rendering, null ModelAndView returned. 2020-06-14 20:14:38,104 WebContainer : 5 Completed 101 SWITCHING_PROTOCOLS, headers={masked} 2020-06-14 20:14:38,118 WebContainer : 5 New WebSocketServerSockJsSession[id=uoboylhr] 2020-06-14 20:14:38,134 WebContainer : 5 Sending TextMessage payload=[o], byteCount=1, last=true], StandardWebSocketSession[id=8b54616f-f37f-8c57-a6c2-80aa04d8e606, uri=/app/msg/endpoint/039/uoboylhr/websocket] 2020-06-14 20:14:38,136 WebContainer : 5 Scheduled heartbeat in session uoboylhr ... 2020-06-14 20:14:39,159 WebContainer : 9 WebSocketServerSockJsSession[id=uoboylhr] closed with CloseStatus[code=1000, reason=] 2020-06-14 20:14:39,159 WebContainer : 9 Clearing session uoboylhr

I can't really see the "open" frame in the Wireshark trace, other than that spurious 'Continuation'. Is that the problem?

github-actions[bot] commented 4 years ago

This issue has been inactive for 30 days. It will be in closed in 5 days without any new activity.

luhongjie commented 5 months ago

我遇到了类似的问题，就我而言，它与 websocket 的服务器端无关，而是与网络负载有关。

在大多数情况下，一切正常，但有时地狱会松动。以下是事件的顺序：

该网站加载并触发了一堆 HTTP 请求。

new SockJS(url)

信息请求飞快（~3ms）

WebSocket 请求触发，但由于某种原因卡住了 5 秒钟，可能是每个网站的并发连接数限制阻碍了它。

在浏览器实际发送 websocket 请求之前，SockJS 的超时过期并关闭它。

服务器接收并接受请求，但发现 WebSocket 已关闭。

如果启用了其他协议，SockJS 会继续尝试它们（我在这种情况下禁用了它们）

我的解决方法：我禁用了 SockJS 的内部超时：
var sockJSConn = new SockJS(url);
sockJSConn._transportTimeout = function() { console.log('gotcha!!!'); };
到目前为止对我有用，但感觉有点像擅自闯入。

希望这会有所帮助，

perfect！！！

sockjs / sockjs-client

SockJS websocket timeout on Websphere #346