Jetty 12.0.5 WebSocket 'per-message-deflate' truncates data received in browsers only under Windows

[minduch]

Jetty version(s) Jetty 12.0.5

Jetty Environment Jetty 12.0.5 websockets

Java version/vendor openjdk version "21.0.1" 2023-10-17 LTS OpenJDK Runtime Environment Temurin-21.0.1+12 (build 21.0.1+12-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (build 21.0.1+12-LTS, mixed mode, sharing) (but it also happens in e.g. Java 17 without Virtual Threads)

OS type/version Client side: Win11Pro and other Windows versions with latest Chrome/ChromeCanary/Firefox/Edge. HOWEVER: on macOS Sonoma -- IT WORKS JUST FINE !!! (don't ask me why) Also seems to be OK on Firefox 120.0 on at least Ubuntu.

Description per-message-deflate extension in WebSockets causes (BINARY) messages to be truncated to the client side. It looks like the client (JavaScript Chrome) gets an uncompressed message with the length of the deflated message, i.e. cut off!

I have not seen this problem in versions of Jetty prior to 12.0.3, however, I saw it in quite early versions like 9.x when websockets came out.

How to reproduce? Our pretty complex platform consists of embedded Jetty 12.0.3 and a JavaScript client. It's available on the net, but for now, we have had to turn off Jetty WebSocket Extensions by calling JettyServerUpgradeResponse.setExtensions(Collections.emptyList());. This "cures" the problem. The REALLY strange thing is that iOS and macOS (Sonoma) Chrome/Safari works just fine with the extensions turned on!

I attached the server-side logging and the client side in two Word documents. Search for "HERE IS THE PROBLEM" in yellow background. We added a header to our message to know internally how long it SHOULD be on the JavaScript side, and in this case, it is truncated from 61937 bytes to 41658 (looks like deflated size), (sorry for the character set being "translated" when pasted into Word).

Right now we have updated all our servers to disable WebSocket Extensions in Jetty in order to make it work, but it's a configurable environment variable, so if you need more info, I will be happy to help!

client-side.docx server-side.docx

[minduch]

Do you think the bug is in Windows versions of Chrome/Firefox/Edge, as I do not get this problem under iOS (17.1.2) or macOS (Sonoma)?

[sbordet]

@minduch I checked the server logs and all seems ok: we split the content into 4 writes of 8192 compressed bytes + some remaining, and we correctly remove the tail bytes added by the compressor, as mandated by the specification.

Seems really a client bug. Does it work if you use other clients that are not browsers, such as JDK's HttpClient, or Jetty's WebSocketClient, etc.?

[minduch]

@sbordet I did check too, however I'm not so good at looking at the Jetty debug output :-(. It sure looks like a client bug, but what really bothers me a LOT is that it affects Firefox, Edge, Chrome and Chrome Canary on Windows, and apparently not iOS and mac Safari (any version), and Chrome on macOS/iOS (latest versions).

On the client side, you see it receiving correctly the first 41658 bytes, but not the entire package of 61933 bytes!

For HttpClient or WebSocketClient, I don't have such clients to this server.

Do you have a test-case Jar I could run in my environment to test it out?

[minduch]

@sbordet Does Jetty have test cases with e.g. Chrome as Client using large binary WebSocket blocks received and send from/to a Jetty Server with and without permessage-deflate? If so, I could perhaps run them here to check...

[minduch]

I ran into the QNAP NAS products having similar problems with WebSockets and Chrome/Edge/Firefox. It did work well on macOS Safari though (from latest Sonoma 14.2.1 or iOS).

Have you hard of similar problems?

[sbordet]

@minduch I just tried with FF 121 on Ubuntu and permessage-deflate works fine for me.

Are you using Jetty core or ee10?

Since 12.0.3 there were very minimal changes to WebSocket, and none to the permessage-deflate code, so it's strange that the problems does not show in 12.0.3.

Can you give https://github.com/jetty/jetty.project/pull/11090 a try, and see if the problem persist?

[minduch]

@sbordet This problem remains in 12.0.5 and the "todays" latest versions of Edge (121.0.2277.4 (Official build) stable app, beta channel (64-bit)), Firefox, Chrome and Chrome Canary under Windows. It doesn't matter if WebSockets use HTTP or HTTPS, it gets the SAME error.

As MANY users do run Windows and not all macOS/Linux, I think this problem is quite severe, alothough there is a workaround to turn the WebSockets Extensions OFF.

Safari works OK on macOS and iOS. I can confirm that Firefox 120.0 under Ubuntu works fine too.

Is the fix #11090 included in 12.0.5, or will it be included in 12.0.6 (later this month)?

[minduch]

I also edited the client environment in the first report of this issue and the Jetty version to 12.0.5.

@sbordet What really bothers me is that it's all the major browsers under Windows, with HTTPS and HTTP, and it's not HTTP/2-related, nor HTTP/3, nor Conscrypt, nor GzipHandler from issue #11095 - I tested to turn off our HTTP/2+3+Conscrypt and it still shows the error every time.

[sbordet]

Is the fix https://github.com/jetty/jetty.project/pull/11090 included in 12.0.5, or will it be included in 12.0.6 (later this month)?

12.0.6

[sbordet]

@minduch would be helpful to understand if there is a message size that triggers this.

Can you test 1-10-100-1000-etc. bytes and see which one works and which one does not?

We're trying to setup a Windows environment to test it out.

[joakime]

Testing on Windows 10 (up to date as of Jan 2, 2024) Adoptium 21.0.1 Jetty 12.0.5 Only the HTTP/1.1 connector (no TLS, no HTTP/2)

There are no problems encountered with permessage-deflate with TEXT or BINARY on ...

• Chrome 120.0.6099.130
• Firefox 121.0
• Edge 120.0.2210.91

This was done on the Jetty 12 ee10 environment. It was using the jakarta.websocket API Sending TEXT or BINARY messages ranging from 10 bytes to 500,000 bytes.

What specific scenarios do you have? What APIs are you using? What size binary messages are you sending?

[joakime]

If you have any active firewall products on your windows computer, turn them off and try again (we've had reports in the past of various security software interfering with networking in browsers).

Also, do you have a proxy in use? (this is another source of odd behaviors)

[sbordet]

Oohh, right, Windows Firewall/Defender -- always forget about this Windows feature! 🙄

[minduch]

OK, so I've modified our Feodora 39 Server on Internet at "mt.iizi.co", running Temurin 21.0.1 with Virtual Threads. We do not use any proxies. Accessing "mt.iizi.co" is direct to Jetty 12.0.5 on ports 80, 8080 and 443 (TLS certs from Let's Encrypt using ACME protocol). Actually, it should also open for HTTP/3 on UDP port 443.

It fails when starting the app in Chrome/Edge/Firefox under Windows 10 and 11 using the URL: Launch test webapp

It works in Ubuntu 22 Firefox 120.0 AND Chrome 120.0.6099.129 (just installed it).

Latest AppStore-available Chrome and Edge under iOS 17.2.1 on iPhone has stopped working, Safari under same iOS works fine.

Latest macOS Sonoma 14.2.1 works with Safari, but not under Chrome [120.0.6099.129 (Official Build) (arm64) on MacMini M2 Pro], DevTools Console shows net::ERR_HTTP2_PROTOCOL_ERROR:

Above, [dollar]release.jquery.file[dollar] is an alias where we reply with a Jetty Resource to the current jQuery minimized file version (jquery-3.7.1.slim.min.js). Then I reload the page in Chrome again, now loading goes a bit further, Console shows:

I don't really understand anymore. The error occurs under Windows when receiving binary data (in this case using HTTPS and perhaps HTTP/2, I'm not sure. We have AVG firewall in use, but I have turned it off with the same results, i.e. when no Firewall is active, AVG and/or Windows Defender.

Our JavaScript program reports:

or

Try it out... We use these Jetty + Jakarta classes/annotations:

import org.eclipse.jetty.ee10.servlet.ServletContextHandler;
import org.eclipse.jetty.ee10.servlet.ServletHolder;
import org.eclipse.jetty.ee10.websocket.server.JettyWebSocketCreator;
import org.eclipse.jetty.ee10.websocket.server.JettyWebSocketServlet;
import org.eclipse.jetty.ee10.websocket.server.JettyWebSocketServletFactory;
import org.eclipse.jetty.ee10.websocket.server.config.JettyWebSocketServletContainerInitializer;
import org.eclipse.jetty.server.handler.ContextHandler;
import org.eclipse.jetty.websocket.core.server.FrameHandlerFactory;

import org.eclipse.jetty.websocket.api.Callback;
import org.eclipse.jetty.websocket.api.Frame;
import org.eclipse.jetty.websocket.api.Session;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketFrame;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage;
import org.eclipse.jetty.websocket.api.annotations.OnWebSocketOpen;
import org.eclipse.jetty.websocket.api.annotations.WebSocket;

import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;

Please find the Jetty log created with the Jetty logger: http-2024_01_03.request.log

Our logger (showing multiple warnings due to Jetty GzipHandler, see other issue @sbordet has generated a PR for): server.log

Please let me know if you need more info, I will be happy to provide you with it.

Again: Happy New Year!

[joakime]

Testing on the same Windows 10 laptop against your URL shows no signs of the failures you are experiencing.

Chrome

Firefox

All but one request succeeds. (The request to https://maps.googleapis.com/maps/api/mapsjs/gen_204?csp_test=true fails in both browsers). Every websocket message succeeds (in both directions)

[joakime]

Edge shows no warnings / errors / failure as well.