Auto compression for HTTP and use zstd

[uasan]

Hey.

I think this topic has already been raised, but once again I want to describe how important it is for real performance.

Most HTTP responses need to be compressed, now this has to be done on the Node side, this is not effective, and Chrome already supports the new compression method zstd: https://chromestatus.com/feature/6186023867908096 Safari and Firefox also plan to implement it.

It is ideal for fast (faster than gzip) stream compression, but Node does not have normal zstd streaming libraries. If uWS implements compression on its side and will check the client Accept-Encoding header, if its value contains zstd then compress zstd, if not then gzip.

This is something that will really speed up all existing servers that use uWS, thank you.

[uNetworkingAB]

Eventually something like this (at least gzip given that zlib which we already depend on, has gzip support) could be added but in what way is not decided (could be automatic like you propose, or just a helper layer, or completely separate). Automatic is kind of promoting inefficient use, as you could in many case pre-compress the file and store it compressed and uncompressed on disk and then serve it based on headers. Then you would have 0 cost server side. So making it in a way that promotes efficient use is key.

[uasan]

Yes, of course, I had that an explicit method was needed, for example something like this: response.isCompresed(true | false) by default there should be no compression.

This method will make it possible not to use compression in Node.js, this will be very useful, over time zstd can be added later

[joshxyzhimself]

if not zstd, brotli can be used too for certain file types.

personally i see this more like a plugin / helper as it's dependent on variable things like

• what files do u serve
• how many cores do u use
• how many servers do u use
• if you'd serve from disk or from memory
• if you'd focus on compression speed or compression ratio

some considerations

• a helper function for serving static files might help, which got its own complexities too
  • use of cache-control and etag headers
  • choice of encoding algos to support
  • normalization and sorting of urls
  • serving the right status codes
  • having authentication and authorization hooks
  • some implementation that we use here: https://gist.github.com/joshxyzhimself/260b4d3b33cc59fd1f64cc0bcd8f83b2
  • hyperexpress has their own implementation at: https://github.com/kartikk221/hyper-express/blob/master/docs/LiveDirectory.md
• if these helpers are to be made for nodejs, there's extra work for deno and bun too
• at certain point you stop optimizing for this and just serve these static files from a proper cdn (like bunny.net which is hella cheap), and use uws where it excels the most which is dynamic http requests and websockets

refs

• https://caniuse.com/zstd
• https://caniuse.com/brotli

[e3dio]

I use a middleware like this:

app.get('/example.js', serveFile('src/example.js'));

app.get('/public/:file', serveDir('public'));

serveFile sets up a file watcher, runs brotliCompress from node:zlib on any file changes, stores the raw bytes and compressed bytes in ram, serves correct file according to request headers and sets Etag header. Might publish that code along with some other helpers

[drag0n-app]

I'm really looking forward to how this new feature can help me be more productive at work.