Response Body Compression

[ofekd]

To reduce request size, often the body is compressed. A mechanism that supports this is currently missing from Wisp.

I am not quite sure what scope of a solution is a good fit for Wisp. A full flow will be:

1. Detect compressions supported by the browser using the accept-encoding header
2. Compress non-binary bodies
3. Set the content-encoding header to indicate the compression for responses compressed on the fly
4. Allow setting the header for static files that are pre-compressed

Steps 1 through 3 are fairly straightforward to do on top of wisp:

  let accept_encoding =
    req.headers
    |> list.find_map(fn(header) {
      case header.0 == "accept-encoding" {
        True -> Ok(header.1)
        False -> Error(Nil)
      }
    })
    |> result.unwrap("")
    |> string.split(",")
    |> list.map(string.trim)

  let deflate_accepted = accept_encoding |> list.contains("deflate")

  case res.body, deflate_accepted {
    wisp.Text(body), True ->
      res
      |> wisp.set_header("content-encoding", "deflate")
      |> wisp.set_body(
        body
        |> string_builder.to_string
        |> bit_array.from_string
        |> gzlib.compress
        |> bytes_builder.from_bit_array
        |> wisp.Bytes,
      )
    _, _ -> res
  }

But there's no clear way to achieve step 4

My current work around is to hard-code the file names:

  let res =
    wisp.serve_static(
      req,
      under: "/priv/static",
      from: client_priv_static,
      next: fun,
    )

  let compressed_statics = [ "script.mjs" ]

  case compressed_statics |> list.any(string.contains(req.path, _)) {
    True -> res |> wisp.set_header("content-encoding", "gzip")
    False -> res
  }

But even beyond the hard-coding of file name, this checks every response

[lpil]

Setting the header for pre-compressed files sounds good!

I'm curious about compressing on the fly. It seems very wasteful to do this every time, so I'm not sure it's something I would want to encourage in Wisp itself.

[ofekd]

Compressing on the fly is unavoidable for some dynamic content. I had a simple backend serving a lustre-rendered html + some inlined tailwind. It was ~65kb uncompressed, and ~7.5kb compressed with zlib.

When benchmarking, the network bandwidth on the server got saturated way before the CPU, which is quite a waste. Compression solved that completely at about 10-20% CPU cost, which led to overall better utilization. I bet if I had used Erlang's brotli library it would have been much easier on the CPU.

Since the exact compression setup is different for every app (some would do it on the BE, some would do on the proxy, some would compress on the BE then cache at the proxy, etc), maybe it is better to leave it to the users to implement as middleware.

This is fairly simple to do when writing your own middleware, but that serve_static situation -- or really, extending any other middleware -- is tricky. I don't know if this calls for tweaking the middleware API, or is just a matter of addressing the serve_static middleware specifically

[lpil]

I was talking about it being wasteful in the context of serve static specifically. Having a package that provides compression sounds like a great idea to me!