Why is (concurrent) HTTP client request/response handling faster in Go than Rust?

[Jaltaire]

I posted this on reddit earlier, but wanted to mention it on GitHub as well just to make sure it gets the right eyes on it --

I'm having a curious result attempting to implement seemingly-equivalent HTTP client request/response handling in Rust and Go...

I propose the following configuration:

• A server returns large, gzip-compressed HTML pages (~100KB when gzip-compressed) of content, for any endpoint https://127.0.0.1:3000/k, where k is an unsigned 32-bit integer.
• A variety of clients (hyper- and reqwest-based in Rust, http-based in Go), where each is configured to concurrently (using tokio::spawn in Rust, goroutines in Go), request n pages from the server where 0 <= k < n, and gzip-decompress the response body.

(For convenience, I uploaded a full repository containing this configuration here, including axum-based server code with self-signed for-dev-only certificates, and the client implementations as described.)

Regardless of the client configuration I have attempted, I find that the Go client implementation consistently beats any Rust client implementation in throughput/speed, and the throughput disparity scales as the number of pages requested increases. For n = 1000, on my machine (M1 Max MacBook Pro, running macOS Sequoia 15.0, with Rust stable 1.82.0), Go runs in ~7.5s, hyper ~8.3s, reqwest ~8.4s (just behind hyper). Of course, there is some variance here, but the overall theme is that Go is around ~1s faster on average for n = 1000. As I note, this difference scales with n. If we change to n=4000, for example, Go runs in ~30.5s, hyper ~35s, reqwest ~36s. Hence, there isn't some constant difference in performance; the performance difference between the two languages becomes exacerbated as n increases.

To this end, I wanted to make note of this issue to ascertain if anyone has hypotheses as to why this might be, or better yet, a solution if there is something missing in the Rust configuration that would allow the Rust implementation to achieve performance parity with or performance gain over the Go implementation. My layman's knowledge of the two languages suggests that Rust should be able to achieve performance at least on par with Go. A couple questions/hypotheses of my own, plus some sourced via reddit comments:

• Are there possible inefficiencies with the hyper or hyper-backed reqwest client implementations (where Go has a "better" implementation for handling/processing responses)? => This comment suggests there is some known issue with a "global lock" in hyper that is causing efficiency issues (though I was unable to find a GitHub Issue discussing this).
• Is the gzip decompression algorithm more-efficiently implemented in Go? => I tried switching to zlib-ng for decompression via the hyper client implementation (per this suggestion), though it didn't seem to offer modest performance gains (if any).
• Do goroutines offer some performance benefit here compared to Rust's async/await or tokio::spawn?
• Is there some (presumably default) client configuration in Go that is available for hyper/reqwest and is not being leveraged?
• Would using a global allocator more-closely mimic Go's memory pool schema, and then lead to similar performance benefits? => My testing suggests this has little to no effect.
• Is printing to stdout faster/more-efficient in Go? => My testing suggests this is not the case (or at least not the main source of a bottleneck).

Note that there are also some configuration changes we can make that improve Rust's performance (e.g., using HTTP/2 over HTTP/1.1) and should offer equivalent benefits in Go, but this is out of scope for the problem identified; Rust and Go implementations are making equivalent requests (both use HTTP/1.1, for example), but Go executes the full program in less time.

As I noted in my reddit comment here, I very much agree with others that the next reasonable step (barring some clear solution) is to profile both Go and Rust implementations in order to identify differences and bottlenecks. As I explain, however, I do not have the bandwidth to continue exploring this myself, but wanted to raise the issue in case (a) there is some easy performance win (e.g., a non-default hyper or reqwest client builder option that should be enabled in most scenarios) and/or some other prior research/investigation on this topic or (b) some true performance bottleneck with hyper/reqwest (or Rust typical concurrency patterns) that has not yet been identified and merits investigation.

[seanmonstar]

I try not to involve myself much in these kinds of comparisons. I know they come from wanting to confirm something is faster, but I've helped too many production environments scale massively using Rust that I don't really want to try to identify what is wrong with a micro benchmark.

The one thing that stuck out as obvious on a quick look was that in Golang, the body is discarded, whereas in Rust it's being copied into a single flat buffer, which can mean wasted copies and reallocs, if the body is sufficiently large.