(suggestion) Use s2n-quic instead of quinn for better performance and stability

[paulocoghi]

According to quic-interop-runner results (repo), the quiche HTTP/3 library from Cloudflare (also made in Rust), is faster than quinn.

In more than one test, quiche was more than 2x faster than quinn, and there were tests where quinn failed while quiche succeeded.

Not only that, quinn's maintainers already stated on https://github.com/quinn-rs/quinn/issues/1320 (still open) that quinn is currently maintained by only two volunteers, and this makes difficult to improve quinn in the same pace of other more robust libraries, like quiche, maintained by Cloudflare.

[junkurihara]

Hi Paulo,

That's a nice suggestion, thanks! Yes, considering the perspectives of performance/maintenance, changing HTTP/3 library from quinn to quiche would be better. But to this end, there are several technical challenges.

• quiche only provides really 'low-level' APIs. So we need to handle low-level event loop.
• quiche uses BoringSSL, but rpxy currently heavily relies on rustls. We should manage both crypto materials inside rpxy. (This probably works, but will be a bit messy in the code)
• We use hyper to quickly handle HTTP messages proxying from/to backend applications. But since quiche::h3 is not compatible with hyper, we should consider how to glue hyper with quiche.
  • One option is to ignore quiche::h3 and implement hyperium/h3 (hyper's HTTP/3 trait) for quiche itself. This was considered here https://github.com/hyperium/h3/issues/147 and an experimental implementation was provided at eagr/h3-quiche.
  • The first option has a problem that quiche doesn't provide sufficient interface to appropriately handle HTTP/3 streams as mentioned at https://github.com/hyperium/h3/issues/147#issuecomment-1331973240. So another option is to glue quiche::h3 with hyper as suggested in the comment . I think this is better than the first option.

Anyways, I think this idea should be seriously considered to maintain rpxy. So first, I will test quiche separately from rpxy.

(I know the most ideal option is that I contribute quinn and improve its performance dramatically... Also from our initial philosophy of rpxy, it should be written in pure Rust with community-based codes as much as possible. Of course, that's really tough, and sometimes impossible...)

[paulocoghi]

If the challenges described become too problematic, maybe it's better to continue with quinn until we have a better scenario.

[junkurihara]

Agree. I will take a look the code in deep.

[paulocoghi]

Update of my previous suggestion: s2n-quic instead of quiche

It is as fast as quiche, is maintained by AWS and can use s2n-tls or rustls, both pure-rust TLS libraries.

The only drawback is that s2n-quic fails in a few tests (as quinn does, as well), but at least it provides the same performance as quiche

(on the tests, I believe that s2n-quic is using s2n-tls, which seems to be the easy and natural path)

[junkurihara]

Hi Paulo, thanks for your information!

Yes, s2n-quic looks a promising option. Moreover it provides s2n-quic-h3 which is an implementation of hyperium/h3 used in rpxy. So, I think it would be easier to use it inside rpxy than quiche!

[junkurihara]

I am roughly testing s2n-quic and s2n-quic-h3 from the viewpoints of

• use of rustls to be compatible with http2 and 1.1,
• serve multi-domain.

At this point, I confirmed that it works as a simple http/3 server using s2n-quic-h3 with hyperium/h3. I think that we can reuse the most part of codes written for h3-quinn for s2n-quic-h2, ands2n-quic and s2n-quic-h3 can be used in rpxy much like quinn. So so I will test more.

Also I think the better way for rpxy is to support both quinn and s2n-quic as features, where either one is default.

[paulocoghi]

Also I think the better way for rpxy is to support both quinn and s2n-quic as features, where either one is default.

If this doesn't create an additional burden on you (and future contributors) regarding the code maintainability, this approach has the advantage of making rpxy more modular with the different libraries and implementations of HTTP/3

[junkurihara]

I just made an example implementation of HTTP/3 server (and client) using s2n-quic with rustls. https://github.com/junkurihara/s2n-quic-h3-examples

As my first impression, s2n-quic seems to probably have limitations on configurability of some parameters with rustls like TLS cipher suites. But I think it would not be a big issue.

[junkurihara]

Here's initial integration of s2n-quic! https://github.com/junkurihara/rust-rpxy/pull/65

[paulocoghi]

Incredible work! :tada: :tada: :tada:

(If you want, I believe we can now close this issue. Mission accomplished!)

[junkurihara]

Thanks Paulo!

Yes, I will close this issue soon after I merge the branch to main :-)

[junkurihara]

Merged!

[paulocoghi]

What a nice journey :checkered_flag: