Add support for tunneling UDP traffic

[PiotrSikora]

I'm going to work on this right now. Could someone assign it to me? Thanks! @howardjohn @costinm

[howardjohn]

Yep, permissions will be fixed by https://github.com/istio/community/pull/879 btw

[costinm]

If it helps: I have https://github.com/istio/ztunnel/pull/145 to add TPROXY, will update it to also capture UDP.

How do you plan to send the packets ? H2 or trying to get QUIC in (for proper MASQUE or 'datagram' use ) ?

[PiotrSikora]

My plan is to use MASQUE's CONNECT-UDP (RFC9297, RFC9298).

This change would be HTTP version agnostic, and since ztunnel is using hyper, it would initially use HTTP/2 (which I agree is terrible for high-throughput and/or low-latency traffic, but it's fine for DNS).

Once we switch the underlying protocol to HTTP/3 (#150), then it would be tunneled over HTTP/3 in QUIC DATAGRAM frames.

[kyessenov]

CONNECT-UDP implementation is in progress in the libraries https://github.com/envoyproxy/envoy/issues/23564.

[costinm]

CONNECT-UDP sounds good, also using H2.

I don't think we will 'switch' to H/3 - but support both H2 and H3.

It is worth looking at WebTransport RFCs - they define a slightly different way to send datagrams over H2, using a regular H2 CONNECT channel and using H3 encapsulation over the H2 stream. There is also an interesting mention that despite H2 stream 'reliability', the datagrams may be dropped ( like UDP ). Not saying we should use it - just worth considering.

My other comment was about capture - let me know if you want me to add the UDP capture to the tproxy script, I only checked in the tcp part but I tested/used the UDP as well. Even if you end up using fancy eBPF, would still be good to have a baseline tproxy-based option ( REDIRECT can't be used with UDP - no way to preserve original dest)

[costinm]

@kyessenov - envoy CONNECT-UDP would not be an issue I think - UDP will mostly be L4, so no Envoy ? And even if it goes to envoy, we can use the 'sandwitch'.

[kyessenov]

Correct, sandwich model removes the need for Envoy to implement MASQUE. That's the main advantage. It boils down to whether we actually need to support any of HBONE/MASQUE/CONNECT-IP in Envoy or always delegate to ztunnel.

[kyessenov]

I guess one benefit of Envoy is QUICHE implementation. Rust one doesn't do UDP over QUIC/UDP.

[PiotrSikora]

Some notes:

• recovering the original destination - iptables REDIRECT and SO_ORIGINAL_DST don't work for UDP, so you have to use iptables TPROXY, IP_TRANSPARENT, IP_RECVORIGDSTADDR and libc::recvmsg (or libc::recvmmsg) to recover the original destination IP:port using ancillary data. This isn't especially complex (see the PR I sent to quinn-rs last month: https://github.com/quinn-rs/quinn/pull/1462), but none of the existing Tokio libraries support that, which means that everything needs to be written from scratch using tokio::try_io.
• UDP stream - the udpflow crate supports a nice TCP-like stream that would work for non-transparent proxying, but it doesn't support recovering the original destination, so similar logic needs to be implemented in ztunnel (or support for recovering the original destination added to udpflow).
• transparent vs non-transparent - transparent proxying is much easier to implement, since it's a stateless fire-and-forget of UDP packets from source to destination and it doesn't require maintaining state to know where to send the response, however it comes with some overhead, since without maintaining a state each UDP packet is a new HBONE request,
• GSO/GRO - I originally started with the quinn-udp crate as a UDP library, but GSO/GRO seems like a premature optimization, and with 64 GRO segments and batching using libc::recvmmsg (32 in quinn-rs), this results in a 18 MB receive buffer when supporting jumbo frames.

I'm going to hand this off to @john-a-joyce and @gilesheron, since it dropped from my priority list and they are eager to work on it.

[stevenctl]

Evaluated some libs for h3: https://docs.google.com/document/d/1MwJlCl0ndhS5ZXSuSTdQUqCb-IGu23m3h7AcnPeHFm0/edit?resourcekey=0-vJ778OjdknOd_QskWPpndg#

For now, I'm focused on getting general H3 support for TCP proxying as groundwork.

[PiotrSikora]

Evaluated some libs for h3: https://docs.google.com/document/d/1MwJlCl0ndhS5ZXSuSTdQUqCb-IGu23m3h7AcnPeHFm0/edit?resourcekey=0-vJ778OjdknOd_QskWPpndg#

For now, I'm focused on getting general H3 support for TCP proxying as groundwork.

I think this comment would be better suited for #150 (QUIC / HTTP/3 between ztunnels) instead of this one (terminating UDP). Also, Nate is already working on this.

[john-a-joyce]

Neither Giles and I have done much work on this and I should probably unassign myself as I am not sure I can make much progress on it. Our initial interest was for how it might intersect with Media Streaming mesh (https://www.mediastreamingmesh.io). Others in the community maybe primarily interested in this for HTTP3 reasons

I think we need some consensus on what the goal of this work should be and which use cases will be supported. Probably some form of a Istio RFC is in order. I am not sure how plausible it is to do the pure tunneling work at the proxy level without deciding if any new or changed APIs are required or at the least how it is enabled/disabled. For example, the scope could be pretty narrow to just allow some UDP ports to be tunneled, allow all udp between two specific workloads, minimal UDP support as required for H3 support or even provide UDP support in the APIs approaching what is provided for TCP.

I can start an RFC if people agree that is a good way to start, but it is also fine if someone else wants to take over ownership.

[stevenctl]

If you've done a bunch of work I think at least compiling your thoughts into some kind of formal or informal doc would be helpful if you do end up handing it off.

[stevenctl]

FWIW I don't think starting with APIs is very necessary at this stage. Enablement/discovery can be figured out later. As long as it's off by-default or even guarded by compile flags we can start proving viability, getting an idea of performance, etc.

[john-a-joyce]

@stevenctl - Most of my work was related to evaluating how UDP support might complement some of what we are doing in the MSM project, but that community is punting on any integration at the current time. That was also why I was suggesting maybe defining what the scope of this work should encompass e.g. just H3 support or support for more general UDP services or even DNS related. I definitely wasn't suggesting trying to define the APIs first. Assuming this is just targeted at H3 support then I agree with you that we can just jump to the getting the plumbing working.

[nmittler]

FYI I just published a first release of BoringSSL support for the Rust Quinn library: https://crates.io/crates/quinn-boring. This will allow us to add h3/quic support to ztunnel and this gives us a path forward with FIPS-compliance (via BoringSSL).

[nmittler]

@john-a-joyce I agree that an RFC is needed. Would you be willing to drive that?

[costinm]

Few comments:

• I don't think we need to worry about API at this stage - encryption/telemetry and getting things delivered is a good first step. After we get it working we can look at routing/balancing/etc ( UDP also tends to need more stickiness )
• I personally don't think H3/MASQUE is the only way to tunnel UDP. The main goal was to have standard-based solutions, but WebTransport and WebRTC are also broadly used standards, maybe even more than CONNECT. If I remember correctly both are dTLD based - and I think they support mutual too.

A very important benefit of WebRTC/WebTransport is broad browser support.

Even for TCP streams - I think the agreement is that CONNECT should be the default tunnel due to the interop and standard-based goals, but not the only option. So if we get a WebRTC based tunneling for UDP first - we can do MASQUE later. WebTransport is also based on QUIC/H3. I suspect there are good Rust libraries for WebRTC.

[costinm]

Looking at mediastreamingmesh - I see they are starting with go, and they have the goal to support per-node model.

Rust is a great language and ztunnel is a clean and efficient tunnel, but we can start with UDP redirected to a second tunnel, and long term it would be great to have 'proxyless' ambient support as well. Which means starting with extensions to mediastreaming mesh to integrate with Istiod over XDS and PTR-DS - in both go and Rust - can be good first steps.

The TProxy part is not very complicated, also need to do transparent to preserve original source, it's even more important than in TCP, many UDP protocols break without it. But there are plenty of examples.

[nmittler]

How does DTLS solve the problem for TCP traffic? Wouldn't we effectively have to implement something like QUIC to handle packet ordering and retransmission? Or is the idea that the tunnel should ALWAYS just be UDP and let the sender handle transmission control (i.e. we only tunnel packets)?

[costinm]

For UDP - no need for ordering or retransmission. That's the problem with UDP-over-H2 or TCP - UDP expects packets to be dropped.

WebRTC does support streams too - similar to QUIC but slightly different.

On Thu, Apr 20, 2023 at 10:07 AM Nathan Mittler @.***> wrote:

How does DTLS solve the problem for TCP? Wouldn't we effectively have to implement QUIC to handle packet ordering and retransmission?

— Reply to this email directly, view it on GitHub https://github.com/istio/ztunnel/issues/148#issuecomment-1516671221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAUR2XTMWXFSJG5WSYIPRDXCFUOJANCNFSM6AAAAAASDQAJ6A . You are receiving this because you were mentioned.Message ID: @.***>

[PiotrSikora]

QUIC DATAGRAM frames support unreliable delivery for proxying UDP (see: RFC9221), so that won't be an issue once you switch to using HTTP/3 as the underlying tunneling protocol (see: #150).

There is no reason to do half-MASQUE half-WebRTC (especially when you'd need to design a tunneling protocol over it), when you can tunnel everything (TCP, UDP, IP) using MASQUE, which is designed specifically for this use case.

[costinm]

On Thu, Apr 20, 2023 at 3:36 PM Piotr Sikora @.***> wrote:

QUIC DATAGRAM frames support unreliable delivery for proxying UDP (see: RFC9221 https://datatracker.ietf.org/doc/html/rfc9221), so that won't be an issue once you switch to using HTTP/3 as the underlying tunneling protocol (see: #150 https://github.com/istio/ztunnel/issues/150).

There is no reason to do half-MASQUE half-WebRTC (especially when you'd need to design a tunneling protocol over it), when you can tunnel everything (TCP, UDP, IP) using MASQUE.

I was not suggesting 1/2 and 1/2.

WebRTC is a widely used protocol - with build in TLS and usually high-throughput/low-latency - it is something we should eventually support natively. It is also a good mechanism to tunnel plain-text UDP - not only in mesh but also on the internet, talking to browsers or devices.

Of course QUIC also has good support for UDP, using MASQUE.

Both are good options and should be supported. Just like for streams we may use H2/CONNECT - but we may also have an option to use H3/CONNECT, maybe H1/CONNECT and other protocols.

We agreed to have H2/CONNECT as the 'default' for streams - and focus on it for the initial release, and add more protocols later. I think QUIC/MASQUE should be the default for datagrams. But if we can get WebRTC working and it proves to be more interoperable or works better - we may start with it.

Message ID: @.***>