search

envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway
https://gateway.envoyproxy.io
Apache License 2.0
1.54k stars 331 forks source link

Add support for UDPRoute API #641

Closed arkodg closed 1 year ago

arkodg commented 1 year ago

Description: Add Support for the UDP Route API defined here

Related https://github.com/envoyproxy/gateway/issues/47

arkodg commented 1 year ago

we should wait on https://github.com/envoyproxy/gateway/issues/413 which will refactor the provider before starting this issue

zhaohuabing commented 1 year ago

@arkodg May I take this one? It would be a relatively easy start for me to dig into the core mechanism of EG since I can refer to the implementation of the current HTTRoute. If this one goes well, I would like to take https://github.com/envoyproxy/gateway/issues/642 as well.

arkodg commented 1 year ago

go for it @zhaohuabing, suggest raising incremental PRs in the below order to increase PR review speeds and reduce merge conflicts :)

youngnick commented 1 year ago

I'd like to see some work done on how the UDP flows will actually work. It's pretty straightforward to set up a UDP listener for Envoy, but it's very difficult to preserve the client address at the same time.

Basically, you have two options:

In particular, on Contour, we've been asked to have Contour handle SIP traffic, and SIP in particular is a very bad fit for this sort of proxying, as the RTP used for the actual data streams (as opposed to the control traffic) is very dynamic and difficult to proxy correctly.

I'm not opposed to doing UDPRoute in Envoy Gateway, but I would like to see significantly more design around how to make it available in a way that meets user expectations. Just adding support for the existing UDP modes in Envoy does not match up to what users will expect when you add UDP forwarding.

zhaohuabing commented 1 year ago

I think transparent proxy mode can be benifical for both TCP and UDP since in some use cases, the server behind the EG would like to know the original addreess of the client. To get this done, we'll need to set the route inside a cluster to get the return traffic route back to the EG, and then EG can send the traffic right back to the client.

arkodg commented 1 year ago

@youngnick the https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/udp_filters/udp_proxy docs states that envoy operates as a non transparent proxy. The xds implementation does not seem too complex, if you prefer, either @zhaohuabing or I could PoC with Envoy proxying UDP traffic to a pod running a TFTP server or UDP Ping server and share those details on this issue. If some UDP protocols dont like non transparent mode , we will have have add those limitations to the EG docs for now.

zhaohuabing commented 1 year ago

non-transparent proxy should work, I'll test that after the translators implemention is done. transparent proxy mode is also helpful, maybe we could discuss how to support it if there're more requests asking for it from the clients. It looks like we need to introduce new components if we want to addd transparent mode, we will need a CNI to set up the route. Besides, we should also consider the transparent mode for TCP listeners together.

arkodg commented 1 year ago

Reread the envoy docs which states that envoy also support transparent mode but needs added privileges ( not given to the envoy deployment today ) @youngnick do you have any info handy on which protocols support non transparent mode and which dont ?

youngnick commented 1 year ago

It's tricky, but in general UDP protocols expect the client address to be the actual one. DNS, and syslog are the ones I can think of from the top of my head. I'm not sure about other ones - I know that lots of gaming applications use UDP as well. It's going to be on a per-protocol basis though - and lots of people roll their own protocol.

If we really want to do it, it will be extremely important to call out the no-client-address limitations of proxied mode, definitely.

For transparent proxying, changing the routing seems like it needs pretty deep integration with the CNI or kube-proxy - I don't feel like us building that into Envoy Gateway is a very good idea without doing a pretty detailed design first.

danehans commented 1 year ago

xref https://github.com/envoyproxy/gateway/pull/646 for adding UDPRoute to IR.

danehans commented 1 year ago

Use Envoy in non-transparent proxy mode, which means that the client address will be seen at the backend pods as Envoy's address. As I said, most UDP protocols don't like this.

Although most UDP protocols don't like this, non-transparent proxy mode could be considered "initial" UDPRoute support. We can document these details while we work on a long-term UDPRoute solution that covers the other UDP protocols.

@mattklein123 PTAL at this issue and #643. Let us know if you have any input on adding L4 support.

danehans commented 1 year ago

For transparent proxying, changing the routing seems like it needs pretty deep integration with the CNI or kube-proxy

Maybe a solution similar to kube-proxy's internalTrafficPolicy: Local can help workaround this issue?

arkodg commented 1 year ago

cc @moderation

youngnick commented 1 year ago

For transparent proxying, changing the routing seems like it needs pretty deep integration with the CNI or kube-proxy

Maybe a solution similar to kube-proxy's internalTrafficPolicy: Local can help workaround this issue?

Sadly, it's not about the inbound traffic, transparent mode will get the packets to the backend properly. It's about the return path from the backend back to the client - because Envoy doesn't do any address translation, the packets will be sent straight from the backend to the client, without going back through Envoy. To change that behavior, you'd need to mess with the routing tables for the pod, basically making Envoy the default gateway for all IP traffic in the pod (this is how traditional load balancers like F5 work, they're almost always the default gateway for the segment where the backends reside).

moderation commented 1 year ago

@youngnick a couple of questions:

  1. NGINX in theory supports UDP ingress today irrespective of Envoy Gateway - do they have a feature or implementation that differs from Envoy that enables them to provide this UDP ingress solution?
  2. Ambassador Labs Emissary Ingress which is based on Envoy supports HTTP/3 - https://blog.getambassador.io/how-to-implement-http3-support-with-ambassador-edge-stack-c57bae385a88. HTTP/3 rides on QUIC which is UDP based. What are they doing that is out of reach for Envoy Gateway supporting UDP?
  3. Should the documentation of the Kubernetes Gateway API specification modify their documentation to indicate that of the reverse proxies (NGINX, HAProxy, Traefik, Caddy etc.), Envoy is unable to support UDPRoute?
youngnick commented 1 year ago

@youngnick a couple of questions:

  1. NGINX in theory supports UDP ingress today irrespective of Envoy Gateway - do they have a feature or implementation that differs from Envoy that enables them to provide this UDP ingress solution

No, from what I can see, their UDP implementation will have the same client address limitation. I seem to recall looking at a version of their documentation that called this out, but I can't see it in their current one.

In addition, the fact that their Service Mesh product uses eBPF to inject PROXY protocol headers in order to make this work tends to bear this out, for me.

  1. Ambassador Labs Emissary Ingress which is based on Envoy supports HTTP/3. HTTP/3 rides on QUIC which is UDP based. What are they doing that is out of reach for Envoy Gateway supporting UDP?

QUIC is different, because QUIC uses UDP as a transport, but has higher-layer constructs to allow the backend to rebuild the actual client address (like X-Forwarded-For in HTTP/1 and HTTP/2). So it doesn't matter in those cases if the actual client IP is lost along the way.

Most other UDP protocols don't have this, sadly.

  1. Should the documentation of the Kubernetes Gateway API specification modify their documentation to indicate that of the reverse proxies, Envoy is unable to support UDPRoute?

It's not really a matter for Gateway API - Gateway API defines what a UDPRoute is - the fact that Envoy running as an in-cluster proxy can't do it isn't really relevant there.

I would love to have a way to make this work the way we all want it to, but trying to proxy UDP with cluster networking really is the worst-case scenario, sadly.

zhaohuabing commented 1 year ago

It seems like we don't have a workable solution for trasparent proxying for both UDP and TCP in the short term. So I'll go with the non-trasparent mode and add some documentation on that to make it clear to the users. Does this sound good?

BTY, if people really relies on the real source address in an application protocol based on UDP and want to deploy their applications in cloud infrastructure, they may need to switch to an application-level solution to get the source address because there're good chances that there're other non-transparent hops between the client and the server.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

arkodg commented 1 year ago

lets keep this issue open until end to end functionality is shown in the docs

danehans commented 1 year ago

@zhaohuabing do you mind adding a user doc to close out this issue?

zhaohuabing commented 1 year ago

@danehans Sorry for the delay. My hands were tide for the last few weeks. I'll add the doc this week and close this issue.