Dynamic forward proxy support STRICT_DNS?

[mirwide]

Dynamic forward proxy now resolve only one ip address from dns. It's limit case for usage. Can possible use STRICT_DNS resolve policy in dynamic forward proxy now or plan support feature?

[yanavlasov]

@alyssawilk do you know of any plans?

[mattklein123]

This would be a feature request. @mirwide could you explain a bit more about your use case and how you would want this to work exactly? Presumably you would want to all specify the LB policy?

[mirwide]

I try use forward proxy to inter service communication for DNS only discovery service. For example:

1. Service have multiple DNS A records, it's dynamically change, not use sidecar proxy in services and no other source for service information.
2. All internal requests send to envoy.
3. Envoy rewrite header for route and use dynamic forward proxy for get cluster. It's work, but not production ready, because only one ip resolve same time. Need outlier detection, eventual consistency for endpoints, retry to next endpoint. Current workaround - static config for clusters. For other case, it's only plan use dynamic forward proxy to simplify control plane. Need additional lb policy and subset balancing.

Currently, dynamic forward proxy may be use only forward requests to other load balancer. Expectation, all cluster feature allow in dynamic forward proxy. But may be this not goal.

[mattklein123]

I think it's definitely doable to have the dynamic proxy cluster look a bit more like a normal cluster, but this will require a bunch of thought to avoid duplicating a lot of code. I wonder if we could make it optionally look more like the aggregate cluster where individual hosts are added to a sub-cluster with all of the normal configuration that a sub-cluster would have. cc @snowp for thoughts.

[snowp]

I think delegating to an inner cluster would work fine. It would be limited to one locality/priority since we're sourcing all the endpoints from a single DNS name so there'd be no clear way to specify the LB hierarchy, if that's sufficient then this shouldn't be too hard (famous last words).

[snowp]

One of the possible issues is stats: we ran into this with the aggregate cluster where people were expecting to see stats for the aggregation (e.g. number of hosts = sum of hosts in all the clusters). We should take some care to make sure that we're handling this in an intuitive way if we end up with some kind of delegation to another cluster.

[snowp]

Giving this some more thought (as it might be something that we have a use case for): one potentially very powerful way of doing this would be to facilitate the forward_proxy to create arbitrary clusters based on the requested DNS names. I can imagine configuration that looks something like this:

cluster_template:
  lb_policty: STRICT_DNS
  health_checks: ...
  circuit_breakres: ...

which when specified would make the the forward_proxy mechanism change from resolving a single endpoint to dynamically creating a cluster based on the template. This would then result in a cluster created using the template, placing the inbound address/port in the load_assingment section of that cluster.

This would skip the forward_proxy DNS resolution completely and instead rely on cluster warming to continue any filters waiting for this to be ready. Once established, no re-resolution would be necessary beyond what the cluster itself is doing through STRICT_DNS/etc.

WDYT @mattklein123? I'm not sure what the current state is for dynamically creating clusters like this, iirc there has been some attempts at doing something similar in the past.

[mattklein123]

WDYT @mattklein123? I'm not sure what the current state is for dynamically creating clusters like this, iirc there has been some attempts at doing something similar in the past.

This would work, yes, and I think could be pretty clean. There is no upstream ready made filter for doing this, but the plumbing has been built with regard to thread local cluster add/remove callbacks, etc. It would need to work in a very similar way where it goes to the main thread, pauses, then comes back, etc.

[ramaraochavali]

FWIW, we have attempted this some time back on creating dynamic clusters on the request path. https://github.com/envoyproxy/envoy/pull/3479 - @mattklein123 had some very good design inputs there

[m1o1]

I was looking for this as well. My use case is that Kubernetes currently load balances TCP connections, not requests, so persistent connections between services meant not all replicas were handling requests evenly.

I used Envoy with STRICT_DNS following this guide which fixed the balancing issue. We would prefer not to have an Envoy proxy sitting in front of every service if we don't need to (we think we could get away with a DaemonSet of Envoy proxies), but this would require us to use a dynamic forward proxy (if my understanding is correct). But without some of the nifty options in the other proxy config (like STRICT_DNS and LEAST_REQUEST), this doesn't seem doable.

Unless envoy provides some way to route based on the path (with capture groups so the upstream host could just be \1 or something), but the docs say "Regex/slug matching is not currently supported, mainly because it makes it difficult/impossible to programmatically determine whether routing rules conflict with each other".

[Sooryaa-A]

@mattklein123 ,any plan for this feature implementation .We too would like to have this feature for our use cases. Ticket was raised for the same. https://github.com/envoyproxy/envoy/issues/16426

[doujiang24]

/assign @doujiang24

I would like to have it a try, we plan to use this extension in our production. Create clusters dynamically, with a hard-code cluster_template: round-robin load-balancer and no health check, it could be a good start.

WDYT @mattklein123 @snowp ? Thanks.

[mattklein123]

Yes I still think this will be useful, though it will be a lot of work. You might consider doing a small design doc first.

[doujiang24]

Thanks @mattklein123 , here is a small design doc, suggestions are welcome 🙏 https://docs.google.com/document/d/12S9c1ZPLFVkGNM0xaP2mtNDwNjwxG0eUvg5WhUuNZ98/edit?usp=sharing

[mattklein123]

At a high level the proposal is fine, but there aren't many implementation details. This is going to be difficult to implement so let's see how it goes if you want to give it a shot.

[doujiang24]

Thanks @mattklein123 , I have implemented a draft PR in https://github.com/envoyproxy/envoy/pull/26420 Could you please take a quick look when you have time? to make sure it's in the proper direction, thanks very much!

[Gollam]

@mattklein123 @doujiang24 this is now part of 1.27, right?

[doujiang24]

@Gollam yep, I do think so. This is the commit https://github.com/envoyproxy/envoy/commit/0a41170f0eec59f284e658b689eeabe83f20a48b, which was already included in 1.27.0.