Support multiple ADS configuration to allow distinct control-planes to serve parts of the configuration

[valerian-roche]

Support multiple ADS configuration to allow distinct control-planes to serve parts of the configuration

Description: We do use distinct control-planes to provide distinct parts of the xDS configuration used by envoy (currently mostly to split EDS and RtDS from the main LDS/RDS/CDS). As envoy currently only supports a single ADS configuration, we do configure each upstream cluster with a specific EDS configuration. As a consequence we end up with hundreds (sometimes more than a thousand) streams from a single envoy instance to our control-plane, duplicating parts such as extensions, metadata and more. While we managed to scale our control-planes through optimizations or dirty workarounds (e.g. redefining the DiscoveryRequest protobuf to drop unused fields that ended up taking a lot of CPU to deserialize and remain unused), this is very inefficient and leads to increase resource usage and harder to maintain configuration. It also makes it unrealistic to use LEDS for instance. The solution provided by https://github.com/envoyproxy/envoy/issues/2943 and its associated PRs looked promising, but it ends up confronted to a few issues:

• EDS requires keeping a local cache (addressed through https://github.com/envoyproxy/envoy/issues/26749 to ADS muxes)
• the new factory model would make this implementation more invasive
• in my opinion, this hash-based choice can be confusing to users, with multiplexing of watches occurring based on a hash of a sub-part of the resources.
• opting it all by default is risky, as some control-planes might not support it, and it changes the behavior of running instances.

A distinct proposal (associated draft PR coming) is to allow defining multiple ADS instances in the bootstrap configuration, and extending the ADS config_source to allow providing a specific ADS instance to use. The draft PR shows a surprisingly limited code surface impacted, while also unlocking more features like LEDS support, native reuse of the eds cache and potentially more (e.g. for ECDS). This is also in my opinion aligned with the grpc xDS configuration direction and could be reused directly through xdstp using the authority as the ADS instance to use. As a further advantage, this is an opt-in change for users, and control-planes supporting ADS for EDS will natively support this.

Could you indicate if the draft PR could be considered? I may have missed major blockers or concerns with such an approach

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[soulxu]

cc @adisuissa

[adisuissa]

Thanks for raising this issue! It is an important feature and we'd like to see the support there. There's been ongoing design and preliminary work in the domain of supporting multiple xDS servers. This is called xDS-Federation. To give a bit of a background, please see the xDS-TP doc. It is likely that we'd need to redesign the xDS layer in Envoy (see #24773) to support this use-case.

Feel free to reach out directly, and we can discuss how we can proceed with this work.

[markdroth]

I think this is a duplicate of #13951, which maps out in detail the changes we need to support this in Envoy.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.