Provide a way to configure custom xds node metadata per dial target

[s-matyukevich]

Please see the FAQ in our main README.md before submitting your issue.

Use case(s) - what problem will this feature solve?

We have an xDS control plane and we want to slightly modify our xDS responses based on the information provided by the clients. For example, 2 different clients may want to connect to the same server, but one client need zonal failover, or outlier detection to be enabled, but the other client doesn't. If we use xDS we can't really configure this on the client directly - those are the settings the control plane needs to know while generating xds response.

One way we can pass this info to the control-plane is by using xds node metadata. We can configure xds metadata in the bootstrap file, but we can't use multiple bootstrap files, so a client can't provide different set of settings while connecting to different servers.

Proposed Solution

• Add grpc.WithXdsNodeMetadata(string) dial option.
• Propagate this information to the xds resolver builder via resolver.BuildOptions
• Pass this information to the xds client. We need it only while making the first LDS request.
• Modify xdsClient to use custom node metadata, when provided.

Alternatives Considered

• Build a custom UX interface + storage to allow users to configure different settings, which impact xds responses and connect management server to it. In fact we already have something like this for configuring xDS with envoy, and in theory we can make our users to use the same UX interface to configure grpc xDS. However, this is a very big change and we want to adopt xDS more incrementally. Another problem with this approach is that xDS support in grpc is way more limited that in envoy, so our users will have to configure some settings in the code and some settings in our management server interface, which is very confusing for them.
• Write a wrapper function around grpc.DialContext() and encode all parameters in the dial target string and decode them in the control plane. If there are no restrictions on the length of the target, this could work, but it feels very hacky.
• Provide a way to configure the whole bootstrap file from within code, looks like this was already discussed https://github.com/grpc/grpc-go/issues/4124

Additional Context

[zasweq]

cc @markdroth

[markdroth]

Note that all gRPC channels within the same binary will share the same XdsClient instance. That XdsClient creates an ADS stream to a given server the first time it is asked to subscribe to a resource on that server. If it then receives a second subscription for the same resource, it does not create a new ADS stream or send a new request; it simply serves the resource it has already obtained out of its cache. This means that for a given resource name, all channels subscribing to that resource will always get the same resource contents -- there is no way to give them separate contents for the same name, since that would break cacheability. This is an intentional structural decision necessary for CSDS to work; it's not just a lack of the ability to plumb the node metadata into the code separately for each channel.

If you need to do this for multiple channels in the same binary, I think the right way to do it is to have the server use different resource names for the different variants of the resource. So, for example, you can have the default version of the resource under the name myserver, and then you can have a variant of it with outlier detection disabled under the name myserver-no-outlier-detection, and any given channel can use whichever of those names it wants in the xds: URI used to create the channel.

If you need to do this for channels in different binaries, then you should be able to have a separate bootstrap config for each binary. In that case, it should be easy to use different contents in the bootstrap files.

I hope this info is helpful.

[s-matyukevich]

XdsClient creates an ADS stream to a given server the first time it is asked to subscribe to a resource on that server.

that is the part that I was missing. Thanks for explanation, it all make sense.