Document behavior for non-proxy (such as gRPC) server side xDS clients

[sanjaypujare]

Title: Document behavior for non-proxy (such as gRPC) server side xDS clients

Description: Non-proxy xDS clients (such as gRPC) may need to fetch inbound listener configurations for specific ports. We need to be able to specify that using existing APIs.

We have added support for non-proxy xDS clients to fetch only the specific API Listener resources they are interested in (https://github.com/envoyproxy/envoy/pull/8170). This is on the client (outbound) side where the xDS client then fetches the RouteConfiguration and other resources required by those Listener resources that allows the xDS client to fetch the whole client-side configuration tree.

We need similar support on the server side where a non-proxy (e.g. gRPC) server application needs to fetch inbound Listener configuration for a specific port the server is going to be listening on. The server will use the response to configure server side features such as security, rate limit, and network/socket parameters.

Currently Envoy uses the "wildcard" mode to request all Listeners that it needs to configure. However a non-proxy server application typically needs to fetch a single Listener for a specific port. We propose supporting this use case by specifying the port as the resource-names in the Listener Discovery request. For example if the port is 7000 then the resource-names will be ":7000". The xDS server will return a single Listener object that contains all the necessary configuration the non-proxy server application needs to configure the listener.

This proposal does not involve modifying any existing protos or code. If this proposal is accepted, we will enhance the documentation to describe this use-case in detail.

[Relevant Links:] Addition of HTTP API Listener for non-proxy clients on the client side

Envoy API flow involving all Listeners

[sanjaypujare]

CC @envoyproxy/api-shepherds @mattklein123 @htuch

[mattklein123]

IMO this sounds fine to me, though I would probably not call it a port and just say that an opaque string can be passed in resource-names to avoid wildcard processing? I think that would be more generic and serve the same purpose? @lizan any objections here?

[sanjaypujare]

IMO this sounds fine to me, though I would probably not call it a port and just say that an opaque string can be passed in resource-names to avoid wildcard processing? I think that would be more generic and serve the same purpose? @lizan any objections here?

Yes that would be ideal except we need to be able to differentiate between client side usage and server side usage by proxyless (aka gRPC) xDS clients.

For client side usage the gRPC client provides "host:port" as resource-names in the request. And we have the xDS server using that to return an ApiListener which is only usable on the client side (to get routing info).

For server side usage we propose using just ":port" because that is all we need for server side Listener configuration and of course there is no "host" to provide.

Note with this, there is now a big difference between "host:port" and ":port" usage in Listener DiscoveryRequest and we would like to formalize that. This is probably not as clean as some would like but we couldn't come up with a better way to achieve this. Using Node metadata fields seems even less clean.

[mattklein123]

I think my point though was that isn't this an implementation detail of the service you are providing? From the xDS perspective it's still just an opaque name?

[sanjaypujare]

The xDS client (proxyless gRPC) and the xDS server (some control plane like Istio Pilot) can come from different environments and in that case they need to have a common understanding of what "host:port" vs ":port" mean in the LDS resource-names field so I think this is not just an implementation detail. I am thinking the xDS spec here is the only place where we can formalize or standardize this understanding. Instead of making the control plane aware of proxy (Envoy) or proxyless clients through things like Node metadata or user-agent strings we think this is the cleanest way of standardizing without breaking any existing use-cases as far as I can see.

Because Listener is the root of the configuration tree, its resource-name has to be made available to xDS clients via some out-of-band mechanism, if that is possible. Otherwise we can say that "host:port" vs ":port" provides a hint to the control plane about whether an ApiListener is requested or a regular server-side Listener is requested if you think we should just stick to documenting an opaque string.

[mattklein123]

What if the server is listening on a unix domain socket? I just don't see why we would specify something like this as part of the protocol itself when an opaque string would work just fine.

[sanjaypujare]

That's a good point. Although I can find ways to support UDS etc I am not going to push this idea any more.

So at xDS level let's just keep it as an opaque string and I will open a PR to enhance the verbiage in the xDS API doc to reflect what we agreed on. This issue will be closed when that PR is merged.

[mattklein123]

+1 sgtm

[srini100]

/cc

[sanjaypujare]

closing the issue for now, pending some offline discussion.