API to send/receive app-specific XDS configs/messages

[costinm]

APIs to interact with the XDS layer would be extremely useful for mobile - in particular a method to allow the native app to make an XDS request with an app-specified type, and to get back the opaque response from the server and ack/nack.

For mobile it is better to reduce the number of active TCP connections ( and associated keep-alives ), such API would allow the XDS server to also push app-specific configs, or configs for filters that are implemented using the app extension APIs. If this is combined with adding an opaque payload to the XDS request, the XDS connection can be used to multiplex a light push messaging mechanism, so apps can keep only one socket open.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[jpsim]

@abeyad as our resident Envoy Mobile xDS expert, can you please help me triage this issue? Is it still useful to keep open?

[abeyad]

APIs to interact with the XDS layer would be extremely useful for mobile

We are actually working on this right now - @caschoener is leading development on it. But I think I'm not clear on the "xDS layer" referenced, see next comment ...

in particular a method to allow the native app to make an XDS request with an app-specified type, and to get back the opaque response from the server and ack/nack

What is an app-specified type? All xDS in Envoy Mobile (which is the same for Envoy) use the xDS protocol types described in https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol.html#resource-types. I'm not aware of a mechanism to define app-specific type, and I'm not clear on what the use case would be?

For mobile it is better to reduce the number of active TCP connections ( and associated keep-alives ), such API would allow the XDS server to also push app-specific configs, or configs for filters that are implemented using the app extension APIs.

Not sure I follow. Right now, there are two xDS protocol variants: REST and gRPC. REST is a polling based mechanism, so it won't keep the connection open long-lived. gRPC will keep a long-lived bidirectional stream open so the xDS server can send the client updates to resources it is subscribed to whenever they become available. This comes at the expense of keeping a long-lived gRPC stream open. Mobile devices might prefer the polling-based mechanism as long as the polling interval is not too short (in which case, the device may be continuously opening and closing connections).

If this is combined with adding an opaque payload to the XDS request, the XDS connection can be used to multiplex a light push messaging mechanism, so apps can keep only one socket open.

Using the gRPC protocol with ADS, a device should only ever have one connection open dedicated to xDS, and the xDS server would keep pushing updates along that gRPC stream as they become available.

[jpsim]

So @abeyad is it fair to say that our xDS integration efforts are already tracked by other tickets and this can be closed because whatever this is suggesting is a bit out of date compared to our current xDS goals?

[costinm]

In Istio we use the XDS protocol to configure proxyless gRPC as well as Istio agent and for the debug configs ( from istioctl to istio-agent and istiod ). Any proto ( as well as json !) can be sent as resource value.

Regarding push vs poll: it's more complicated. Apps can't hold a connection open on android - unless they are in foreground. And they can't poll either if they're not active - it needs to be initiated by a push message. If they are in foreground - usually it is best to not have too many long-lived connections active, and certainly not ideal to make polling requests, all RTTs and bytes cost money and battery. If the app is using envoy and XDS streaming - exposing the XDS protocol with an API and allowing the app to request (subscribe) to specific resource types and get them would reuse the XDS connection.

A smart xDS server for mobile would send not only Envoy config - but also app config and possibly messages.

[costinm]

And it's fine to close this ticket - there are other options and protocols for a mobile app to share a TLS connection and send/receive configs and messages.

[abeyad]

Sorry JP & Costin for the late reply, just seeing this. Thanks for the clarification @costinm !

In Istio we use the XDS protocol to configure proxyless gRPC as well as Istio agent and for the debug configs ( from istioctl to istio-agent and istiod ). Any proto ( as well as json !) can be sent as resource value.

Thanks for sharing, wasn't aware of that! What do you specify for the resource type url when sending DiscoveryRequests?

If the app is using envoy and XDS streaming - exposing the XDS protocol with an API and allowing the app to request (subscribe) to specific resource types and get them would reuse the XDS connection.

Do you mean creating a way for an app to make an API call directly to Envoy Mobile that would initiate an xDS request/response chain? I assume then that you mean xDS would be used to just send opaque bytes that the application is responsible for understanding and using, is that correct?

A smart xDS server for mobile would send not only Envoy config - but also app config and possibly messages.

I think this speaks again to the desire to have xDS as a general communication mechanism to receive a stream of byes from an xDS server, is this correct? If so, I'm not aware of any prior discussions on this and I'm not sure that's the direction xDS wants to go in. It seems like we should either (1) define new resource types to handle more use cases or (2) use a different mechanism (like a data plane request) to fetch additional opaque "config" for the app?

cc @adisuissa @alyssawilk in case they have different thoughts.

[alyssawilk]

I like the idea of xDS for app configuration, but I suspect it would literally be snagging an any proto / typed config and handing it up blindly. I thought there was a planned extension point for Envoy for generic data that core Enovy didn't know about but custom Envoy would. cc @envoyproxy/api-shepherds who can confirm or deny :-)

[adisuissa]

We are currently working on making the xDS interaction more accessible for any kind of resource. The idea is to keep the the pub-sub nature of xDS when subscribing to any type of resource, while abstracting over the underlying connection and message handling. I hope that we'll start seeing PRs in this direction in the upcoming weeks.

[abeyad]

Thanks for the feedback @alyssawilk and @adisuissa !

@jpsim I think we should keep this issue open and move it over to the Envoy repo, and it can be tracked against the aforementioned work Adi indicated might commence in a few weeks.

[jpsim]

Thanks for the dialogue here. I'll keep this issue open and once we're done triaging all the tickets in this repo, we can move them all over to the Envoy one in a consistent way.

[costinm]

Sorry JP & Costin for the late reply, just seeing this. Thanks for the clarification @costinm !

In Istio we use the XDS protocol to configure proxyless gRPC as well as Istio agent and for the debug configs ( from istioctl to istio-agent and istiod ). Any proto ( as well as json !) can be sent as resource value.

Thanks for sharing, wasn't aware of that! What do you specify for the resource type url when sending DiscoveryRequests?

We have a lot of types defined - XdsResourceGenerator is the interface. For example debug info is using "istio.io/debug"

If the app is using envoy and XDS streaming - exposing the XDS protocol with an API and allowing the app to request (subscribe) to specific resource types and get them would reuse the XDS connection.

Do you mean creating a way for an app to make an API call directly to Envoy Mobile that would initiate an xDS request/response chain? I assume then that you mean xDS would be used to just send opaque bytes that the application is responsible for understanding and using, is that correct?

Pretty much. API would make a call like "subscribe(myResourceType)" to EnvoyMobile, which forwards over existing xDS ( and deals with reconnects, etc).

On server side some 'publish' interface exposed by the XDS control plane.

Very similar to Webpush or GCM (cloud messaging) - but using envoy persistent connection and without going trough a push provider ( with message limits, quotas, extra latency ).

A smart xDS server for mobile would send not only Envoy config - but also app config and possibly messages.

I think this speaks again to the desire to have xDS as a general communication mechanism to receive a stream of byes from an xDS server, is this correct? If so, I'm not aware of any prior discussions on this and I'm not sure that's the direction xDS wants to go in. It seems like we should either (1) define new resource types to handle more use cases or (2) use a different mechanism (like a data plane request) to fetch additional opaque "config" for the app?

xDS doesn't define a strict list of resource types that are allowed - so far using Istio-specific types has worked pretty well. Not sure what direction they want to go - but I doubt they can stop the use of custom resource types any more than HTTP or mqtt protocols maintainers can control what goes in a POST or message.

Even in context of envoy - with WASM/Lua it seems quite reasonable to allow custom config types for the custom code the user would write. And if Envoy is embedded into an app - why would the app have to use a different config protocol - in addition to XDS, in particular on mobile where maintaining connections costs battery and data plan bytes ?

I should mention in Istio we have an XDS proxy ( in istio-agent which starts envoy ) - which gets Istio specific XDS ( like root certs, etc) and forwards to envoy the stripped down protocol it expects. That would probably work on mobile as well without any change on your side - have the app maintain the XDS connection, and mobile envoy connect to localhost. That will also allow the H2 connection to be shared and used for other app-specific gRPC if it's to the same origin, avoiding extra TLS RTT.

[abeyad]

Thanks for the detailed explanations @costinm ! Agree with JP, this should be one of the issues we move over to the Envoy repo to enhance the Mobile APIs to enable this once support is built out in core Envoy as Adi stated will hopefully start happening over the coming weeks/months.