When would a connection persist through a filter chain change?

[omnipo]

Title: When would a connection persist through a filter chain change?

Description:

Hi Envoy devs, apologies if this sounds an uninformed question. Doc [1] says that network filter changes will result in connection draining, and I do see that happen for network filter changes. But it seems that route changes via RDS in HTTP connection manager doesn't drain connections. So I'm wondering what is the logic under the hood: Are all http level config changes eligible to escape draining? Or is it because the protobuf message happen to be the same in the RDS case? So if ECDS is specified [2] would the connection also survive, even though ECDS may push new configs? Thanks!

Relevant Links:

[1] https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/listeners/network_filter_chain#filter-chain-only-update [2] https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-field-extensions-filters-network-http-connection-manager-v3-httpfilter-config-discovery

[mattklein123]

cc @lambdai @phlax agreed we can do better here on docs. This is all pretty complicated.

[lambdai]

What you mentioned regarding RDS and ECDS is correct. Network filter chain see the name (or protobuf message as you mentioned) of Route and Extension. The route or extension bind to that name is escaped from the change from network filterchain's view. That's the goal of RDS and ECDS(and also CDS), do not trigger the drain at network filter. Meanwhile, even though the L4 connection is not drained, the new http request within the surviving connection will see the new Route, Extensions, and Clusters.

I can add to the network filter that what RDS/ECDS/CDS pushed config doesn't cause connection drain.

[omnipo]

Thanks for the reply @lambdai . To clarify a bit, when you say the route or extension is bound to a name, that's only true for the "fine grained" changes from RDS/ECDS/CDS, right? If the filter updates come from e.g. LDS, there is no way to escape draining as the protobuf message of the whole chain is considered different?

[lambdai]

Sorry I don't get your question but let me clarify.

During the LDS update, a network filter chain in a listener config can only see a name in RDS response. See route_config_0 in the below config. https://github.com/envoyproxy/envoy/blob/main/test/config/integration/server_xds.lds.typed_struct.yaml#L20

For the only network filter chain in this example

1. if the name route_config_0 is updated to route_config_1, the connections owned by the old network filter chain will be drained, period.
2. If somehow the content of route_config_0(something like this) is changed due to RDS push, and then you push another listener config with the same listener config , your network filter chain will not be drained.

In short, when Envoy evaluate the differences between the old and new network filter chain config, Envoy could evaluate

diff("route_config_0", "route_config_1"),

Not the

diff(a full and huge route config with the name route_config_0, a full and huge route config with the name route_config_1)

[omnipo]

@lambdai Thanks for being patient with me. My question was to confirm that instead of pushing routes via RDS, if I push via LDS:

  listeners:
  - name: sample-listener
    address: ...
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP2
          drain_timeout: 5s
          stat_prefix: router
          route_config:
            name: route_config_0
            virtual_hosts: <data version 0>

, then push:

  listeners:
  - name: sample-listener
    address: ...
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP2
          drain_timeout: 5s
          stat_prefix: router
          route_config:
            name: route_config_0
            virtual_hosts: <data version 1>

Then connection will be drained since Envoy is doing a: diff(**old** full and huge route config with the name route_config_0, **new** full and huge route config with the name route_config_0)

And the only way to escape the draining is through RDS, also the similar goes for http filter vs. ECDS.

[lambdai]

Sorry for the late response:

if I push via LDS: , then push: Then connection will be drained since Envoy is doing a: And the only way to escape the draining is through RDS, also the similar goes for http filter vs. ECDS.

Yes. I guess that's probably the motivation that RDS was introduced.

[Eiji911]

If only http filter changes，before the old connection drains，can the new comming request of the old connections see the new http filter chain？