Envoy 1.29

[jbohanon]

Gloo Edge Product

Open Source

Gloo Edge Version

1.17

Is your feature request related to a problem? Please describe.

I want Gloo Edge to use the latest stable Envoy version

Describe the solution you'd like

• [x] Walk through the Envoy release notes, understanding what has changed since 1.27 (https://www.envoyproxy.io/docs/envoy/v1.29.0/version_history/v1.29/v1.29.0)
• [x] Write up a document that outlines the changes (including field deprecations, new features, breaking changes) that are relevant to the Gloo Edge product. In this document, propose which of these changes we will incorporate in this feature request, and perhaps a set of features that we should consider supporting in the future. This MUST include all versions since the last version of envoy used in edge. This MUST include the risk of adopting the changes.
• [x] Schedule a review of this plan, inviting members of the Edge team, @solo-io/gloo-edge-leads , get buy in on the plan.
• [x] Execute the plan, following https://github.com/solo-io/gloo/issues/8514 and previous upgrades as inspiration
• [x] Identify additional tests that will improve the stability of our Envoy integration, or provide a rationale for why that is not necessary. These should be scoped to new features, feature changes or pieces of functionality that broke in manual testing.
• [ ] Work with @solo-io/solo-docs to update our support matrix and any other relevant user-facing docs that are impacted by this change

Describe alternatives you've considered

No response

Additional Context

Leapfrogs https://github.com/solo-io/gloo/issues/8979

┆Issue is synchronized with this Asana task by Unito

[nfuden]

Closed existing issue from the cost of doing business and updated this to be part of 1.17 cost of doing business.

[jbohanon]

Potential Problem

Incompatible behavior changes

• [x] eds: Introduced caching of EDS assignments when used with ADS. Prior to this change, Envoy required that EDS assignments were sent after an EDS cluster was updated. If no EDS assignment was received for the cluster, it ended up with an empty assignment. Following this change, after a cluster update, Envoy waits for an EDS assignment until initial_fetch_timeout times out, and will then apply the cached assignment and finish updating the warmed cluster. This change is disabled by default, and can be enabled by setting the runtime flag envoy.restart_features.use_eds_cache_for_ads to true.
  • disabled by default
• [x] http: Introduced a new runtime flag envoy.reloadable_features.no_downgrade_to_canonical_name to disable the name downgrading in the per filter config searching. See doc Http filter route specific config or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples.
• [x] http: Flip runtime flag envoy.reloadable_features.no_downgrade_to_canonical_name to true. Name downgrading in the per filter config searching will be disabled by default. This behavior can be temporarily reverted by setting the flag to false explicitly. See doc Http filter route specific config or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples.
  • I'm not entirely sure how to validate this without manually walking through each plugin's filter construction and making sure we're not using any per-filter configs with canonical names where we have filter definitions with non-canonical names. I would think our tests would catch this if we were not able to overwrite filter configs with more-specific.
• [x] jwt: Changed behavior of the JWT extraction, passing entire token for validation, instead cut him in the non-Base64 character. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.token_passed_entirely to false.
• [x] zone-aware routing: Zone-aware routing is now enabled even when the originating and upstream cluster have different numbers of zones. Previously, zone-aware routing was disabled in that case and the lb_zone_number_differs stat on the cluster was incremented. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.enable_zone_routing_different_zone_counts to false. Additionally, zone-aware routing now works correctly even when the originating and upstream cluster have different zone sets. Previously, zone-aware routing would not route fairly in this case. To revert the entire change, set the runtime flag envoy.reloadable_features.locality_routing_use_new_routing_logic to false to get the old behavior and well-tested codepaths, undoing both changes.
• [x] http: Remove the hop by hop TE header from downstream request headers. This change can be temporarily reverted by setting envoy.reloadable_features.sanitize_te to false.
• [x] http2: Discard the Host header if the :authority header was received to bring Envoy into compliance with https://www.rfc-editor.org/rfc/rfc9113#section-8.3.1 This behavioral change can be reverted by setting runtime flag envoy.reloadable_features.http2_discard_host_header to false.

Minor behavior changes

• [x] ext_authz: Removing any query parameter in the presence of repeated query parameter keys no longer drops the repeats.
• [x] ext_proc: Envoy will only take mode_override when waiting for the header responses. It will be ignored if it is in other processing states.
  • We have not yet supported mode_override at all
• [x] http: Abort the HTTP filter chain iteration when the stream is reset by resetStream() of filter callbacks. This behavior can be reverted by setting runtime flag envoy.reloadable_features.abort_filter_chain_on_stream_reset to false.
• [x] http: Change the proxy status for UpstreamRequestTimeout to HttpResponseTimeout. It can be disabled by the runtime guard envoy.reloadable_features.proxy_status_upstream_request_timeout.
• [x] outlier detection: A node must stay in healthy state for at least one period of check interval before ejection time multiplier is decremented.
• [x] outlier detection: Outlier detection will always respect max_ejection_percent now. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.check_mep_on_first_eject to false.
• [x] config parsing, http cache filter: Replaces Protobuf hashing by human-readable string with a dedicated deterministic hashing algorithm. The performance of the hash operation is improved by 2-10x depending on the structure of the message, which is expected to reduce config update time or startup time by 10-25%. The new algorithm is also used for http_cache_filter hashing, which will effectively cause a one-time cache flush on update for users with a persistent cache. To enable this behavior set envoy.restart_features.use_fast_protobuf_hash to true.

Bug fixes

• [x] healthcheck: The default behavior of unejecting outlier-detection-ejected host on successful active health checking can be disabled by setting outlier_detection.successful_active_health_check_uneject_host to false. This new configuration flag is a substitute for the removed runtime option envoy.reloadable_features_successful_active_health_check_uneject_host.
• [x] http: Close HTTP/2 and HTTP/3 connections that prematurely reset streams. The runtime key overload.premature_reset_min_stream_lifetime_seconds determines the interval where received stream reset is considered premature (with 1 second default). The runtime key overload.premature_reset_total_stream_count, with the default value of 500, determines the number of requests received from a connection before the check for premature resets is applied. The connection is disconnected if more than 50% of resets are premature, or if the number of suspect streams is already large enough to guarantee that more than 50% of the streams will be suspect upon reaching the total stream threshold (even if all the remaining streams are considered benign). Setting the runtime key envoy.restart_features.send_goaway_for_premature_rst_streams to false completely disables this check.
• [x] dynamic_forward_proxy: Fixed a bug where the preresolved hostnames specified in the Dynamic Forward Proxy cluster config would not use the normalized hostname as the DNS cache key, which is the same key used for retrieval. This caused cache misses on initial use, even though the host DNS entry was pre-resolved. The fix is guarded by runtime guard envoy.reloadable_features.normalize_host_for_preresolve_dfp_dns, which defaults to true.
• [x] ext_proc: Fixed content_length related issues when body mutation by external processor is enabled. The ext_proc filter removes the content length header in 1)STREAMED BodySendMode 2) BUFFERED_PARTIAL BodySendMode and 3) BUFFERED BodySendMode + SKIP HeaderSendMode. This will enable chunked-encoding whenever feasible in HTTP1.1. Besides, ext_proc filter keep content length header in BUFFERED BodySendMode + SEND HeaderSendMode. It is now external processor’s responsibility to set the content length correctly matched to the mutated body. if those two doesn’t match, the mutation will be rejected and local reply with error status will be returned.
• [x] jwt_authn: Fixed JWT extractor, which concatenated headers with a comma, resultig in invalid tokens.

Warrants Investigation

Incompatible behavior changes

• [ ] access_log: Added issuer in certificate_properties to the gRPC access log service(AlS).

Minor behavior changes

• [x] local_rate_limit: Added new configuration field always_consume_default_token_bucket to allow for setting if default token bucket should be always consumed or only be consumed when there is no matching descriptor.
• [ ] quic: Enable QUICHE request and response headers validation. This behavior can be reverted by setting runtime flag envoy.reloadable_features.FLAGS_envoy_quic_reloadable_flag_quic_act_upon_invalid_header to false.
• [x] header_mutation: If per route configuration is configured at multiple levels (route, virtual host, and route table), all specified levels’ mutations are applied. Default order is from least to most specific level (i.e. most specific level wins).
• [x] local_rate_limit: Added new configuration field rate_limited_as_resource_exhausted to allow for setting if rate limit gRPC response should be RESOURCE_EXHAUSTED instead of the default UNAVAILABLE.

Bug fixes

• tracing: Added support for configuring resource detectors on the OpenTelemetry tracer.

New Features (unfiltered)

• access_log: Added %RESPONSE_FLAGS_LONG% substitution string, that will output a pascal case string representing the response flags. The output response flags will correspond with %RESPONSE_FLAGS%, only with a long textual string representation.
• access_log: Added a field lookup to %FILTER_STATE% for objects that have reflection enabled.
• access_logs: Added json_format_options config option to support JSON output formatting and the sort_properties option to print the JSON output with sorted properties.
• admin: Added a new skip_exit query parameter to /drain_listeners to skip exiting after the drain period.
• admin_logging: added support for glob control of fine-grain loggers in admin /logging interface.
• compression: Added remove_accept_encoding_header for per-route configuration of this value.
• config: Added the capability to defer broadcasting of certain cluster (CDS, EDS) to worker threads from the main thread. This optimization can save significant amount of memory in cases where there are (1) a large number of workers and (2) a large amount of config, most of which is unused. This capability is guarded by enable_deferred_cluster_creation.
• ext_proc: Added disable_immediate_response config API to ignore the immediate_response message from the external processing server.
• extension_discovery_service: Added ECDS support for downstream network filters.
• extension_discovery_service: Added ECDS support for upstream network filters.
• extension_discovery_service: Added metric listener.listener_stat.network_extension_config_missing to track closed connections due to missing config.
• filters: Added filters to update the filter state for the HTTP requests and the TCP connections.
• geoip: Added support for Maxmind geolocation provider.
• http: Added Json-To-Metadata filter.
• http: Added disabled flag to disable a filter in the filter chain by default. The filter can be enabled explicitly by setting valid per filter config in the route configuration.
• listener: Added possibility to track global downstream connection limit via downstream connections monitor in overload manager.
• lua: Added downstreamRemoteAddress() method to the Stream info object API.
• oauth2: Added use_refresh_token to support updating an access token via a refresh token if that is provided by authorization server.
• original_dst: added support for the internal listener address recovery using the original destination listener filter.
• otlp_stats_sink: Added stats prefix option to OTLP stats sink that enables adding a static prefix to all stats flushed by this sink.
• quic: Added support for QUIC listener filters with ECDS support reusing the same config API listener_filters as TCP does.
• redis: Added new configuration field read_command_policy to specify Envoy should route read commands to another cluster.
• redis: Added support for lmove command.
• redis: Added support for time command (returns a local response).
• router: Added metadata support for virtual host and route configuration.
• tap: Added custom_sink type to enable writing tap data out to a custom sink extension.
• tap: Added record_downstream_connection to control writing downstream connection address info in trace output.
• tap: Added record_headers_received_time to control writing request and response headers received time in trace output.
• tcp: Added the support to detect and send TCP RST for raw buffer socket based connections. This is currently supported on Linux only. It can be disabled by the runtime guard envoy_reloadable_features_detect_and_raise_rst_tcp_connection.
• tls: Added disable_stateful_session_resumption config option to disable stateful TLS session resumption.
• tls: Added fallback fallback to support private key provider to fallback to boringssl TLS handshake. If the private key provider isn’t available (eg. the required hardware capability doesn’t existed), Envoy will fallback to the BoringSSL default implementation when the fallback is true. The default value is false.
• tracing: Added spawn_upstream_span to control whether to create separate upstream span for upstream request.
• tracing: Added support for exporting spans via HTTP on the OpenTelemetry tracer.
• udp_proxy: Added http_capsule UDP session filter that can be used to encapsule or decapsulate UDP datagrams in HTTP, when used for UDP tunneling.
• udp_proxy: Added session_filters config to support optional filters that will run for each upstream UDP session. More information can be found in the UDP proxy documentation.
• udp_proxy: Added injectDatagramToFilterChain() callback to UDP session filters that allows session filters to inject datagrams downstream or upstream the filter chain during a filter chain iteration. This can be used, for example, by session filters that are required to buffer datagrams due to an asynchronous call.
• udp_proxy: added dynamic_forward_proxy UDP session filter that can be used to have dynamic forward proxy UDP flows, when used in conjunction with another session filter that sets required filter state values.
• upstream: Added allow_redundant_keys to suppport redundant keys in request metadata for subset load balancing.
• upstream: Added the ability to specify a custom upstream local address selector using local_address_selector.
• zookeeper: Added support for emitting per opcode request bytes metrics via enable_per_opcode_request_bytes_metrics. added support for emitting per opcode response bytes metrics via enable_per_opcode_response_bytes_metrics.
• CEL-attributes: Added attribute connection.transport_failure_reason for looking up connection transport failure reason.
• access_log: Added new access log command operator %EMIT_TIME% to get the time when the log entry is emitted.
• access_log: Added support for %CONNECTION_ID% command operator for UDP session access log.
• access_log: Added support for listener metadata in %METADATA% formatter.
• attributes: Added support for listener metadata and listener direction in xDS attributes.
• attributes: Added support for node data in %CEL% formatter.
• aws: Added support for AWS common utility to fetch metadata credentials from AWS STS by using WebIdentityToken. To enable you need to set envoy.reloadable_features.use_http_client_to_fetch_aws_credentials to true so that web identity credentials provider can use http async client to fetch credentials. Web identity credentials provider cannot use current default libcurl credentials fetcher which is under deprecation and will soon be removed. Web identity credentials provider is not compatible with Grpc Credentials AWS IAM plugin which can only support deprecated libcurl credentials fetcher, see https://github.com/envoyproxy/envoy/pull/30626.
• aws_request_signing: Added support for specifying the aws signing algorithm, either AWS_SIGV4 or AWS_SIGV4A via signing_algorithm config API.
• composite filter: Added ExtensionConfiguration discovery service support for composite filter.
• composite filter: Added composite filter’s matched action name to filter state with key envoy.extensions.filters.http.composite.matched_actions.
• compression: Added qatzip compressor.
• ext_authz: Forward filter_metadata selected by route_metadata_context_namespaces and typed_filter_metadata selected by route_typed_metadata_context_namespaces from the metadata of the selected route to external auth service. This metadata propagation is independent from the dynamic metadata from connection and request.
• ext_authz: New config parameter charge_cluster_response_stats for not incrementing cluster statistics on ext_authz response. Default true, no behavior change.
• ext_authz_filter: Added with_request_body to optionally override the default behavior of sending the request body to the authorization server from the per-route filter.
• filter state: Added config name of filter sending a local reply in filter state with key envoy.filters.network.http_connection_manager.local_reply_owner. See the well-known filter state keys for more detail.
• filters: Added the Basic Auth filter, which can be used to authenticate user credentials in the HTTP Authentication heaer defined in RFC7617.
• grpc async client: Added max_cached_entry_idle_duration to control the cached gRPC client eviction time in the cache.
• grpc_http_bridge: Added ignore_query_parameters option for automatically stripping query parameters in request URL path.
• internal_redirects: Added support to copy headers from the redirect response to the triggered request. See response_headers_to_copy.
• jwt: The JWT authentication filter supports changing the routes when either the headers or the dynamic metadata are modified.
• jwt: The JWT filter can now serialize non-primitive custom claims when maping claims to headers. These claims will be serialized as JSON and encoded as Base64.
• lua: Added Lua extension of router cluster specifier plugin to support selecting cluster dynamically by Lua code.
• ratelimit: Ratelimit supports optional additional prefix to use when emitting statistics with stat_prefix configuration flag.
• ratelimit: Ratelimit supports setting the HTTP status that is returned to the client when the ratelimit server returns an error or cannot be reached with status_on_error configuration flag.
• rbac filter: allow listed HttpAttributesCelMatchInput to be used with the xDS matcher in the RBAC filter.
• redis: Added support for the getdel command.
• redis: Added support for the watch command (aborts multi transactions if watched keys change).
• set_metadata: Added support for injecting typed and untyped dynamic metadata with this filter, also adds the ability to add multiple namespaces with one filter and config to overwrite existing metadata is opt-in. untyped_metadata may now be used to configure the set_metadata filter.
• stateful_session: Added strict mode to cookie and header based stateful session. If a destination encoded in cookie or in specified header respectively is not available, Envoy will return 503 instead of selecting another destination from the cluster.
• stats: Added per_endpoint_stats to get some metrics for each endpoint in a cluster.
• stream info: Added time spent reading request headers to DownstreamTiming.
• tcp_proxy: Added support to TCP Proxy for recording the latency in UpstreamTiming from when the first initial connection to the upstream cluster was attempted to when either the connection was successfully established or the filiter failed to initialize any connection to the upstream.
• tracing: Added support for configuring resource detectors on the OpenTelemetry tracer.
• tracing: Added support to configure a Dynatrace resource detector for the OpenTelemetry tracer.
• tracing: Added support to configure a sampler for the OpenTelemetry tracer.
• tracing: Provide initial span attributes to a sampler used in the OpenTelemetry tracer.
• udp_proxy: Add access log options to allow recording an access log entry periodically for the UDP session, and allow recording an access log entry on the connection tunnel created successfully to upstream when UDP tunneling is configured.
• udp_proxy: Added support for propagating the response headers in UdpTunnelingConfig and response trailers in UdpTunnelingConfig to the downstream info filter state. upstream: Implmented API drop_overloads which can be used to drop certain percentage of traffic from Envoy.
• zookeeper: Added support for emitting per opcode decoder error metrics via enable_per_opcode_decoder_error_metrics.

[jbohanon]

Comparing the commit our current go-control-plane-fork-v2 forked from to the commit synced with upstream v1.29.2 go-control-plane changes https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890

• [x] //go:build !disable_pgv
• [x] base comments https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-1f928c81e32dfec284a89c2b87a8d5d42ada14cdef6c507b2bc846868061d147
• [x] route comments https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-cd3763c35a2226e9c477a4132a14ae87cbc0208e7c2783d200a3b18857e69a97
• [x] access_log Issuer https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-1c480249be07a4e871e54b342bc288a4cb88e3171b0a434b445c59150d20d91e
  • I can't find anywhere that we are using this API
• [x] set_filter_state https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-d0c18dad61e2ab52366051a8fcf04fa227929067092fa4363f3d1a0484bab635
  • we do not configure this filter
• [x] AWS request signing SigningAlgorithm new feature https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-06d914dd7ecbf03169961e73f6fbddcd7ed2c05c6797f164fc159f2a8d854b1b
• [x] Header mutation MostSpecificHeaderMutationWins new feature https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-2a2a937082a3783d833c80b7e81aa9f807f2820580377a3254a9364d623d597a
• [x] OAuth DefaultExpiresIn new feature https://github.com/envoyproxy/go-control-plane/compare/989e83d4a05c74448fdc72e5a67df5529387c021...5401a878d8bb6ef0ee8ae8df2f916aa1faaa1890#diff-0b816a0390afae8ad9a3fc7c1a3e131aa7ed89f4162fab5cbf02edc3b8548d9b

[ben-taussig-solo]

@jbohanon @ashishb-solo and I met to discuss the changes listed in Jacob's comment above (see here)

Here are our notes -- they are pretty messy, but the general conclusion is that we need to look further into the envoy.reloadable_features.no_downgrade_to_canonical_name change and the header mutation change. We determined that we are either unaffected by other changes, or that those changes should be communicated to end users and field engineers

Meeting Notes

Solo issues -- to look into

• no_downgrade_to_canonical name is difficult to verify
  • look at routeOptions and virtualHostOptions
  • is there some sort of find/grep we could use to
  • conclusion: this is a hard one
  • option: write issue to punt, flip the runtime flag
  • look into setRoutePerFilterConfig/ setVhostPerFilterConfig
    • header_mutation change
    • now all levels (route/vh/rt) of config are applied, from least to most specific
    • where do we configure header mutations? can we configure gloo edge to be affected by this?
    • we need to call this out in the upgrade doc

      End-user issues

• jwt filter
  • breaking change, presumably passed through from solo filter to enterprise filter
  • solution: call out the breaking change -- will possibly affect end users
• Transfer encoding header
  • would be a strange use case for this to affect a user, but it's possible
  • solution: call this out on release/to field engineers
    • discussion: should we give users the runtime flag if this is a problem?
• Http2 host header
  • similar to transfer encoding header
  • solution: call this out on release/to field engineers
    • discussion: should we give users the runtime flag if this is a problem?
• health check uneject behavior
  • solution: call out to end users/field eng
  • Update: per Jacob's comment below, this does not need to be called out
• verify ext_proc content length issue in kube2e ext_proc tests
  • this issue was widely known before we released ext_proc functionality
  • we recommended ext_proc users not to use bodies for this reason

    Field eng notes

• http2 reset fix
  • pretty niche -- we don't think this will affect users
  • would be worth noting to field engineers -- could in theory affect performance

Unaffected

• access_log certificate_properties
  • doesn't seem like we use this
  • not related to the access log service/access logger filter -- it's something else
  • TODO: Ben -- verify
• zone aware routing
  • can't be configured in gloo edge (could reference slack convo)
  • solution: doesn't affect us
• dfp fix
• jwt extractor header comma fix
• new local rate limit field
• quic/QUICHE req/res header validation
• new local_rate_limit filed
• OTEL resource detectors -- seems like a new feature

[jbohanon]

No downgrade to canonical name

This change will be considered unsafe for us if there are any places where we are using custom filter names which might not align with one or more overrides in TypedPerFilterConfig.

To start, I have checked that we are always setting the TypedPerFilterConfig with the canonical name as the key. I validated this by grepping through gloo and solo-projects for calls into pluginutils[.].*PerFilterConfig[(] which covers all of these helper functions. solo-projects:

grep -rE "pluginutils[.].*PerFilterConfig[(]" ./
./projects/gloo/pkg/plugins/graphql/plugin.go:  return pluginutils.SetRoutePerFilterConfig(out, FilterName, routeConf)
./projects/gloo/pkg/plugins/dlp/plugin.go:              pluginutils.SetVhostPerFilterConfig(out, FilterName, dlpConfig)
./projects/gloo/pkg/plugins/dlp/plugin.go:              pluginutils.SetRoutePerFilterConfig(out, FilterName, dlpConfig)
./projects/gloo/pkg/plugins/extauth/sanitize_filter.go: return pluginutils.SetVhostPerFilterConfig(out, SanitizeFilterName, customAuthConfig)
./projects/gloo/pkg/plugins/extauth/sanitize_filter.go: return pluginutils.SetWeightedClusterPerFilterConfig(out, SanitizeFilterName, customAuthConfig)
./projects/gloo/pkg/plugins/extauth/sanitize_filter.go: return pluginutils.SetRoutePerFilterConfig(out, SanitizeFilterName, customAuthConfig)
./projects/gloo/pkg/plugins/extauth/plugin.go:  return pluginutils.SetVhostPerFilterConfig(out, wellknown.HTTPExternalAuthorization, extAuthPerRouteConfig)
./projects/gloo/pkg/plugins/extauth/plugin.go:  return pluginutils.SetRoutePerFilterConfig(out, wellknown.HTTPExternalAuthorization, extAuthPerRouteConfig)
./projects/gloo/pkg/plugins/extauth/plugin.go:  return pluginutils.SetWeightedClusterPerFilterConfig(out, wellknown.HTTPExternalAuthorization, extAuthPerRouteConfig)
./projects/gloo/pkg/plugins/rbac/plugin.go:             pluginutils.SetVhostPerFilterConfig(out, FilterName, perRouteRbac)
./projects/gloo/pkg/plugins/rbac/plugin.go:     pluginutils.SetVhostPerFilterConfig(out, FilterName, perRouteRbac)
./projects/gloo/pkg/plugins/rbac/plugin.go:             pluginutils.SetRoutePerFilterConfig(out, FilterName, perRouteRbac)
./projects/gloo/pkg/plugins/jwt/plugin.go:      return pluginutils.SetRoutePerFilterConfig(out, SoloJwtFilterName, stagedCfg)
./projects/gloo/pkg/plugins/jwt/plugin.go:      return pluginutils.SetVhostPerFilterConfig(out, SoloJwtFilterName, stagedCfg)
./projects/gloo/pkg/plugins/waf/plugin.go:      pluginutils.SetVhostPerFilterConfig(out, FilterName, perVhostCfg)
./projects/gloo/pkg/plugins/waf/plugin.go:      pluginutils.SetRoutePerFilterConfig(out, FilterName, perRouteCfg)
./projects/gloo/pkg/plugins/extproc/plugin.go:  return pluginutils.SetVhostPerFilterConfig(out, FilterName, extProcPerRoute)
./projects/gloo/pkg/plugins/extproc/plugin.go:  return pluginutils.SetRoutePerFilterConfig(out, FilterName, extProcPerRoute)

gloo:

grep -rE "pluginutils[.].*PerFilterConfig[(]" ./
./projects/gloo/pkg/plugins/rest/plugin.go:     return pluginutils.MarkPerFilterConfig(params.Ctx, params.Snapshot, in, out, transformation.FilterName,
./projects/gloo/pkg/plugins/buffer/plugin.go:           return pluginutils.SetRoutePerFilterConfig(out, wellknown.Buffer, getNoBufferConfig())
./projects/gloo/pkg/plugins/buffer/plugin.go:           return pluginutils.SetRoutePerFilterConfig(out, wellknown.Buffer, config)
./projects/gloo/pkg/plugins/buffer/plugin.go:           return pluginutils.SetVhostPerFilterConfig(out, wellknown.Buffer, getNoBufferConfig())
./projects/gloo/pkg/plugins/buffer/plugin.go:           return pluginutils.SetVhostPerFilterConfig(out, wellknown.Buffer, config)
./projects/gloo/pkg/plugins/buffer/plugin.go:           return pluginutils.SetWeightedClusterPerFilterConfig(out, wellknown.Buffer, getNoBufferConfig())
./projects/gloo/pkg/plugins/buffer/plugin.go:           return pluginutils.SetWeightedClusterPerFilterConfig(out, wellknown.Buffer, config)
./projects/gloo/pkg/plugins/dynamic_forward_proxy/plugin.go:    return pluginutils.SetRoutePerFilterConfig(out, FilterName, dfpRouteCfg)
./projects/gloo/pkg/plugins/azure/plugin.go:    return pluginutils.MarkPerFilterConfig(p.ctx, params.Snapshot, in, out, transformation.FilterName,
./projects/gloo/pkg/plugins/extauth/plugin.go:  return pluginutils.SetVhostPerFilterConfig(out, wellknown.HTTPExternalAuthorization, extAuthPerRouteConfig)
./projects/gloo/pkg/plugins/extauth/plugin.go:  return pluginutils.SetRoutePerFilterConfig(out, wellknown.HTTPExternalAuthorization, extAuthPerRouteConfig)
./projects/gloo/pkg/plugins/extauth/plugin.go:  return pluginutils.SetWeightedClusterPerFilterConfig(out, wellknown.HTTPExternalAuthorization, extAuthPerRouteConfig)
./projects/gloo/pkg/plugins/grpc/plugin.go:     return pluginutils.MarkPerFilterConfig(params.Ctx, params.Snapshot, in, out, transformation.FilterName,
./projects/gloo/pkg/plugins/csrf/plugin.go:     return pluginutils.SetRoutePerFilterConfig(out, FilterName, envoyCsrfConfig)
./projects/gloo/pkg/plugins/csrf/plugin.go:     return pluginutils.SetVhostPerFilterConfig(out, FilterName, envoyCsrfConfig)
./projects/gloo/pkg/plugins/csrf/plugin.go:     return pluginutils.SetWeightedClusterPerFilterConfig(out, FilterName, envoyCsrfConfig)
./projects/gloo/pkg/plugins/faultinjection/plugin.go:   return pluginutils.MarkPerFilterConfig(params.Ctx, params.Snapshot, in, out, wellknown.Fault, markFilterConfigFunc)
./projects/gloo/pkg/plugins/transformation/plugin.go:   return pluginutils.ModifyVhostPerFilterConfig(out, FilterName, mergeFunc(envoyTransformation))
./projects/gloo/pkg/plugins/transformation/plugin.go:   return pluginutils.ModifyRoutePerFilterConfig(out, FilterName, mergeFunc(envoyTransformation))
./projects/gloo/pkg/plugins/transformation/plugin.go:   return pluginutils.ModifyWeightedClusterPerFilterConfig(out, FilterName, mergeFunc(envoyTransformation))
./projects/gloo/pkg/plugins/cors/plugin.go:     return pluginutils.SetVhostPerFilterConfig(out, wellknown.CORS, corsPolicy)
./projects/gloo/pkg/plugins/cors/plugin.go:     return pluginutils.SetRoutePerFilterConfig(out, wellknown.CORS, corsPolicy)
./projects/gloo/pkg/plugins/local_ratelimit/utils.go:   return pluginutils.ModifyVhostPerFilterConfig(out, HTTPFilterName, modIfNoExisting(filter))
./projects/gloo/pkg/plugins/local_ratelimit/utils.go:   return pluginutils.ModifyRoutePerFilterConfig(out, HTTPFilterName, modIfNoExisting(filter))
./projects/gloo/pkg/plugins/aws/plugin.go:      err := pluginutils.MarkPerFilterConfig(params.Ctx, params.Snapshot, in, out, FilterName,
./projects/gloo/pkg/plugins/aws/plugin.go:      return pluginutils.ModifyPerFilterConfig(params.Ctx, params.Snapshot, in, out, transformation.FilterName,

Next concern is whether any filters are being configured with names which do not match canonical names, since we know all of our TPFC have canonical. Checking this should be exhaustive by checking calls to NewStagedFilter and NewFilterWithTypedConfig for Network and HTTP filters, respectively. gloo:

grep -rEA1 "(NewStagedFilter|NewFilterWithTypedConfig)[(]" ./
./projects/gloo/pkg/translator/utils.go:func NewFilterWithTypedConfig(name string, config proto.Message) (*envoy_config_listener_v3.Filter, error) {
./projects/gloo/pkg/translator/utils.go-
--
./projects/gloo/pkg/translator/network_filters.go:      hcmFilter, err := NewFilterWithTypedConfig(wellknown.HTTPConnectionManager, httpConnectionManager)
./projects/gloo/pkg/translator/network_filters.go-      if err != nil {
--
./projects/gloo/pkg/translator/network_filters.go:      newStagedFilter, err := plugins.NewStagedFilter(
./projects/gloo/pkg/translator/network_filters.go-              wellknown.Router,
--
./projects/gloo/pkg/plugins/ratelimit/plugin.go:        stagedRateLimitFilter, err := plugins.NewStagedFilter(
./projects/gloo/pkg/plugins/ratelimit/plugin.go-                wellknown.HTTPRateLimit,
--
./projects/gloo/pkg/plugins/buffer/plugin.go:   bufferFilter, err := plugins.NewStagedFilter(wellknown.Buffer, bufferConfig, pluginStage)
./projects/gloo/pkg/plugins/buffer/plugin.go-   if err != nil {
--
./projects/gloo/pkg/plugins/dynamic_forward_proxy/plugin.go:    c, err := plugins.NewStagedFilter(FilterName, dfp, pluginStage)
./projects/gloo/pkg/plugins/dynamic_forward_proxy/plugin.go-    if err != nil {
--
./projects/gloo/pkg/plugins/extauth/config_generator.go:                stagedFilter, err := plugins.NewStagedFilter(wellknown.HTTPExternalAuthorization, extAuthCfg, stage)
./projects/gloo/pkg/plugins/extauth/config_generator.go-                if err != nil {
--
./projects/gloo/pkg/plugins/grpc/plugin.go:             shf, err := plugins.NewStagedFilter(wellknown.GRPCJSONTranscoder, filterConfig, pluginStage)
./projects/gloo/pkg/plugins/grpc/plugin.go-             if err != nil {
--
./projects/gloo/pkg/plugins/grpcjson/plugin.go: grpcJsonFilter, err := plugins.NewStagedFilter(wellknown.GRPCJSONTranscoder, envoyGrpcJsonConf, pluginStage)
./projects/gloo/pkg/plugins/grpcjson/plugin.go- if err != nil {
--
./projects/gloo/pkg/plugins/grpcjson/plugin.go:         grpcJsonFilter, err := plugins.NewStagedFilter(wellknown.GRPCJSONTranscoder, envoyGrpcJsonConf, pluginStage)
./projects/gloo/pkg/plugins/grpcjson/plugin.go-         if err != nil {
--
./projects/gloo/pkg/plugins/grpcjson/plugin.go:         grpcJsonFilter, err := plugins.NewStagedFilter(wellknown.GRPCJSONTranscoder, envoyGrpcJsonConf, pluginStage)
./projects/gloo/pkg/plugins/grpcjson/plugin.go-         if err != nil {
--
./projects/gloo/pkg/plugins/grpcweb/plugin.go:  return []plugins.StagedHttpFilter{plugins.MustNewStagedFilter(wellknown.GRPCWeb, &envoygrpcweb.GrpcWeb{}, pluginStage)}, nil
./projects/gloo/pkg/plugins/grpcweb/plugin.go-}
--
./projects/gloo/pkg/plugins/gzip/plugin.go:     gzipFilter, err := plugins.NewStagedFilter(CompressorFilterName, envoyGzipConfig, pluginStage)
./projects/gloo/pkg/plugins/gzip/plugin.go-     if err != nil {
--
./projects/gloo/pkg/plugins/healthcheck/plugin.go:      healthCheckFilter, err := plugins.NewStagedFilter(wellknown.HealthCheck, hc, pluginStage)
./projects/gloo/pkg/plugins/healthcheck/plugin.go-      if err != nil {
--
./projects/gloo/pkg/plugins/csrf/plugin.go:     csrfFilter, err := plugins.NewStagedFilter(FilterName, envoyCsrfConfig, pluginStage)
./projects/gloo/pkg/plugins/csrf/plugin.go-     if err != nil {
--
./projects/gloo/pkg/plugins/faultinjection/plugin.go:   return []plugins.StagedHttpFilter{plugins.MustNewStagedFilter(wellknown.Fault, &envoyhttpfault.HTTPFault{}, pluginStage)}, nil
./projects/gloo/pkg/plugins/faultinjection/plugin.go-}
--
./projects/gloo/pkg/plugins/transformation/plugin_test.go:                              expectedFilter := plugins.MustNewStagedFilter(
./projects/gloo/pkg/plugins/transformation/plugin_test.go-                                      FilterName,
--
./projects/gloo/pkg/plugins/transformation/plugin.go:           earlyFilter, err := plugins.NewStagedFilter(FilterName, earlyStageConfig, earlyPluginStage)
./projects/gloo/pkg/plugins/transformation/plugin.go-           if err != nil {
--
./projects/gloo/pkg/plugins/transformation/plugin.go:           plugins.MustNewStagedFilter(FilterName,
./projects/gloo/pkg/plugins/transformation/plugin.go-                   &envoytransformation.FilterTransformations{
--
./projects/gloo/pkg/plugins/staged_filters.go:  return NewStagedFilter(name, config, stage)
./projects/gloo/pkg/plugins/staged_filters.go-}
--
./projects/gloo/pkg/plugins/staged_filters.go:func MustNewStagedFilter(name string, config proto.Message, stage FilterStage) StagedHttpFilter {
./projects/gloo/pkg/plugins/staged_filters.go:  s, _ := NewStagedFilter(name, config, stage)
./projects/gloo/pkg/plugins/staged_filters.go-  return s
--
./projects/gloo/pkg/plugins/staged_filters.go:func NewStagedFilter(name string, config proto.Message, stage FilterStage) (StagedHttpFilter, error) {
./projects/gloo/pkg/plugins/staged_filters.go-
--
./projects/gloo/pkg/plugins/cors/plugin.go:     return []plugins.StagedHttpFilter{plugins.MustNewStagedFilter(wellknown.CORS, &envoy_config_cors_v3.Cors{}, pluginStage)}, nil
./projects/gloo/pkg/plugins/cors/plugin.go-}
--
./projects/gloo/pkg/plugins/local_ratelimit/plugin.go:  stagedRateLimitFilter, err := plugins.NewStagedFilter(
./projects/gloo/pkg/plugins/local_ratelimit/plugin.go-          HTTPFilterName,
--
./projects/gloo/pkg/plugins/tcp/plugin.go:      tcpFilter, err := translatorutil.NewFilterWithTypedConfig(wellknown.TCPProxy, cfg)
./projects/gloo/pkg/plugins/tcp/plugin.go-      if err != nil {
--
./projects/gloo/pkg/plugins/aws/plugin.go:      f, err := plugins.NewStagedFilter(FilterName, filterConfig, pluginStage)
./projects/gloo/pkg/plugins/aws/plugin.go-      if err != nil {
--
./projects/gloo/pkg/plugins/aws/plugin.go:              tf, err := plugins.NewStagedFilter(transformation.FilterName, awsStageConfig, transformPluginStage)
./projects/gloo/pkg/plugins/aws/plugin.go-              if err != nil {

solo-projects:

grep -rEA1 "(NewStagedFilter|NewFilterWithTypedConfig)[(]" ./
./projects/gloo/pkg/plugins/graphql/plugin.go:  stagedFilter, err := plugins.NewStagedFilter(FilterName, emptyConf, FilterStage)
./projects/gloo/pkg/plugins/graphql/plugin.go-  if err != nil {
--
./projects/gloo/pkg/plugins/tap/plugin.go:      stagedFilter, err := plugins.NewStagedFilter(FilterName, envoyTapConfig, filterStage)
./projects/gloo/pkg/plugins/tap/plugin.go-      if err != nil {
--
./projects/gloo/pkg/plugins/proxylatency/plugin.go:             stagedFilter, err := plugins.NewStagedFilter(FilterName, pl, FilterStage)
./projects/gloo/pkg/plugins/proxylatency/plugin.go-             if err != nil {
--
./projects/gloo/pkg/plugins/extproc/plugin.go:  stagedFilter, err := plugins.NewStagedFilter(FilterName, extProcFilter, *convertedStage)
./projects/gloo/pkg/plugins/extproc/plugin.go-  if err != nil {

[jbohanon]

health check uneject behavior solution: call out to end users/field eng

This has actually been default-on behavior since 1.26, so I don't think it makes sense to call out this change. In this, we are basically changing from default-true runtime guard to a default-true config option. https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.26/v1.26.0

[ben-taussig-solo]

Thanks for verifying the downgrade to canonical name issue so thoroughly. With that, I expect to be comfortable approving these PRs once I am able to look further into the header_mutation change

[jbohanon]

Thanks for verifying the downgrade to canonical name issue so thoroughly. With that, I expect to be comfortable approving these PRs once I am able to look further into the header_mutation change

It looks to me that this only affects the header_mutation HTTP filter, which we do not use. AFAICT we only configure header mutations on the route, which are applied during execution of the router filter.

our plugin and the related envoy codepaths: https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/plugins/headers/plugin.go https://github.com/envoyproxy/envoy/blob/main/source/common/router/config_impl.cc

PR which added this changelog: https://github.com/envoyproxy/envoy/pull/30220/files

[ben-taussig-solo]

It looks to me that this only affects the header_mutation HTTP filter, which we do not use. AFAICT we only configure header mutations on the route, which are applied during execution of the router filter.

Makes sense -- I was unsure of whether we used this filter while reviewing the changelogs

[ben-taussig-solo]

After another review of the changelogs:

• looks like we might've dodged a bullet by upgrading directly from 1.27 to 1.29 -- the runtime key overload.global_downstream_max_connections was deprecated in 1.28, and un-deprecated in 1.29. We appear to configure it in a lot of our helm templates

That's the only thing I could turn up, so I'm ready to approve these PRs

[jbohanon]

Released in: Gloo Edge OSS v1.17.0-beta15 Will be released in: Gloo Edge EE v1.17.0-beta1

Documentation for breaking changes should be added to upgrade guide

[jbohanon]

Documentation work to be tracked with https://github.com/solo-io/gloo/issues/9271