[1.17]: Early Header Mutation

[ryanrolds]

Backport of https://github.com/solo-io/gloo/pull/10262.

Description

This PR add Early Header Manipulation to the HTTP Connection Manager settings. It allows request headers to be manipulated before they are acted on by tracing and other logic. The goal of the work is to allow customers to override Zipkin/B3 tracing headers.

API changes

• Added headers.options.gloo.solo.io.EarlyHeaderManipulation to HttpConnectionManagerSettings.

Code changes

• Updated ENVOY_GLOO_IMAGE a version with the EHM extension enabled in Envoy
• During tests, the wapper was using a 1.18 image tag. Passing VERSION as ENVOY_IMAGE_TAG in the test Make target to get it using the locally built wrapper
• Added go-httpbin and and a new upstream generator to make testing headers easier
• Added ProcessHcmNetworkFilter to the headers plugin, handled the new setting

Docs changes

• Added details about the new settings

Context

• https://soloio.slab.com/posts/early-header-mutation-design-doc-u5bw3c1m
• https://github.com/solo-io/gloo/issues/9604
• https://github.com/solo-io/envoy-gloo-ee/issues/734

Testing steps

I've test locally with Kind by setting and exercising the new logic with curl, including configuring a real Zipkin tracer.

1. Stand-up an empty Kind cluster with this PR's Gloo running

2. Apply this manifest:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
   name: zipkin
   ---
   apiVersion: v1
   kind: Service
   metadata:
   name: zipkin
   labels:
   app: zipkin
   spec:
   ports:
   - name: http
   port: 8000
   targetPort: 80
   selector:
   app: zipkin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   name: zipkin
   spec:
   replicas: 1
   selector:
   matchLabels:
     app: zipkin
     version: v1
   template:
   metadata:
     labels:
       app: zipkin
       version: v1
   spec:
     serviceAccountName: zipkin
     containers:
     - image: openzipkin/zipkin
       imagePullPolicy: IfNotPresent
       name: zipkin
       ports:
       - containerPort: 9411
   ---
   apiVersion: gloo.solo.io/v1
   kind: Upstream
   metadata:
   name: zipkin
   namespace: default
   spec:
   discoveryMetadata: {}
   kube:
   serviceName: zipkin
   serviceNamespace: default
   servicePort: 9411
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
   name: httpbin
   ---
   apiVersion: v1
   kind: Service
   metadata:
   name: httpbin
   labels:
   app: httpbin
   spec:
   ports:
   - name: http
   port: 8000
   targetPort: 80
   selector:
   app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   name: httpbin
   spec:
   replicas: 1
   selector:
   matchLabels:
     app: httpbin
     version: v1
   template:
   metadata:
     labels:
       app: httpbin
       version: v1
   spec:
     serviceAccountName: httpbin
     containers:
     - image: docker.io/kennethreitz/httpbin
       imagePullPolicy: IfNotPresent
       name: httpbin
       ports:
       - containerPort: 80
   ---
   apiVersion: gloo.solo.io/v1
   kind: Upstream
   metadata:
   name: default-httpbin-8000
   namespace: default
   spec:
   discoveryMetadata: {}
   kube:
   serviceName: httpbin
   serviceNamespace: default
   servicePort: 8000
   ---
   apiVersion: gateway.solo.io/v1
   kind: VirtualService
   metadata:
   name: httpbin
   namespace: gloo-system
   spec:
   virtualHost:
   domains:
   - '*'
   routes:
   - matchers:
     - prefix: /
     routeAction:
       single:
         upstream:
           name: default-httpbin-8000
           namespace: default
   ---
   apiVersion: gateway.solo.io/v1
   kind: Gateway
   metadata:
   labels:
   app: gloo
   name: gateway-proxy
   namespace: gloo-system
   spec:
   bindAddress: '::'
   bindPort: 8080
   httpGateway:
   options:
     httpConnectionManagerSettings:
       tracing:
         verbose: true
         requestHeadersForTags:
           - path
           - origin
         zipkinConfig:
           collectorEndpoint: /api/v2/spans
           collectorEndpointVersion: HTTP_JSON
           collectorUpstreamRef:
             name: zipkin
             namespace: default
       earlyHeaderManipulation: 
         headersToAdd:
         - header:
             key: X-B3-TRACEID
             value: "%REQ(x-override-traceid)%"
         - header:
             key: X-B3-SPANID
             value: "%REQ(x-override-spanid)%"
   proxyNames:
   - gateway-proxy
   ssl: false
   useProxyProto: false
   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
   annotations:
   meta.helm.sh/release-name: gloo
   meta.helm.sh/release-namespace: gloo-system
   labels:
   app: gloo
   app.kubernetes.io/managed-by: Helm
   gateway-proxy-id: gateway-proxy
   gloo: gateway-proxy
   name: gateway-proxy-envoy-config
   namespace: gloo-system
   data:
   envoy.yaml: |
   layered_runtime:
     layers:
     - name: static_layer
       static_layer:
         overload:
           global_downstream_max_connections: 250000
         upstream:
           healthy_panic_threshold:
             value: 0
     - name: admin_layer
       admin_layer: {}
   node:
     cluster: gateway
     id: "{{.PodName}}.{{.PodNamespace}}"
     metadata:
       # Specifies the proxy's in-memory xds cache key (see projects/gloo/pkg/xds/envoy.go)
       # This value needs to match discoveryNamespace (or "writeNamespace") in the settings template
       role: gloo-system~gateway-proxy
   static_resources:
     listeners:
       - name: prometheus_listener
         address:
           socket_address:
             address: 0.0.0.0
             port_value: 8081
         filter_chains:
           - filters:
               - name: envoy.filters.network.http_connection_manager
                 typed_config:
                   "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                   codec_type: AUTO
                   stat_prefix: prometheus
                   route_config:
                     name: prometheus_route
                     virtual_hosts:
                       - name: prometheus_host
                         domains:
                           - "*"
                         routes:
                           - match:
                               path: "/ready"
                               headers:
                               - name: ":method"
                                 exact_match: GET
                             route:
                               cluster: admin_port_cluster
                           - match:
                               prefix: "/metrics"
                               headers:
                               - name: ":method"
                                 exact_match: GET
                             route:
                               prefix_rewrite: /stats/prometheus
                               cluster: admin_port_cluster
                   http_filters:
                     - name: envoy.filters.http.router
                       typed_config:
                         "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
   
     clusters:
     - name: gloo.gloo-system.svc.cluster.local:9977
       alt_stat_name: xds_cluster
       connect_timeout: 5.000s
       load_assignment:
         cluster_name: gloo.gloo-system.svc.cluster.local:9977
         endpoints:
         - lb_endpoints:
           - endpoint:
               address:
                 socket_address:
                   address: gloo.gloo-system.svc.cluster.local
                   port_value: 9977
       http2_protocol_options: {}
       upstream_connection_options:
         tcp_keepalive:
           keepalive_time: 60
       type: STRICT_DNS
       respect_dns_ttl: true
     - name: rest_xds_cluster
       alt_stat_name: rest_xds_cluster
       connect_timeout: 5.000s
       load_assignment:
         cluster_name: rest_xds_cluster
         endpoints:
         - lb_endpoints:
           - endpoint:
               address:
                 socket_address:
                   address: gloo.gloo-system.svc.cluster.local
                   port_value: 9976
       upstream_connection_options:
         tcp_keepalive:
           keepalive_time: 60
       type: STRICT_DNS
       respect_dns_ttl: true
     - name: wasm-cache
       connect_timeout: 5.000s
       load_assignment:
         cluster_name: wasm-cache
         endpoints:
         - lb_endpoints:
           - endpoint:
               address:
                 socket_address:
                   address: gloo.gloo-system.svc.cluster.local
                   port_value: 9979
       upstream_connection_options:
         tcp_keepalive:
           keepalive_time: 60
       type: STRICT_DNS
       respect_dns_ttl: true
     - name: admin_port_cluster
       connect_timeout: 5.000s
       type: STATIC
       lb_policy: ROUND_ROBIN
       load_assignment:
         cluster_name: admin_port_cluster
         endpoints:
         - lb_endpoints:
           - endpoint:
               address:
                 socket_address:
                   address: 127.0.0.1
                   port_value: 19000
     - name: zipkin
       connect_timeout: 1s
       type: STRICT_DNS
       load_assignment:
         cluster_name: zipkin
         endpoints:
         - lb_endpoints:
           - endpoint:
               address:
                 socket_address:
                   address: zipkin
                   port_value: 9411
   dynamic_resources:
     ads_config:
       transport_api_version: V3
       api_type: GRPC
       rate_limit_settings: {}
       grpc_services:
       - envoy_grpc: {cluster_name: gloo.gloo-system.svc.cluster.local:9977}
     cds_config:
       resource_api_version: V3
       ads: {}
     lds_config:
       resource_api_version: V3
       ads: {}
   admin:
     access_log_path: /dev/null
     address:
       socket_address:
         address: 127.0.0.1
         port_value: 19000

3. Port forward to the proxy a. kubectl -n gloo-system port-forward services/gateway-proxy 8080:80
4. Confirm behavior with curl. a. curl -H "x-override-traceid: asdfasdf" -H "x-override-spanid: zxcvzxcv" http://localhost:8080/get b. curl http://localhost:8080/get

Checklist:

• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works

BOT NOTES: resolves https://github.com/solo-io/gloo/issues/9604

[solo-changelog-bot[bot]]

Issues linked to changelog: https://github.com/solo-io/gloo/issues/9604

[ryanrolds]

/kick