search

envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway
https://gateway.envoyproxy.io
Apache License 2.0
1.45k stars 297 forks source link

Changes to Gateway infrastructure labels fail to propagate to the service and pods #3666

Open christiancadieux opened 2 weeks ago

christiancadieux commented 2 weeks ago

Description: Changes to Gateway infrastructure labels do not propagate to the service and pods

Repro steps:

Note: maybe related to other 'immutable' bugs like https://github.com/envoyproxy/gateway/issues/1818 Deleting the Gateway does delete the envoy-proxy deployment

Environment:

Include the environment like gateway version, envoy version and so on.

Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
  gatewayClassName: envoygateway-tenant1
  infrastructure:
    labels:
      infra1-label: infra1-value23
...

PODS

$ kubectl get pod --show-labels
NAME                                                        READY   STATUS    RESTARTS   AGE     LABELS
envoy-gateway-5769559676-8rqh4                              1/1     Running   0          17m     app.kubernetes.io/instance=eg-tenant1,app.kubernetes.io/name=gateway-helm,control-plane=envoy-gateway,pod-template-hash=5769559676,tsf.io/service=service1,tsf.io/tenant=tenant1
envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl   2/2     Running   0          8m22s   app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-value2,pod-template-hash=6979c4cbf5

Logs: the logs when the gateway labels are updated:

2024-06-24T17:51:48.500Z    INFO    provider    kubernetes/controller.go:165    reconciling gateways    {"runner": "provider"}
2024-06-24T17:51:48.500Z    INFO    provider    kubernetes/controller.go:803    processing Gateway  {"runner": "provider", "namespace": "tenant1-ns1", "name": "envoy-gateway"}
2024-06-24T17:51:48.500Z    INFO    provider    kubernetes/routes.go:268    processing HTTPRoute    {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/controller.go:576    processing OIDC HMAC Secret {"runner": "provider", "namespace": "tenant1-eg", "name": "envoy-oidc-hmac"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/controller.go:1597   processing envoyproxy   {"runner": "provider", "namespace": "tenant1-eg", "name": "proxy-config-tenant1"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/controller.go:374    processing Backend  {"runner": "provider", "kind": "Service", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/controller.go:388    added Service to resource tree  {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/controller.go:436    added EndpointSlice to resource tree    {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend-z8xs8"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/controller.go:313    reconciled gateways successfully    {"runner": "provider"}
2024-06-24T17:51:48.501Z    INFO    gateway-api runner/runner.go:58 received an update  {"runner": "gateway-api"}
2024-06-24T17:51:48.501Z    INFO    provider    kubernetes/status_updater.go:141    received a status update    {"runner": "provider", "namespace": "", "name": "envoygateway-tenant1"}
2024-06-24T17:51:48.502Z    INFO    provider.envoygateway-tenant1   kubernetes/status_updater.go:105    status unchanged, bypassing update  {"runner": "provider"}
2024-06-24T17:51:48.503Z    INFO    gateway-api runner/runner.go:111    proxy:
  config:
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyProxy
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyProxy","metadata":{"annotations":{},"name":"proxy-config-tenant1","namespace":"tenant1-eg"},"spec":{"logging":{"level":{"default":"warn"}},"provider":{"kubernetes":{"envoyDeployment":{"container":{"image":"hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless"}}},"type":"Kubernetes"}}}
      creationTimestamp: "2024-06-20T23:22:25Z"
      generation: 1
      managedFields:
      - apiVersion: gateway.envoyproxy.io/v1alpha1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:annotations:
              .: {}
              f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:spec:
            .: {}
            f:logging:
              .: {}
              f:level:
                .: {}
                f:default: {}
            f:provider:
              .: {}
              f:kubernetes:
                .: {}
                f:envoyDeployment:
                  .: {}
                  f:container:
                    .: {}
                    f:image: {}
              f:type: {}
        manager: kubectl-client-side-apply
        operation: Update
        time: "2024-06-20T23:22:25Z"
      name: proxy-config-tenant1
      namespace: tenant1-eg
      resourceVersion: "24267218"
      uid: b867d886-6c17-47ef-b535-afa743d49e03
    spec:
      logging:
        level:
          default: warn
      provider:
        kubernetes:
          envoyDeployment:
            container:
              image: hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless
        type: Kubernetes
    status: {}
  listeners:
  - address: null
    name: tenant1-ns1/envoy-gateway/http
    ports:
    - containerPort: 8080
      name: http-8080
      protocol: HTTP
      servicePort: 8080
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: envoy-gateway
      gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1
      infra1-label: infra1-value2243
  name: tenant1-ns1/envoy-gateway
    {"runner": "gateway-api", "infra-ir": "tenant1-ns1/envoy-gateway"}
2024-06-24T17:51:48.504Z    INFO    infrastructure  runner/runner.go:78 received an update  {"runner": "infrastructure"}
2024-06-24T17:51:48.504Z    INFO    gateway-api runner/runner.go:122    accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: tenant1-ns1/envoy-gateway/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 8080
  routes:
  - destination:
      name: httproute/tenant1-ns1/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 198.19.5.80
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: www.tenant1.example.com
    isHTTP2: false
    name: httproute/tenant1-ns1/backend/rule/0/match/0/www_tenant1_example_com
    pathMatch:
      distinct: false
      name: ""
      prefix: /
    {"runner": "gateway-api", "xds-ir": "tenant1-ns1/envoy-gateway"}
2024-06-24T17:51:48.504Z    INFO    provider    kubernetes/status_updater.go:141    received a status update    {"runner": "provider", "namespace": "tenant1-ns1", "name": "backend"}
2024-06-24T17:51:48.511Z    INFO    provider    kubernetes/status_updater.go:141    received a status update    {"runner": "provider", "namespace": "tenant1-ns1", "name": "envoy-gateway"}
2024-06-24T17:51:48.524Z    ERROR   infrastructure  runner/runner.go:94 failed to create new infra  {"runner": "infrastructure", "error": "failed to create or update deployment tenant1-eg/envoy-tenant1-ns1-envoy-gateway-d016235c: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-tenant1-ns1-envoy-gateway-d016235c  tenant1-eg    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[] [] [] []},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1,infra1-label: infra1-value2243,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-tenant1-ns1-envoy-gateway-d016235c,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless [envoy] [--service-cluster tenant1-ns1/envoy-gateway --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n  access_log:\n  - name: envoy.access_loggers.file\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n      path: /dev/null\n  address:\n    socket_address:\n      address: 127.0.0.1\n      port_value: 19000\nlayered_runtime:\n  layers:\n  - name: global_config\n    static_layer:\n      envoy.restart_features.use_eds_cache_for_ads: true\n      re2.max_program_size.error_level: 4294967295\n      re2.max_program_size.warn_level: 1000\ndynamic_resources:\n  ads_config:\n    api_type: DELTA_GRPC\n    transport_api_version: V3\n    grpc_services:\n    - envoy_grpc:\n        cluster_name: xds_cluster\n    set_node_on_first_message_only: true\n  lds_config:\n    ads: {}\n    resource_api_version: V3\n  cds_config:\n    ads: {}\n    resource_api_version: V3\nstatic_resources:\n  listeners:\n  - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n    address:\n      socket_address:\n        address: 0.0.0.0\n        port_value: 19001\n        protocol: TCP\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: eg-ready-http\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: prometheus_stats\n              domains:\n              - \"*\"\n              routes:\n              - match:\n                  prefix: /stats/prometheus\n                route:\n                  cluster: prometheus_stats\n          http_filters:\n          - name: envoy.filters.http.health_check\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n              pass_through_mode: false\n              headers:\n              - name: \":path\"\n                string_match:\n                  exact: /ready\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n  clusters:\n  - name: prometheus_stats\n    connect_timeout: 0.250s\n    type: STATIC\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: prometheus_stats\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: 127.0.0.1\n                port_value: 19000\n  - connect_timeout: 10s\n    load_assignment:\n      cluster_name: xds_cluster\n      endpoints:\n      - load_balancing_weight: 1\n        lb_endpoints:\n        - load_balancing_weight: 1\n          endpoint:\n            address:\n              socket_address:\n                address: envoy-gateway\n                port_value: 18000\n    typed_extension_protocol_options:\n      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n        \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n        explicit_http_config:\n          http2_protocol_options:\n            connection_keepalive:\n              interval: 30s\n              timeout: 5s\n    name: xds_cluster\n    type: STRICT_DNS\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          tls_params:\n            tls_maximum_protocol_version: TLSv1_3\n          tls_certificate_sds_secret_configs:\n          - name: xds_certificate\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-certificate.json\"\n              resource_api_version: V3\n          validation_context_sds_secret_config:\n            name: xds_trusted_ca\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-trusted-ca.json\"\n              resource_api_version: V3\noverload_manager:\n  refresh_interval: 0.25s\n  resource_monitors:\n  - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n      max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads]  [{http-8080 0 8080 TCP } {metrics 0 19001 TCP }] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}] []} [] <nil> [{certs true <nil> /certs  <nil> } {sds false <nil> /sds  <nil> }] [] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false} {shutdown-manager hub.comcast.net/k8s-eng/envoyproxy/gateway:v1.0.1 [envoy-gateway] [envoy shutdown-manager]  [] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>}  BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false}] [] Always 0xc0009fe038 <nil> ClusterFirst map[] envoy-tenant1-ns1-envoy-gateway-d016235c  0xc0009fe035  false false false <nil> nil []   nil default-scheduler [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-tenant1-ns1-envoy-gateway-d016235c\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/component\":\"proxy\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"envoy\", \"gateway.envoyproxy.io/owning-gateway-name\":\"envoy-gateway\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"tenant1-ns1\", \"infra1-label\":\"infra1-value2243\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}
2024-06-24T17:51:48.524Z    ERROR   watchable   message/watchutil.go:56 observed an error   {"runner": "infrastructure", "error": "failed to create or update deployment tenant1-eg/envoy-tenant1-ns1-envoy-gateway-d016235c: failed to create/update resource with server-side apply for obj &Deployment{ObjectMeta:{envoy-tenant1-ns1-envoy-gateway-d016235c  tenant1-eg    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[] [] [] []},Spec:DeploymentSpec{Replicas:nil,Selector:&v1.LabelSelector{MatchLabels:map[string]string{app.kubernetes.io/component: proxy,app.kubernetes.io/managed-by: envoy-gateway,app.kubernetes.io/name: envoy,gateway.envoyproxy.io/owning-gateway-name: envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace: tenant1-ns1,infra1-label: infra1-value2243,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/component:proxy app.kubernetes.io/managed-by:envoy-gateway app.kubernetes.io/name:envoy gateway.envoyproxy.io/owning-gateway-name:envoy-gateway gateway.envoyproxy.io/owning-gateway-namespace:tenant1-ns1 infra1-label:infra1-value2243] map[prometheus.io/path:/stats/prometheus prometheus.io/port:19001 prometheus.io/scrape:true] [] [] []} {[{certs {nil nil nil nil nil SecretVolumeSource{SecretName:envoy,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {sds {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:envoy-tenant1-ns1-envoy-gateway-d016235c,},Items:[]KeyToPath{KeyToPath{Key:xds-trusted-ca.json,Path:xds-trusted-ca.json,Mode:nil,},KeyToPath{Key:xds-certificate.json,Path:xds-certificate.json,Mode:nil,},},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [] [{envoy hub.comcast.net/k8s-eng/envoyproxy/envoy:v1.0.1.distroless [envoy] [--service-cluster tenant1-ns1/envoy-gateway --service-node $(ENVOY_POD_NAME) --config-yaml admin:\n  access_log:\n  - name: envoy.access_loggers.file\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\n      path: /dev/null\n  address:\n    socket_address:\n      address: 127.0.0.1\n      port_value: 19000\nlayered_runtime:\n  layers:\n  - name: global_config\n    static_layer:\n      envoy.restart_features.use_eds_cache_for_ads: true\n      re2.max_program_size.error_level: 4294967295\n      re2.max_program_size.warn_level: 1000\ndynamic_resources:\n  ads_config:\n    api_type: DELTA_GRPC\n    transport_api_version: V3\n    grpc_services:\n    - envoy_grpc:\n        cluster_name: xds_cluster\n    set_node_on_first_message_only: true\n  lds_config:\n    ads: {}\n    resource_api_version: V3\n  cds_config:\n    ads: {}\n    resource_api_version: V3\nstatic_resources:\n  listeners:\n  - name: envoy-gateway-proxy-ready-0.0.0.0-19001\n    address:\n      socket_address:\n        address: 0.0.0.0\n        port_value: 19001\n        protocol: TCP\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: eg-ready-http\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: prometheus_stats\n              domains:\n              - \"*\"\n              routes:\n              - match:\n                  prefix: /stats/prometheus\n                route:\n                  cluster: prometheus_stats\n          http_filters:\n          - name: envoy.filters.http.health_check\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck\n              pass_through_mode: false\n              headers:\n              - name: \":path\"\n                string_match:\n                  exact: /ready\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n  clusters:\n  - name: prometheus_stats\n    connect_timeout: 0.250s\n    type: STATIC\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: prometheus_stats\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: 127.0.0.1\n                port_value: 19000\n  - connect_timeout: 10s\n    load_assignment:\n      cluster_name: xds_cluster\n      endpoints:\n      - load_balancing_weight: 1\n        lb_endpoints:\n        - load_balancing_weight: 1\n          endpoint:\n            address:\n              socket_address:\n                address: envoy-gateway\n                port_value: 18000\n    typed_extension_protocol_options:\n      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:\n        \"@type\": \"type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions\"\n        explicit_http_config:\n          http2_protocol_options:\n            connection_keepalive:\n              interval: 30s\n              timeout: 5s\n    name: xds_cluster\n    type: STRICT_DNS\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          tls_params:\n            tls_maximum_protocol_version: TLSv1_3\n          tls_certificate_sds_secret_configs:\n          - name: xds_certificate\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-certificate.json\"\n              resource_api_version: V3\n          validation_context_sds_secret_config:\n            name: xds_trusted_ca\n            sds_config:\n              path_config_source:\n                path: \"/sds/xds-trusted-ca.json\"\n              resource_api_version: V3\noverload_manager:\n  refresh_interval: 0.25s\n  resource_monitors:\n  - name: \"envoy.resource_monitors.global_downstream_max_connections\"\n    typed_config:\n      \"@type\": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig\n      max_active_downstream_connections: 50000\n --log-level warn --cpuset-threads]  [{http-8080 0 8080 TCP } {metrics 0 19001 TCP }] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}] []} [] <nil> [{certs true <nil> /certs  <nil> } {sds false <nil> /sds  <nil> }] [] nil &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 19001 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/shutdown/ready,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false} {shutdown-manager hub.comcast.net/k8s-eng/envoyproxy/gateway:v1.0.1 [envoy-gateway] [envoy shutdown-manager]  [] [] [{ENVOY_GATEWAY_NAMESPACE  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {ENVOY_POD_NAME  &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] {map[] map[cpu:{{10 -3} {<nil>} 10m DecimalSI} memory:{{33554432 0} {<nil>}  BinarySI}] []} [] <nil> [] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{0 19002 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil &Lifecycle{PostStart:nil,PreStop:&LifecycleHandler{Exec:&ExecAction{Command:[envoy-gateway envoy shutdown],},HTTPGet:nil,TCPSocket:nil,Sleep:nil,},} /dev/termination-log File IfNotPresent nil false false false}] [] Always 0xc0009fe038 <nil> ClusterFirst map[] envoy-tenant1-ns1-envoy-gateway-d016235c  0xc0009fe035  false false false <nil> nil []   nil default-scheduler [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:RollingUpdate,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:*10,Paused:false,ProgressDeadlineSeconds:*600,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: Deployment.apps \"envoy-tenant1-ns1-envoy-gateway-d016235c\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/component\":\"proxy\", \"app.kubernetes.io/managed-by\":\"envoy-gateway\", \"app.kubernetes.io/name\":\"envoy\", \"gateway.envoyproxy.io/owning-gateway-name\":\"envoy-gateway\", \"gateway.envoyproxy.io/owning-gateway-namespace\":\"tenant1-ns1\", \"infra1-label\":\"infra1-value2243\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"}
arkodg commented 2 weeks ago

seeing field is immutable in the logs, so this is same as https://github.com/envoyproxy/gateway/issues/1818

christiancadieux commented 2 weeks ago

I don't think it's the same but it's related. for example with Services, it's important to update the labels of the service and not delete/re-create the service since re-creating would assign a new external-IP to the service, which is not good. Also, when labels come from the Gateway infrastructure, they could be important labels related to the ownership (tenant) of the Gateway for example, and it's important that the envoy-proxy pod and the service be updated.

arkodg commented 2 weeks ago

i'll bring this up in the community meeting tomorrow, the issue is the same - should Envoy Gateway recreate resources when it hits this specific error field is immutable by default , or should it be based on an opt in flag

christiancadieux commented 2 weeks ago

no need to re-create resources to update labels. It is possible to update labels with PATCH:

$ kubectl label  service/envoy-tenant1-ns1-envoy-gateway-d016235c infra1-label=infra1-test123 --overwrite  -v6
I0624 15:46:02.121803 1444301 loader.go:395] Config loaded from file:  /home/ccadie883/.kube/config
I0624 15:46:02.504242 1444301 round_trippers.go:553] GET https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/services/envoy-tenant1-ns1-envoy-gateway-d016235c 200 OK in 376 milliseconds
I0624 15:46:02.630137 1444301 round_trippers.go:553] PATCH https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/services/envoy-tenant1-ns1-envoy-gateway-d016235c?fieldManager=kubectl-label 200 OK in 124 milliseconds
service/envoy-tenant1-ns1-envoy-gateway-d016235c labeled

$ kubectl get service --show-labels
NAME                                       TYPE           CLUSTER-IP        EXTERNAL-IP     PORT(S)                         AGE     LABELS
envoy-gateway                              ClusterIP      192.168.235.139   <none>          18000/TCP,18001/TCP,19001/TCP   4h18m   app.kubernetes.io/instance=eg-tenant1,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=gateway-helm,app.kubernetes.io/version=v1.0.1,control-plane=envoy-gateway,helm.sh/chart=gateway-helm-v1.0.1
envoy-tenant1-ns1-envoy-gateway-d016235c   LoadBalancer   192.168.254.13    10.112.182.62   8080:9153/TCP                   4h10m   app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-test123

or pod:

$kubectl label  pod/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl infra1-label=infra1-test123 --overwrite  -v6
I0624 15:47:13.898528 1444420 loader.go:395] Config loaded from file:  /home/ccadie883/.kube/config
I0624 15:47:14.284137 1444420 round_trippers.go:553] GET https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/pods/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl 200 OK in 380 milliseconds
I0624 15:47:14.547887 1444420 round_trippers.go:553] PATCH https://10.112.182.142:6443/api/v1/namespaces/tenant1-eg/pods/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl?fieldManager=kubectl-label 200 OK in 138 milliseconds
pod/envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl labeled

$kubectl get pod envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl  --show-labels
NAME                                                        READY   STATUS    RESTARTS   AGE     LABELS
envoy-tenant1-ns1-envoy-gateway-d016235c-6979c4cbf5-grrgl   2/2     Running   0          4h11m   app.kubernetes.io/component=proxy,app.kubernetes.io/managed-by=envoy-gateway,app.kubernetes.io/name=envoy,gateway.envoyproxy.io/owning-gateway-name=envoy-gateway,gateway.envoyproxy.io/owning-gateway-namespace=tenant1-ns1,infra1-label=infra1-test123,pod-template-hash=6979c4cbf5
guydc commented 1 week ago

-1 to recreation. As stated, there are many possible side effects, including IP change, disruption to traffic, etc. If possible to solve this with a different strategy (e.g. patch), that should be fine.

arkodg commented 1 week ago

hey @sanposhiho can you help with this one if you have a cycle ? can we make the Patch API https://github.com/envoyproxy/gateway/blob/9a2a7f607e1db52d7aa22daa4c22749cadbf3a91/internal/infrastructure/kubernetes/infra_client.go#L29C24-L29C66 behave like kubectl --overwrite so it doesnt throw an error of field is immutable when updating labels, and also does this w/o recreating the pod or service

sanposhiho commented 1 week ago

/assign

I'll take a look.

sanposhiho commented 1 week ago

Had a bit of time checking this issue.

According to the provided logs, looks like it doesn't get a conflict at labels, but get conflicted at deployment's selector. If we fail at updating deployment here, we don't update other following resources, which is why your service isn't updated. https://github.com/envoyproxy/gateway/blob/main/internal/infrastructure/kubernetes/infra.go#L72-L87

So, I believe this issue is the same as https://github.com/envoyproxy/gateway/issues/1818, as @arkodg mentioned first.