search

envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway
https://gateway.envoyproxy.io
Apache License 2.0
1.54k stars 330 forks source link

HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264

Open BadLiveware opened 3 days ago

BadLiveware commented 3 days ago

Description: I have 3 different gateways up and running, only 2 of which are used in this issue. Only one gateway is shown in status, while both are configured. This results in things like external-dns only setting up DNS records for one gateway address.

Repro steps: Overview:

─❯ kubectl get envoyproxy -A
NAMESPACE       NAME       AGE
envoy-gateway   cluster    2d22h
envoy-gateway   external   2d22h
envoy-gateway   internal   2d22h

─❯ kubectl get gatewayclass -A
NAME       CONTROLLER                                      ACCEPTED   AGE
cluster    gateway.envoyproxy.io/gatewayclass-controller   True       2d22h
external   gateway.envoyproxy.io/gatewayclass-controller   True       2d22h
internal   gateway.envoyproxy.io/gatewayclass-controller   True       13d

─❯ kubectl get gateway -A
NAMESPACE       NAME       CLASS      ADDRESS          PROGRAMMED   AGE
envoy-gateway   cluster    cluster    10.182.165.187   True         2d22h
envoy-gateway   external   external   10.182.83.109    True         2d22h
envoy-gateway   internal   internal   10.180.0.60      True         13d

─❯ kubectl get httproute -A
NAMESPACE     NAME          HOSTNAMES                                                         AGE
dev-ops       webapi        ["webapi.dev-ops.c.example.org","webapi.dev-ops.i.example.org"]   2d22h # This is attached to internal and cluster
echo-server   echo-server   ["echo-server-gw.dev.example.org"]                                14d # This is attached to external
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal
  namespace: envoy-gateway
spec:
  gatewayClassName: internal
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: '*.i.example.org'
    name: http
    port: 80
    protocol: HTTP
status:
  addresses:
  - type: IPAddress
    value: 10.180.0.60
  conditions:
  - lastTransitionTime: "2024-09-14T11:07:57Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 8
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-09-14T11:09:29Z"
    message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
    observedGeneration: 8
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 8
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener has been successfully translated
      observedGeneration: 8
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener references have been resolved
      observedGeneration: 8
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cluster
  namespace: envoy-gateway
spec:
  gatewayClassName: cluster
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: '*.c.example.org'
    name: http
    port: 80
    protocol: HTTP
status:
  addresses:
  - type: IPAddress
    value: 10.182.165.187
  conditions:
  - lastTransitionTime: "2024-09-14T11:07:56Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 3
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-09-14T11:08:37Z"
    message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
    observedGeneration: 3
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 3
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener has been successfully translated
      observedGeneration: 3
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener references have been resolved
      observedGeneration: 3
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

Same HTTPRoute switches its status between gateways 

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: webapi
  namespace: dev-ops
spec:
  hostnames:
  - webapi.dev-ops.c.example.org
  - webapi.dev-ops.i.example.org
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: cluster
    namespace: envoy-gateway
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: internal
    namespace: envoy-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-auth
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /auth
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-browse
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /browse
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-dev
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /dev
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-discover
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /discover
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-images
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /images
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-marketing
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /marketing
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-member
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /member
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-myexample
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /myexample
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-selling
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /selling
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-shopping
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /shopping
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-09-17T08:00:53Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-17T08:00:53Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: internal
      namespace: envoy-gateway
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: webapi
  namespace: dev-ops
spec:
  hostnames:
  - webapi.dev-ops.c.example.org
  - webapi.dev-ops.i.example.org
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: cluster
    namespace: envoy-gateway
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: internal
    namespace: envoy-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-auth
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /auth
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-browse
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /browse
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-dev
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /dev
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-discover
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /discover
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-images
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /images
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-marketing
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /marketing
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-member
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /member
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-myexample
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /myexample
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-selling
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /selling
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-shopping
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /shopping
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-09-17T08:31:21Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-17T08:31:21Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: cluster
      namespace: envoy-gateway

(Part) Config dump to show both gateways are actually configured: internal:

        {
          "@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
          "dynamicRouteConfigs": [
            {
              "lastUpdated": "2024-09-14T11:09:04.604Z",
              "routeConfig": {
                "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
                "ignorePortInHostMatching": true,
                "name": "envoy-gateway/internal/http",
                "virtualHosts": [
                  {
                    "domains": [
                      "webapi.dev-ops.i.example.org"
                    ],
                    "metadata": {
                      "filterMetadata": {
                        "envoy-gateway": {
                          "resources": [
                            {
                              "kind": "Gateway",
                              "name": "internal",
                              "namespace": "envoy-gateway",
                              "sectionName": "http"
                            }
                          ]
                        }
                      }
                    },
                    "name": "envoy-gateway/internal/http/webapi_dev-ops_i_example_org",
                    "routes": [
                      {
                        "match": {
                          "pathSeparatedPrefix": "/marketing"
                        },
                        "metadata": {
                          "filterMetadata": {
                            "envoy-gateway": {
                              "resources": [
                                {
                                  "kind": "HTTPRoute",
                                  "name": "webapi",
                                  "namespace": "dev-ops"
                                }
                              ]
                            }
                          }
                        },
                        "name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_i_example_org",
                        "route": {
                          "cluster": "httproute/dev-ops/webapi/rule/5",
                          "regexRewrite": {
                            "pattern": {
                              "regex": "^/marketing\\/*"
                            },
                            "substitution": "/"
                          },
                          "upgradeConfigs": [
                            {
                              "upgradeType": "websocket"
                            }
                          ]
                        }

cluster

        {
          "@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
          "dynamicRouteConfigs": [
            {
              "lastUpdated": "2024-09-14T11:08:30.023Z",
              "routeConfig": {
                "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
                "ignorePortInHostMatching": true,
                "name": "envoy-gateway/cluster/http",
                "virtualHosts": [
                  {
                    "domains": [
                      "webapi.dev-ops.c.example.org"
                    ],
                    "metadata": {
                      "filterMetadata": {
                        "envoy-gateway": {
                          "resources": [
                            {
                              "kind": "Gateway",
                              "name": "cluster",
                              "namespace": "envoy-gateway",
                              "sectionName": "http"
                            }
                          ]
                        }
                      }
                    },
                    "name": "envoy-gateway/cluster/http/webapi_dev-ops_c_example_org",
                    "routes": [
                      {
                        "match": {
                          "pathSeparatedPrefix": "/marketing"
                        },
                        "metadata": {
                          "filterMetadata": {
                            "envoy-gateway": {
                              "resources": [
                                {
                                  "kind": "HTTPRoute",
                                  "name": "webapi",
                                  "namespace": "dev-ops"
                                }
                              ]
                            }
                          }
                        },
                        "name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_c_example.org",
                        "route": {
                          "cluster": "httproute/dev-ops/webapi/rule/5",
                          "regexRewrite": {
                            "pattern": {
                              "regex": "^/marketing\\/*"
                            },
                            "substitution": "/"
                          },
                          "upgradeConfigs": [
                            {
                              "upgradeType": "websocket"
                            }
                          ]
                        }

Environment: GKE 1.29 Envoy Gateway v1.1.0 Upgraded to envoy gateway v1.1.1 after writing this issue, and it remains an issue

Logs: downloaded-logs-20240917-102549.csv

BadLiveware commented 3 days ago

Tried removing the extraneous resources, and it didnt help.

egctl status

❯ egctl x status all -v -A
NAME                    TYPE       STATUS    REASON     MESSAGE              OBSERVED GENERATION   LAST TRANSITION TIME
gatewayclass/cluster    Accepted   True      Accepted   Valid GatewayClass   1                     2024-09-14 12:00:45 +0200 CEST
gatewayclass/internal   Accepted   True      Accepted   Valid GatewayClass   2                     2024-09-14 12:00:45 +0200 CEST

NAMESPACE       NAME               TYPE         STATUS    REASON       MESSAGE                                                                    OBSERVED GENERATION   LAST TRANSITION TIME
envoy-gateway   gateway/cluster    Programmed   True      Programmed   Address assigned to the Gateway, 1/1 envoy Deployment replicas available   1                     2024-09-17 11:24:37 +0200 CEST
                                   Accepted     True      Accepted     The Gateway has been scheduled by Envoy Gateway                            1                     2024-09-17 11:24:27 +0200 CEST
envoy-gateway   gateway/internal   Programmed   True      Programmed   Address assigned to the Gateway, 1/1 envoy Deployment replicas available   1                     2024-09-17 11:28:25 +0200 CEST
                                   Accepted     True      Accepted     The Gateway has been scheduled by Envoy Gateway                            1                     2024-09-17 11:25:23 +0200 CEST

NAMESPACE   NAME               TYPE           STATUS    REASON         MESSAGE                                            OBSERVED GENERATION   LAST TRANSITION TIME
dev-ops     httproute/webapi   ResolvedRefs   True      ResolvedRefs   Resolved all the Object references for the Route   1                     2024-09-17 11:34:22 +0200 CEST
                               Accepted       True      Accepted       Route is accepted                                  1                     2024-09-17 11:34:22 +0200 CEST

Error: no matches for kind "BackendTLSPolicy" in version "gateway.networking.k8s.io/v1alpha2"
no matches for kind "BackendTLSPolicy" in version "gateway.networking.k8s.io/v1alpha2"
arkodg commented 2 days ago

another bug in the area of parentRef status https://github.com/envoyproxy/gateway/issues/4127