envoyproxy / gateway

Manages Envoy Proxy as a Standalone or Kubernetes-based Application Gateway
https://gateway.envoyproxy.io
Apache License 2.0
1.53k stars 328 forks source link

Unsupported listener protocol type should not cause an error while updating Gateway Status #4216

Open vlasov-y opened 6 days ago

vlasov-y commented 6 days ago

Description: If I try to create GRPC and HTTPS listeners in one gateway, they are not created

Repro steps:

# GatewayClass
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-proxy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: default
    namespace: system
---
# EnvoyProxy
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: default
  namespace: system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          lbipam.cilium.io/sharing-key: default
          lbipam.cilium.io/sharing-cross-namespace: "*"
        externalTrafficPolicy: Cluster
---
# Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
  name: https
  namespace: system
spec:
  gatewayClassName: envoy-proxy
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: grpc.example.com
    name: grpc.example.com
    port: 443
    protocol: GRPC
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: https
      mode: Terminate
  - allowedRoutes:
      namespaces:
        from: All
    hostname: example.com
    name: example.com
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: https
      mode: Terminate
  - allowedRoutes:
      namespaces:
        from: All
    hostname: '*.example.com'
    name: wildcard.example.com
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: https
      mode: Terminate
status:
  addresses:
  - type: IPAddress
    value: 1.2.3.4
  conditions:
  - lastTransitionTime: "2024-09-11T14:22:48Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 4
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-09-11T14:22:48Z"
    message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
    observedGeneration: 4
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 4
    conditions:
    - lastTransitionTime: "2024-09-11T14:22:48Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 4
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-09-11T14:22:48Z"
      message: Listener has been successfully translated
      observedGeneration: 4
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-11T14:22:48Z"
      message: Listener references have been resolved
      observedGeneration: 4
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: example.com
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
  - attachedRoutes: 4
    conditions:
    - lastTransitionTime: "2024-09-11T14:22:48Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 4
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-09-11T14:22:48Z"
      message: Listener has been successfully translated
      observedGeneration: 4
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-11T14:22:48Z"
      message: Listener references have been resolved
      observedGeneration: 4
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: wildcard.example.com
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute
---
# GRPC Route
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
    external-dns.alpha.kubernetes.io/enabled: "true"
    external-dns.alpha.kubernetes.io/ttl: "1"
  name: app
  namespace: system
spec:
  hostnames:
  - grpc.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: https
    namespace: system
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: app
      port: 80
      weight: 1
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-09-11T14:29:15Z"
      message: There are no ready listeners for this parent ref
      observedGeneration: 1
      reason: NoReadyListeners
      status: "False"
      type: Accepted
    - lastTransitionTime: "2024-09-11T14:29:15Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: https
      namespace: system

Environment: envoy-gateway v1.1.0 (from manifests) Talos v1.7.6 Cilium v1.16.1 Kubernetes v1.30.3 arm64

Logs:

2024-09-11T14:33:07.422Z    ERROR    provider    kubernetes/status_updater.go:114    unable to update status
{"runner": "provider", "name": "https", "namespace": "system", "error": "Gateway.gateway.networking.k8s.io \"https\" is invalid:
[listeners[0].supportedKinds: Required value, <nil>: Invalid value: \"null\":
some validation rules were not checked because the object was invalid;
correct the existing errors to complete validation]"}
arkodg commented 6 days ago

hey looks like there are 2 issues here

1) protocol: GRPC . This is an invalid value and should be changed to HTTPS https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.ProtocolType and will fix your issue

2)

{"runner": "provider", "name": "https", "namespace": "system", "error": "Gateway.gateway.networking.k8s.io \"https\" is invalid:
[listeners[0].supportedKinds: Required value, <nil>: Invalid value: \"null\":
some validation rules were not checked because the object was invalid;

this is probably failing for the grpc.example.com listener, due to some data interpretation error of nil and null

Required value, <nil>: Invalid value: \"null\"

keeping this issue open to resolve this

vlasov-y commented 6 days ago

hey looks like there are 2 issues here

  1. protocol: GRPC . This is an invalid value and should be changed to HTTPS https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.ProtocolType and will fix your issue
{"runner": "provider", "name": "https", "namespace": "system", "error": "Gateway.gateway.networking.k8s.io \"https\" is invalid:
[listeners[0].supportedKinds: Required value, <nil>: Invalid value: \"null\":
some validation rules were not checked because the object was invalid;

this is probably failing for the grpc.example.com listener, due to some data interpretation error of nil and null

Required value, <nil>: Invalid value: \"null\"

keeping this issue open to resolve this

Thanks, changing of protocol helped.