search

traefik / traefik

The Cloud Native Application Proxy
https://traefik.io
MIT License
51.27k stars 5.1k forks source link

Gateway API TLSRoutes appear to be handled in invalid scenarios #11112

Closed travisghansen closed 1 month ago

travisghansen commented 1 month ago

Welcome!

What did you do?

Consider the following set of assets:

    - apiVersion: gateway.networking.k8s.io/v1
      kind: Gateway
      metadata:
        name: eastwestgateway
        labels:
          topology.istio.io/network: {{ .Values.cluster.globalID }}
          # default is 15443, can override with this label
          # networking.istio.io/gatewayPort: 35443
      spec:
        gatewayClassName: istio
        listeners:
        - name: istiod-grpc
          port: 15012
          protocol: TLS
          tls:
            mode: Passthrough
        - name: istiod-webhook
          port: 15017
          protocol: TLS
          tls:
            mode: Passthrough
        - name: tls-passthrough
          hostname: "*.local"
          port: 15443
          protocol: TLS
          tls:
            mode: Passthrough
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      kind: TLSRoute
      metadata:
        name: eastwestgateway-grpc
      spec:
        parentRefs:
        - group: gateway.networking.k8s.io
          name: eastwestgateway
          kind: Gateway
          sectionName: istiod-grpc
        rules:
        - backendRefs:
          - group: ''
            kind: Service
            name: istiod
            port: 15012
            weight: 1
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      kind: TLSRoute
      metadata:
        name: eastwestgateway-webhook
      spec:
        parentRefs:
        - group: gateway.networking.k8s.io
          name: eastwestgateway
          kind: Gateway
          sectionName: istiod-webhook
        rules:
        - backendRefs:
          - group: ''
            kind: Service
            name: istiod
            port: 15017
            weight: 1

When applied it appears traefik is attempting to handle the tlsroutes even though the parent gateways does not include a traefik-managed gateway.

apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: TLSRoute
  metadata:
    creationTimestamp: "2024-09-20T15:48:22Z"
    generation: 1
    labels:
      app: raw
      argocd.argoproj.io/instance: dev-na01-workload-00-istio-system
      chart: raw-0.2.5
      heritage: Helm
      release: istio-system-common
    name: eastwestgateway-grpc
    namespace: istio-system
    resourceVersion: "1630261085"
    uid: d743d191-b7e3-4649-9a09-d3ea1e6edf4e
  spec:
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eastwestgateway
      sectionName: istiod-grpc
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: istiod
        port: 15012
        weight: 1
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2024-09-20T15:48:23Z"
        message: ""
        observedGeneration: 1
        reason: NoMatchingParent
        status: "False"
        type: Accepted
      controllerName: traefik.io/gateway-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eastwestgateway
        sectionName: istiod-grpc
- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: TLSRoute
  metadata:
    creationTimestamp: "2024-09-20T15:48:22Z"
    generation: 1
    labels:
      app: raw
      argocd.argoproj.io/instance: dev-na01-workload-00-istio-system
      chart: raw-0.2.5
      heritage: Helm
      release: istio-system-common
    name: eastwestgateway-webhook
    namespace: istio-system
    resourceVersion: "1630261084"
    uid: 28937a7e-a5e5-48cb-bc57-3012e55d9c9d
  spec:
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eastwestgateway
      sectionName: istiod-webhook
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: istiod
        port: 15017
        weight: 1
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2024-09-20T15:48:23Z"
        message: ""
        observedGeneration: 1
        reason: NoMatchingParent
        status: "False"
        type: Accepted
      controllerName: traefik.io/gateway-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: eastwestgateway
        sectionName: istiod-webhook
kind: List
metadata:
  resourceVersion: ""

What did you see instead?

Should I expect traefik to touch these assets at all given they are not associated with a traefik-managed gateway?

What version of Traefik are you using?

3.1.3

What is your environment & configuration?

# kubectl get gatewayclasses.gateway.networking.k8s.io traefik-internal -o yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  creationTimestamp: "2024-07-03T21:17:14Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: traefik-internal-adc-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    argocd.argoproj.io/instance: dev-na01-workload-00-adc-system
    class: traefik-internal
    helm.sh/chart: traefik-31.0.0
  name: traefik-internal
  resourceVersion: "1622704718"
  uid: 76bc08c5-07c6-4520-8bf7-4d5e93e2b297
spec:
  controllerName: traefik.io/gateway-controller
status:
  conditions:
  - lastTransitionTime: "2024-07-03T21:17:27Z"
    message: Handled by Traefik controller
    observedGeneration: 1
    reason: Handled
    status: "True"
    type: Accepted

Add more configuration information here.

        - --providers.kubernetesgateway.labelselector=class=traefik-internal

If applicable, please paste the log output in DEBUG level

No response

travisghansen commented 1 month ago

Also note, it appears traefik is messing with the gateway itself as well:

# kubectl -n istio-system get gateway eastwestgateway -o yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    gateway.istio.io/controller-version: "5"
  creationTimestamp: "2024-09-20T15:48:22Z"
  generation: 1
  labels:
    app: raw
    argocd.argoproj.io/instance: dev-na01-workload-00-istio-system
    chart: raw-0.2.5
    heritage: Helm
    release: istio-system-common
    topology.istio.io/network: dev-na01-workload-00
  name: eastwestgateway
  namespace: istio-system
  resourceVersion: "1630359689"
  uid: 159e342d-7bdf-459f-b0e6-684975fc591b
spec:
  gatewayClassName: istio
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: istiod-grpc
    port: 15012
    protocol: TLS
    tls:
      mode: Passthrough
  - allowedRoutes:
      namespaces:
        from: Same
    name: istiod-webhook
    port: 15017
    protocol: TLS
    tls:
      mode: Passthrough
  - allowedRoutes:
      namespaces:
        from: Same
    hostname: '*.local'
    name: tls-passthrough
    port: 15443
    protocol: TLS
    tls:
      mode: Passthrough
status:
  addresses:
  - type: IPAddress
    value: 172.28.42.32
  conditions:
  - lastTransitionTime: "2024-09-20T16:32:55Z"
    message: Gateway successfully scheduled
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-09-20T16:32:55Z"
    message: Gateway successfully scheduled
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed

Notice the 2 conditions (presumably 1 from traefik and 1 from istio) but the key here is the IPAddress getting set is the traefik svc IP.

  addresses:
  - type: IPAddress
    value: 172.28.42.32

kubectl -n adc-system get svc | grep 'traefik'
traefik-internal                            LoadBalancer   100.127.150.81    172.28.42.32   80:31355/TCP,3661:31245/TCP,443:30371/TCP,443:30997/UDP   426d

That also shouldn't be happening as well correct? Thankfully it appears istio is setting an address somewhere else as well...

kubectl -n istio-system get gateway eastwestgateway NAME              CLASS   ADDRESS        PROGRAMMED   AGE
eastwestgateway   istio   172.28.42.77   True         51m

kubectl -n istio-system get svc
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                           AGE
eastwestgateway-istio   LoadBalancer   100.127.197.10   172.28.42.77   15021:30241/TCP,15012:31728/TCP,15017:30610/TCP,15443:31255/TCP   51m
travisghansen commented 1 month ago

Yeah, traefik is fighting istio and the IP is constantly switching around for gateways with the class set to istio. Conversely, I have dedicated gateway classes for traefik which are being left alone by istio.

rtribotte commented 1 month ago

Hello @travisghansen,

Thanks for reporting this!

We have confirmed the bug and have opened #11170 which should fix this issue.

travisghansen commented 1 month ago

Great! Let me know when it hits a release and I will test it out!

rtribotte commented 1 month ago

Hello @travisghansen,

The PR is merged, you can test the fix with the experimental-v3.2 image until the new v3.2.0-rc2 is released.

I'm closing this issue accordingly.