Dashboard starts returning 404 after 28.2.0 update

mmeier86 commented 3 months ago

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What version of the Traefik's Helm Chart are you using?

v28.3.0

What version of Traefik are you using?

v3.0.2

What did you do?

I upgraded the Helm chart to v28.2.0 (and today to v28.3.0) and realized today that the Traefik dashboard suddenly started to return 404 not found, even though many other IngressRoute and Ingress manifests still work perfectly fine.

What did you see instead?

Dashboard returns 404.

What is your environment & configuration?

I'm deploying the Helm chart on my Kubernetes cluster, with the following relevant values.yaml values:

deployment:
  healthchecksPort: 4435
  podLabels:
    homelab/ingress: "true"
additionalArguments:
  - "--ping.entryPoint=health"
  - "--providers.kubernetesingress.ingressendpoint.hostname=ingress-k8s.mei-home.net"
  - "--providers.kubernetescrd.allowCrossNamespace=true"
ingressRoute:
  dashboard:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: null
    entryPoints:
      - admin
    middlewares:
      - name: admin-basic-auth
        namespace: traefik-ingress
  admin:
    port: 4434
    exposedPort: 4434
    expose:
      default: true
    protocol: TCP
    tls:
      enabled: true
    middlewares:
      - traefik-ingress-headers-security@kubernetescrd
      - traefik-ingress-local-net@kubernetescrd

I'm not changing the ingressClass value at all.

Additional Information

I do not see any entries related to the dashboard route during Traefik startup in the logs, but I do see log entries for all of the other IngressRoutes and Ingresses which do work. When I delete and re-add the dashboard IngressRoute created by the Helm chart, the only log entry that I see which fits the timing is this one:

level=debug providerName=kubernetescrd caller=github.com/traefik/traefik/v3/pkg/provider/kubernetes/crd/kubernetes.go:177 message="Skipping Kubernetes event kind *v1alpha1.IngressRoute"

The traefik-ingress-dashboard resource created by the Helm chart looks like this:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    meta.helm.sh/release-name: traefik-ingress
    meta.helm.sh/release-namespace: traefik-ingress
    kubernetes.io/ingress.class: traefik-ingress
  creationTimestamp: "2024-06-20T21:38:57Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: traefik-ingress-traefik-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-28.3.0
    homelab/part-of: traefik-ingress
  name: traefik-ingress-dashboard
  namespace: traefik-ingress
  resourceVersion: "72257444"
  uid: ce8958e7-6ebd-4a26-b396-9a97f38a8089
spec:
  entryPoints:
  - admin
  routes:
  - kind: Rule
    match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    middlewares:
    - name: admin-basic-auth
      namespace: traefik-ingress
    services:
    - kind: TraefikService
      name: api@internal

If I make a copy of it and remove the kubernetes.io/ingress.class annotation, Traefik happily creates the router and I can access the dashboard again.

I believe the culprit might be PR #1079. This sets the ingress class for the route to the traefik.fullname value. But the IngressClass for the Kubernetes CRD provider does not seem to get set in the chart, and so it uses Traefik's default "Otherwise, resources missing the annotation, having an empty value, or the value traefik are processed." value as documented. And because the annotation for the dashboard route is now traefik-ingress, from traefik.fullname, it is neither missing, nor empty, nor "traefik", and so the IngressRoute for the dashboard is ignored.

mloiseleur commented 3 months ago

@samox73 Wdyt ?

oscrx commented 3 months ago

I have finally debugged the same issue in our clusters and found this issue.

The docs state:

only Ingresses containing an annotation with the same value are processed. Otherwise, Ingresses missing the annotation, having an empty value, or the value traefik are processed.

Defaulting to traefik.fullname is not the correct behaviour IMHO

I think there are 2 possible solutions

Defaulting to "traefik" which is the default when no ingress class is defined.
Only including the annotation when .Values.providers.kubernetesCRD.ingressClass is set.

samox73 commented 3 months ago

@oscrx the IngressClass that is deployed uses the fullname as a default, if .ingressClass.name is not set: https://github.com/traefik/traefik-helm-chart/blob/42e574516560492db94f7ada7c2a61c391603df3/traefik/templates/ingressclass.yaml#L9 The annotation on the dashboard IngressRoute is set in the same way: https://github.com/traefik/traefik-helm-chart/blob/42e574516560492db94f7ada7c2a61c391603df3/traefik/templates/dashboard-ingressroute.yaml#L9 While it is true that this is a regression, annotations for setting the IngressClass on chart-internal IngressRoutes should be set by the chart and not manually in the values file. Your fix in https://github.com/traefik/traefik-helm-chart/pull/1098 addresses the concern that the dashboard IngressRoute should have the same IngressClass as specified in .Values.providers.kubernetesCRD.ingressClass, which is passed as an argument to the deployment.

@mloiseleur However, I cannot quite comprehend why one would want to create an IngressClass via .Values.ingressClass and use a different IngressClass name in .Value.providers.kubernetesCRD or .Value.providers.kubernetesIngress. I believe it would be clearer, if .Values.ingressClass.name (or the default of the provisioned IngressClass) would be used as an argument to --providers.kubernetescrd.ingressClass and --providers.kubernetesingress.ingressClass instead. I understand that traefik enables the user to set these values independently. However, I cannot think of a use case where I would want to use both K8s Ingresses and IngressRoutes with different names than the provisioned IngressClass from .Values.ingressClass.name.

oscrx commented 3 months ago

Yeah that is exactly what made me wondering and I tried to express that feeling in the PR.

This is one of the possible fixes for it. But honestly it made me doubt if the implementation of custom ingressClass names are implemented correctly in this chart...

I chose .Values.providers.kubernetesCRD.ingressClass specifically because the annotation on the dashboard is set on an ingressRoute and not on a native ingress. Setting .Value.providers.kubernetesIngress or .Values.ingressClass.name will not configure Traefik to handle the annotation on the dashboard correctly. I don't have a lot of experience on actually using the functionality but it feels like you need to configure all 3 for correct Traefik behaviour.

IMHO .Values.ingressClass.name should be the only place where an ingressClass for a specific traefik instance is configured. But doing that without a breaking change is/feels hard. But it could be my understanding of the functionality as well..

In the PR I mainly focussed on fixing the regression in the dashboard functionality.

traefik / traefik-helm-chart