Open sebadob opened 1 month ago
I have the same setup with Cert Manager and I also see this behaviour in the newly released 3.0.1, I saw it since at least 3.0.0-rc3
. I see 404 in the Traefik logs on requests to /.well-known/acme-challenge/.*
paths. My workaround is similar, I copy the generated rule to my IngressRoute
. Fortunately I have only a handful of certificates to manage.
I don't think it's related to cert-manager at all, just to the default ingress class for kubernetes.
I just have this problem with cert-manager, because it creates the default ingress instead of the IngressRoute
CRD.
Yes that's an issue with IngressRoute.
As Traefik v3 changes the apiVersion from traefik.containo.us/v1alpha1
to traefik.io/v1alpha1
. Previous IngressRoutes are ignored.
I wrote a little migration script (note: I'm considering here that the IR and the namespace name are the same)
# Create a new IR with the proper API
for irname in $(kubectl get ingressroutes.traefik.containo.us --all-namespaces -o json | jq '.items | .[] | .metadata.name' -r); do kubectl -n ${irname} get ingressroute ${irname} -oyaml | grep -v -e "uid:" -e "resourceVersion:" -e "generation:" -e "creationTimestamp" | sed 's/.containo.us/.io/' | kubectl -n ${irname} apply -f -; done
# Delete the previous one
for irname in $(kubectl get ingressroutes.traefik.containo.us --all-namespaces -o json | jq '.items | .[] | .metadata.name' -r); do kubectl -n ${irname} delete ingressroutes.traefik.containo.us ${irname}; done
EDIT: This impact all others resources (Middleware, IngressRouteTCP...) not only Ingressroute. The following command can help listing the resources using the previous api:
for res in $(kubectl api-resources --api-group=traefik.containo.us | awk '{ print $1 }'); do echo "== ${res}.traefik.containo.us =="; kubectl get $res.traefik.containo.us --all-namespaces; done
Yes that's an issue with IngressRoute.
No it's not, all the IngressRoute
s work fine after the migration. It's the default Ingress
class not being taken into account for the routing.
Welcome!
What did you do?
I was using Traefik v2 for a long time together with
cert-manager
inside Kubernetes.For my usual routes, I am using the
IngressRoute
CRD which works perfectly fine again after I followed the migration guide.However, all my existing
cert-manager
integrations and automatic certificate renewals in the whole cluster stopped working silently. Theacme-solver
s created by thecert-manager
are default Kubernetes Ingress and they have not changed at all. I just noticed this today after a certificate could not be renewed for a very long time and gut stuck.The Ingress they create is nothing special and as mentioned this has not changed, I have not done any other updates to the cluster than Traefik v3.0 since this problem came up.
What did you see instead?
It seems that the Ingress route is simply ignored. I tried by removing all existing
IngressRoutes
in the same namespace and only leaving the auto-created Ingress for a testing certificate. I tried to manually debug the whole flow and see where it gets stuck and it was for sure Traefik v3.0.I am using the Helm Chart to deploy Traefik and
providers.kubernetesIngress.enabled
is set to true and is the default value in there.The auto-created spec looks like this:
I usually have https-redirects for each host name and as soon as I added the IngressRoute again, I got back the HTTP 301 from it, even though I should have gotten the acme challenge. But as mentioned even without this redirect route I alway received a
404 page not found
.I am using a very nasty workaround everywhere now so I can at least get my certificates renewed. This is simply ignoring the auto-created Ingres resources and I manually added the following IngressRoute definitions to at least make it working again:
This solution is very brittle and tedious though.
I was very lucky that the failed certificate over night was on some staging environment instead of production.
What version of Traefik are you using?
Helm Chart version:
traefik-28.0.0
App-Version:
3.0.0
What is your environment & configuration?
Custom
values.yaml
:If applicable, please paste the log output in DEBUG level
No response