Traefik routes and SSL broken since 2.9.9_17.0.28

oblivioncth commented 1 year ago

App Name

Traefik

SCALE Version

22.12.2

App Version

2.9.9_17.031

Application Events


2023-04-25 18:10:15
Scaled down replica set traefik-7c875b446f to 0 from 1
2023-04-25 18:10:15
Deleted pod: traefik-7c875b446f-rvttg
2023-04-25 18:10:15
Stopping container traefik
2023-04-25 18:10:15
Readiness probe failed: dial tcp 172.16.0.99:9000: connect: connection refused
2023-04-25 18:09:56
Add eth0 [172.16.0.101/16] from ix-net
2023-04-25 18:09:56
Container image "tccr.io/truecharts/traefik:2.9.9@sha256:f4211d7c735677653b19d4b1be78434913161ed33bace2733d9b98609c61dfa9" already present on machine
2023-04-25 18:09:56
Created container traefik
2023-04-25 18:09:56
Started container traefik
2023-04-25 18:09:55
Job completed
2023-04-25 18:09:55
Scaled up replica set traefik-79475585bf to 1
2023-04-25 18:09:55
Created pod: traefik-79475585bf-9czjs
2023-04-25 18:09:55
Successfully assigned ix-traefik/traefik-79475585bf-9czjs to ix-truenas
2023-04-25 18:08:17
Successfully pulled image "tccr.io/truecharts/kubectl:v1.26.0@sha256:323ab7aa3e7ce84c024df79d0f364282c1135499298f54be2ade46508a116c4b" in 4.190850365s
2023-04-25 18:08:17
Created container traefik-manifests
2023-04-25 18:08:17
Started container traefik-manifests
2023-04-25 18:08:13
Add eth0 [172.16.0.100/16] from ix-net
2023-04-25 18:08:13
Pulling image "tccr.io/truecharts/kubectl:v1.26.0@sha256:323ab7aa3e7ce84c024df79d0f364282c1135499298f54be2ade46508a116c4b"
2023-04-25 18:08:12
Created pod: traefik-manifests-xrscg
2023-04-25 18:08:12
Successfully assigned ix-traefik/traefik-manifests-xrscg to ix-truenas

Application Logs

2023-04-25 22:17:53.737203+00:00E0425 18:17:53.737141       1 reflector.go:138] k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ix-traefik:traefik" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

failed to create fsnotify watcher: too many open files

Application Configuration

As far as I'm aware, default other than:

websecure Accept Forwarded Headers (Enabled): Local and K3 IP ranges as trusted
Use of middlewares:
- forwardAuth (with authentik)
- chain
- rateLimit
- ipWhiteList

I am using TrueNAS generated certificates and not cert-manager generated ones, though some others with this issue are using cert-manager and are still experiencing the same behavior.

Describe the bug

On the affected Traefik chart versions there are two issues:

SSL certificates are not served correctly, resulting in an "Invalid SSL certificate 526 error when accessing services externally via Cloudflare for example, or NET::ERR_CERT_AUTHORITY_INVALID in chromium based browsers when accessing them locally
Routes are broken in some fashion despite no errors shown in the Traefik dashboard. Bypassing the invalid certifcate warning screen mentioned above leads to a 404 page not found screen. This is true regardless of whether the service is in-cluster or an external-service.

The above log errors are produced periodically when on the affected versions, and seem like they may correlate with starting/stopping a chart that uses ingress.

To Reproduce

Update to Trafeik > 17.0.27
Observe non-functional ingress
Rollback to 17.0.26 (.27 doesn't seem to be available) and everything works again

Expected Behavior

Ingress routes and SSL certificate serving function correctly.

Screenshots

N/A

Additional Context

This was intended to be fixed via the container revert in the 17.0.29 release but the issue has persisted, at least for some, since then.

I've read and agree with the following

[X] I've checked all open and closed issues and my issue is not there.

jehos commented 1 year ago

I'm not sure if it's relevant, but it can't look up ingressclasses on traefik with version 26 or higher.

see: https://www.truenas.com/community/threads/traefk-truecharts-failed-to-install.108356/#post-749062

larseggert commented 1 year ago

I just installed TrueCharts on my new TrueNAS. I installed traefik per the instructions, which seemed to work. Then I tried to install external-service per the instructions, but that didn't even start. Looking at the traefik logs, I see the same error mentioned above several times, i.e.,

Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ix-traefik:traefik" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

(I don't get the message about "too many open files", however.)

PrivatePuffin commented 1 year ago

I'm closing this issue has it's a mixupped mess we cannot work from. As it mixes multiple completely unrelated things into one.

The issues in 17.0.28 are solved in 17.0.29

Sadly enough a typo was made, which will be fixed shortly as well.

oblivioncth commented 1 year ago

Well, regardless of which issues manifested in which versions, both of the problems stated in the original submission (broken routes and certificate serving) appear to be resolved as of 17.0.32 as everything seems to be working for me again. Seems the typo was the cause of both.

truecharts / public