Ingressclasses not available on namespace scope level

[bschoenbach]

Hi traefik community,

After upgrading of k8s cluster to v1.18+ we get the following error during startup of traefik v2.3+

E0107 06:45:40.895430 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:traefik:traefik" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope`

The problem: Within k8s v1.18+ the "IngressClass" was introduced to avoid bad annotations on "Ingress" resource objects themselves (see here) . However, the idea of that concept is pretty cool but lacks in implementation because it requires access on cluster scope level. Hence, developers or teams which can only work on namespace scope cannot use traefik ingress > 2.3.x on gke > 1.18.x.

Possible solutions could be:

• Introduce a configuration switch to disable the listing/ usage of ingressclasses during startup of traefik v2.3+ OR
• Raise this issue to k8s community so that they can work on the cluster scope issue of "ingressclasses" on k8s v1.18+. In my opinion it must be possible to define ingressclasses also on namespace scope.

Temporary Solution

Currently we downgraded our traefik to v2.2.11 where lisitng of "ingressclasses" is not implemented and treafik starts up correctly also on gke 1.18+

[nustiueudinastea]

Having the same issue when trying to run the latest Helm chart using namespaced deployment.

[joeldeteves]

Just ran into the same issue, running Kubernetes 1.19.3.

Downgrading Traefik for now.

Note: 2.3.7 is working for me, whereas the 2.4.0 upgrade breaks it, there was no need for me to downgrade all the way to v2.2.11 per @bschoenbach's comment

[joeldeteves]

Hello, any update on this issue?

We are unable to patch Traefik for security releases until this is resolved.

Thank you,

[alexangas]

Also experiencing this issue on Kubernetes 1.19.7 with Traefik 2.3.3.

Possibly related issue posted on Stack Overflow but the suggested fix was already in the helm chart.

[rtribotte]

Hello @bschoenbach @nustiueudinastea @joeldeteves @alexangas,

Thanks for your interest in Traefik!

This error can happen because Traefik, as a Kubernetes client, has no rights to fetch resources. As mentioned in the migration guide, when updating to Traefik v2.3+, the support of IngressClasses makes necessary to update the RBACs.

Can you please check that your RBACs granted to Traefik contains:

- apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch

More generally you can find up-to-date RBACs in the documentation in the dynamic-configuration reference for Kubernetes CRD provider page.

[joeldeteves]

Hello @bschoenbach @nustiueudinastea @joeldeteves @alexangas,

Thanks for your interest in Traefik!

This error can happen because Traefik, as a Kubernetes client, has no rights to fetch resources. As mentioned in the migration guide, when updating to Traefik v2.3+, the support of IngressClasses makes necessary to update the RBACs.

Can you please check that your RBACs granted to Traefik contains:

- apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch

More generally you can find up-to-date RBACs in the documentation in the dynamic-configuration reference for Kubernetes CRD provider page.

Thank you, I see the documentation was recently updated to reflect this change: https://doc.traefik.io/traefik/migration/v2/

Note: I also had to add the following to my CRD:

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: serverstransports.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: ServersTransport
    plural: serverstransports
    singular: serverstransport
  scope: Namespaced

This worked for me, I believe this issue can be closed

[alexangas]

Unfortunately I'm still experiencing this issue.

First thing to say is that I'm using the traefik-helm-chart and am increasingly wondering if the issue lies there, specifically this one already logged: https://github.com/traefik/traefik-helm-chart/issues/358. The RBACs for ClusterRole that @rtribotte points to when setting namespaced = false are in the chart, but it does not contain them for Role which is used when namespaced = true. I tried adding them and still had the same error. Happy for this issue to be closed if the problem is the chart.

By the way, I tried updating to the Traefik 2.4.6 version of the chart and found the CRD missing that @joeldeteves provided (thanks!).

[joeldeteves]

Unfortunately I'm still experiencing this issue.

First thing to say is that I'm using the traefik-helm-chart and am increasingly wondering if the issue lies there, specifically this one already logged: traefik/traefik-helm-chart#358. The RBACs for ClusterRole that @rtribotte points to when setting namespaced = false are in the chart, but it does not contain them for Role which is used when namespaced = true. I tried adding them and still had the same error. Happy for this issue to be closed if the problem is the chart.

By the way, I tried updating to the Traefik 2.4.6 version of the chart and found the CRD missing that @joeldeteves provided (thanks!).

That does seem to be a separate issue with the helm chart. I will add my commentary there.

EDIT: I spoke to soon. After I looked at that error I saw it was the exact issue described by @bschoenbach:

The problem: Within k8s v1.18+ the "IngressClass" was introduced to avoid bad annotations on "Ingress" resource objects themselves (see here) . However, the idea of that concept is pretty cool but lacks in implementation because it requires access on cluster scope level. Hence, developers or teams which can only work on namespace scope cannot use traefik ingress > 2.3.x on gke > 1.18.x.

Therefore this issue remains for users who use Traefik at the Namespaced level :(

[bschoenbach]

Sorry guys, i was a little bit backed up by other k8s topics. However, Yes this issue remains for users on namespace scope. I raised this issue to the community themselves because the root-cause is placed within "ingressclasses" resource of k8s >= 1.18. See the issue below:

https://github.com/kubernetes/kubernetes/issues/99824

In the meantime I tried several other ingress-controllers (nginx etc.). The problem occurs, too.

[traefiker]

Hi! I'm Træfiker :robot: the bot in charge of tidying up the issues.

I have to close this one because of its lack of activity :disappointed:

Feel free to re-open it or join our Community Forum.

[YitzyD]

Hello, we are too stuck using v2.2.X versions of traefik for namespaced deployments - has there been any effort to add a flag to the controller to disable the use of ingressClass resources in namespaced mode?

[rtribotte]

Hello @joeldeteves @bschoenbach @YitzyD,

Traefik started to support the Kubernetes IngressClass resources in v2.3.0.

Thus, while this is the preferred way to scope the Ingress provider ingresses discovery process, it is still possible to rely on annotations.

For any Traefik upgrade in a Kubernetes cluster, the necessary actions to take are (depending on the provider):

• Kubernetes Ingress provider: update RBACs
• Kubernetes IngressRoute provider: update RBACs and CRDs

The reason is that without the appropriate rights needed to look up for resources in the cluster, nor the definitions of those resources (Kubernetes IngressRoute CRDs), Traefik will fail to start the providers.

In case one is using Traefik HelmChart, upgrading it to the version supporting the targeted Traefik version should work.

Regarding the use of ingressClass in namespace mode, this is not supported because the ingressClass resource is by definition a cluster scoped resource. Thus, as previously said, this is not mandatory to use an ingressClass resource with the Kubernetes Ingress provider.

Therefore, as this issue turned out to be a configuration issue, and as we think we addressed all the questions, we are going close this issue. Feel free to reopen if you think we missed something.

[bschoenbach]

Hi,

Problem still remains. As long as traefik controller scrapes for ingressClass resources during startup the proposed solution (updating RBACs etc) does not work on namespace scope. As mentioned by @YitzyD implementation of a dedicated configuration switch is still necessary to avoid scraping ingressClass resources (i.e. listing of all ingressClass objects via kube api during startup procedure within traefik) when using old annotation scheme with kubernetes ingress provider instead of ingressClass.

@rtribotte please re-open to keep discussion ongoing

[rtribotte]

Hello @bschoenbach,

Thanks for your feedback.

I understand that you are requesting an option to disable the ingressClass feature globally.

Nonetheless, would defining a cluster role with only rights to discover ingressClasses for Traefik be an acceptable/sufficient solution? What would be the downside of that?

I'm reopening the issue to discuss further the proposal to solve that use case.

[bschoenbach]

Hi @rtribotte

Nonetheless, would defining a cluster role with only rights to discover ingressClasses for Traefik be an acceptable/sufficient solution? What would be the downside of that?

Unfortunately not, we (our company) work on namespace scope only. This means we are not authorized to create cluster roles/rolebindings. For sure we could ask our k8s system administrators to do so but this will the end up to work directly on cluster scope level. Besides, this would be a global setting for the whole cluster (valid for all namespaces). Hence, a dedicated configuration switch to disable ingressClass scraping would be very helpful for the namespace scope.

[rtribotte]

Thanks for the explanation, we think it makes sense and we are then labeling the issue as an enhancement.

Unfortunately, this would not make it to our roadmap for a while as we are focused elsewhere. If you or another community member would like to build it, let us know and we will work with you to make sure you have all the information needed so that it can be merged.

[YitzyD]

I agree with the necessity. Especially with the namespace "scope" becoming GA in Ingress Classes in 1.23, but without dedicated namespaced rbac, a flag to disable listing the clusterscoped Ingress Class in the controller is required for backwards compatability. If namespaced rbac is added for Ingress Classes in the future that would be the best solution but does not seem to be planned.

[jandillenkofer]

Hi, I am interested in contributing to this issue. Here are the two commits I needed to implement this feature. I added a new flag DisableIngressClassLookup inside the k8s ingress provider. If this flag is enabled, traefik does not watch for ingress class resource changes on the cluster.

We are currently facing this issue on a cluster where we do not have the permission to watch/list ingress-classes.

Can someone suggest me some testcases I need to check for? I am also not sure if I should open a PR yet or if I should wait until after I implemented the required testcases for the new feature. Thanks in advance!

[mpl]

Hi @jandillenkofer ,

yes, in general the PR is more likely to get reviews if it does include relevant tests, so it's better to open it once you have them implemented.

[rtribotte]

Fixed by #9281