Helm Upgrade - 2.5.0 MiddlewaresTCP Missing

evanreichard commented 3 years ago

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What version of the Traefik's Helm Chart are you using?

traefik-10.3.0

What version of Traefik are you using?

2.5.0

What did you do?

Automated helm upgrade to Traefik 2.5.0

What did you see instead?

Logs:

Failed to watch *v1alpha1.MiddlewareTCP: failed to list *v1alpha1.MiddlewareTCP: the server could not find the requested resource (get middlewaretcps.traefik.containo.us)

All IngressRoutes failing (404)

What is your environment & configuration?

k8s v1.19.4 (k3os)

Additional Information

Added the new CRD and things started working again:

kubectl apply -f https://raw.githubusercontent.com/traefik/traefik-helm-chart/master/traefik/crds/middlewarestcp.yaml

Not sure if this because my environment, but for whatever reason it didn't add the new MiddlewaresTCP CRD. Manually adding fixed it.

jakubhajek commented 3 years ago

hello @evanreichard

Would you please let me know what tool do you use for automated Helm deployment and how the CRD deployment is managed by that tool?

budimanjojo commented 3 years ago

I have the same problem here, After updating the crds to v10.3.0 I can no longer access all my ingressroutes. Here's the log in traefik pod:

E0819 12:24:29.250516       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.0/tools/cache/reflector.go:167: Faile
d to watch *v1alpha1.MiddlewareTCP: failed to list *v1alpha1.MiddlewareTCP: middlewaretcps.traefik.containo.us is fo
rbidden: User "system:serviceaccount:ingress-system:traefik" cannot list resource "middlewaretcps" in API group "tra
efik.containo.us" at the cluster scope

I'm using flux and renovatebot only to update CRDs from this repository. This is my deployment manifest (updated to traefik 2.5.0): https://github.com/budimanjojo/home-cluster/blob/main/cluster/core/traefik/deployment.yaml and here is the CRDs deployment: https://github.com/budimanjojo/home-cluster/blob/main/cluster/crds/traefik/gitrepository-fluxCRD.yaml

budimanjojo commented 3 years ago

Sorry, seems like I know what's wrong with mine, it's because I haven't update my service account to use the CRDs yet. I have the middlewaresTCP but the service account is forbidden to use it. Please just ignore my post :)

jakubhajek commented 3 years ago

Hello!

This is a commonly known Helm issue that there is no support for upgrading or deleting CRD.

The problem you were facing was related to the lack of the new CRD: Middleware TCP that has been introduced together with the new 2.5 release. That CRD has to be available on a cluster once you deploy Traefik

For FluxCD I use the following code to manage CRDs for Traefik. I hope that helps:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: traefik-crds
  namespace: flux-system
spec:
  interval: 30m
  url: https://github.com/traefik/traefik-helm-chart.git
  ref:
    tag: v10.3.0
  ignore: |
    # exclude all
    /*
    # path to crds
    !/traefik/crds/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: traefik-api-crds
  namespace: flux-system
spec:
  interval: 15m
  prune: false
  sourceRef:
    kind: GitRepository
    name: traefik-crds
    namespace: flux-system
  healthChecks:
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: ingressroutes.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: ingressroutetcps.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: ingressrouteudps.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: middlewares.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: middlewaretcps.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: serverstransports.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: tlsoptions.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: tlsstores.traefik.containo.us
  - apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: traefikservices.traefik.containo.us

jasaltvik commented 3 years ago

I'm sorry if this is not the right place, and that it should be asked in a different issue (or forum), but I hit the same error using Helmsman for deploying the chart. In Helmsman you can add hooks (e.g. preInstall/preUpgrade, and use that to install/update CRDs before the chart is deployed), and it would be a lot simpler if you released a CRD YAML for each chart release (that contained all CRDs for Traefik). As an example, cert-manager does this (direct link), which means it's possible to just apply that to ensure everything is up-to-date before the chart install/upgrade. It is of course possible to work around this by running a local script instead, but it would be so nice if we just needed to update the chart version, and all CRD updates would be handled without having to maintain something else. There might be gotchas here that I haven't thought about though :)

jakubhajek commented 3 years ago

hello @kozejonaz

Thanks a lot for suggesting that, we will discuss that internally and keep you posted.

evanreichard commented 3 years ago

Thanks @jakubhajek - Yes I'm using the FluxCD Helm Operator. Appreciate you dropping your Kustomization 👍

This can be closed as it's a documented Helm limitation and not a chart issue. (I wasn't aware of that limitation)

jakubhajek commented 3 years ago

Hi @kozejonaz

I have been thinking about your proposal. Here are some of my thoughts in regards that topic.

Helmsman for deploying the chart. In Helmsman you can add hooks (e.g. preInstall/preUpgrade,

I am not familiar with Helmsman but from what I've just read about it is a tool that helps you to manage Helm code from a source code. In my example, I presented how to achieve a similar approach using FluxCD which also automates K8S configuration manifests, Helm Releases from Git. It introduces Helm Operator to manage Helm Releases.

You need to first add CRDs using the example from my previous comment. The CRDs are stored at our Helm Repo and we can directly refer to each file and applying it using kustomization file.

Then you can create the Helm Release of the Traefik Chart but use the builtin-in Flux feature dependsOn to tell Flux to create the CRD items before releasing Traefik or any other Helm Chart that relays on CRD.

That will prevent us from facing problems when. you deployed Traefik but there are no CRDs available on a cluster.

I will try to create a fully working scenario to introduces the way of releasing Traefik with Flux:

by our Helm Chart
and by manually created Kubernetes manifests.

Thanks once again for your comment :)

jasaltvik commented 3 years ago

@jakubhajek Thanks for the quick reply!

I am not familiar with Helmsman but from what I've just read about it is a tool that helps you to manage Helm code from a source code. In my example, I presented how to achieve a similar approach using FluxCD which also automates K8S configuration manifests, Helm Releases from Git. It introduces Helm Operator to manage Helm Releases.

You need to first add CRDs using the example from my previous comment. The CRDs are stored at our Helm Repo and we can directly refer to each file and applying it using kustomization file.

Then you can create the Helm Release of the Traefik Chart but use the builtin-in Flux feature dependsOn to tell Flux to create the CRD items before releasing Traefik or any other Helm Chart that relays on CRD.

Yeah, I'm a big fan of the GitOps approach! Hopefully we'll be able to migrate to this approach soon, and your example looks like a good way to handle this in an automated way. I understand that this is not something that you want to spend time on. It might be useful to be able to deploy a single YAML that you always know includes the CRDs necessary to install a certain version of Traefik though. This could also be useful with the GitOps approach, to avoid having to update the kustomization file.

Again, thanks for the detailed reply :)

MisterTimn commented 3 years ago

Just ran into this same issue when upgrading. Maybe the Readme can be extended with some upgrade notes, similar to how the prometheus-community handles the upgrades of the kube-prometheus-stack helm chart. https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-17x-to-18x

They list the necessary CRDs to update between versions allong with kubectl apply directives with raw github links.

jakubhajek commented 3 years ago

Thanks a lot for your notes, we will definitely add that kind of information to future releases.

Startouf commented 3 years ago

I came upon this issue later, I happened to make a gist just like the prometheus readme that helped me migrate from 2.4 to 2.5 with almost no downtime (helm update through a terragrunt apply)

https://gist.github.com/Startouf/bd961cee307a6ad93aba51f082f4b7f6

kevinpollet commented 3 years ago

Fixed by https://github.com/traefik/traefik-helm-chart/pull/443

traefik / traefik-helm-chart