[grafana] last grafana chart release was a breaking change

[Legion2]

in the last grafana chart update new lables where added to the selector of the deployment, however the selector is immutabale, so the helm release upgrade failes.

[andrew-pickin-epi]

Specifically this commit results in the following error:

error updating the resource "grafana":   cannot patch "grafana" with kind Deployment: Deployment.apps "grafana" is invalid: 
spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/component":"grafana", "app.kubernetes.io/instance":"grafana", "app.kubernetes.io/name":"grafana"}, 
MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Consequently this causes Grafana to yield status 503.

[zanhsieh]

@yeplaa Could you take a look at your commit?

[yeplaa]

Indeed it's immutable. it fails on update. Sorry No solution i think to have a breaking change. @zanhsieh , what do we do in this case? Could you rollback my commit?

[johanfleury]

Sorry, just linking to my comment on the PR: https://github.com/grafana/helm-charts/pull/994#issuecomment-1022192208

[waterdrop01]

helm command to revert to previous chart version:

helm upgrade grafana grafana/grafana --version 6.21.0 --values helm-grafana-monitoring-values.yml

[zanhsieh]

Reverted. https://github.com/grafana/helm-charts/pull/999

Thank you @yeplaa .

[waterdrop01]

I think this broke my Traefik somehow?

kubectl describe endpoints grafana -n monitoring gives mes:

Name:         grafana
Namespace:    monitoring
Labels:       app.kubernetes.io/component=grafana
              app.kubernetes.io/instance=grafana
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=grafana
              app.kubernetes.io/version=8.3.0
              helm.sh/chart=grafana-6.17.10
Annotations:  <none>
Subsets:
Events:  <none>

Traefik can't map the service to the endpoint because there is a new Label I suppose? (app.kubernetes.io/component=grafana).

Any idea how to remove this label? Reverting to previous helm revision (chart version) doesn't help :(

Thanks

[waterdrop01]

Well I don't know happened, but I can't access my Grafana dashboard anymore.

Traefik routing is broken. As a result of running helm upgrade grafana grafana/grafana --version 6.21.1 --values helm-grafana-monitoring-values.yml.

As we can see by running kubectl get endpoints -n monitoring

NAME                            ENDPOINTS                                               AGE
grafana                         <none>                                                  7m19s

! Reverting to a previous chart version doesn't help.

[Legion2]

Deleting the deployment and recreating it with a helm upgrade should work.

[waterdrop01]

Thanks @Legion2 for your help.

Here is what I did:

helm uninstall grafana --keep-history --namespace=monitoring
release "grafana" uninstalled

Then

helm upgrade grafana grafana/grafana --version 6.21.2 --namespace=monitoring --values helm-grafana-monitoring-values.yml

But it had the following error: Error: UPGRADE FAILED: "grafana" has no deployed releases, So I tried:

helm install grafana grafana/grafana --version 6.21.0 --namespace=monitoring --values helm-grafana-monitoring-values.yml

But it had the following error: Error: cannot re-use a name that is still in use, so I tried using --replace:

helm install grafana grafana/grafana --version 6.21.0 --namespace=monitoring --values helm-grafana-monitoring-values.yml --replace

This time it worked.

PROBLEM, I lost all my dashboards! :( days of work gone. can't believe it

[waterdrop01]

well I have NO IDEA why the PVC was deleted. I just asked on the helm github repo (link).

but frankly, apart from being terribly angry, this leads me to the following conclusions:

1. how is it possible to have pushed a new helm chart version (6.21.2) like this without any testing? Just running the helm upgrade command shows that adding a component label will make the upgrade FAIL right away.
2. Why there is NO EASY WAY to backup grafana dashboards?? Hell this is so a pain to backup it and to be able to commit the changes in a repo. I must have missed something here? and it's probably not the right repo to discuss the issue.

ofc I take my part of responsibility. I shouldn't have tried helm uninstall without a back up of grafana. But this issue says that helm uninstall DOESNT REMOVE PVC.

too naive. once again :(

[nalanj]

I noticed this issue was reverted in 6.22.2, but despite that being released and built it's not up in the repository yet.

[xairos]

I noticed this issue was reverted in 6.22.2, but despite that being released and built it's not up in the repository yet.

It seems that chart-releaser released 6.21.2 to the gh-pages branch (which backs https://grafana.github.io/helm-charts) in 5b6a15fe4340faf3a464b740e4c686868efa879f.

But then, for some reason, chart-releaser reverted the change to add that release: 49073c615cfd1cda4873f16a952d7fcd8539c77e.

[alexjplant]

The easiest fix for this is to delete the Service then update the Helm release; we're using the Terraform Helm provider so we just updated the values to trigger a re-deploy and subsequent re-creation of the Service. This worked for us after a revert of the upgrade didn't bring the Endpoints back