Closed ygbingo closed 3 years ago
Same here.
+1
having exactly the same issue, even with all default values.
I'm with
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.13-gke.2600", GitCommit:"fc4bf3b03703b51c48ba123e8abc53b3051ba8a7", GitTreeState:"clean", BuildDate:"2020-11-11T09:20:10Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
update:
We have to follow the suggestions in the thread to delete all CRDs manually to resolve the problem. And yes, there was a small downtime we have to deal with.
context: we were migrating from the helm/stable chart to the new Prometheus chart, and was mainly following the migration guide documented here: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#redeploy-with-new-name-downtime
deleting crds manually is an extra step we have to take to make it work.
I resolved this problem after I delete all config that involes prometheus. Anyone else got this problem, maybe you have install prometheus or got error when install prometheus. You can check your config use this command:
$ kubectl get configmap --all-namespaces
And then, if you got some config's name like "prometheus-***". You could delete that configs and try again.
It looks as though the following CRDs were updated. Reapplying them fixed for me.
and also
Issue is from previous installations of other prometheus-relevant manifest files. As defined in the uninstallation guide, delete the following:
kubectl delete crd alertmanagerconfigs.monitoring.coreos.com kubectl delete crd alertmanagers.monitoring.coreos.com kubectl delete crd podmonitors.monitoring.coreos.com kubectl delete crd probes.monitoring.coreos.com kubectl delete crd prometheuses.monitoring.coreos.com kubectl delete crd prometheusrules.monitoring.coreos.com kubectl delete crd servicemonitors.monitoring.coreos.com kubectl delete crd thanosrulers.monitoring.coreos.com
Dry-run still failed for me after but not the actual deployment... strange.
Warning : deleting all CRDs will delete all instances of those CRDs (i.e all your ServiceMonitors, etc). Probably what you do NOT want to do in production.
Yes deleting the CRDs will result in deleting any servicemonitors and so on, which have previously been created by other charts.
If you read the helm docs here https://helm.sh/docs/chart_best_practices/custom_resource_definitions/ it explains that --dry-run
and upgrading CRDs with helm is not supported at this time.
I believe the solution I posted above is the correct one. We need a note to be added to the upgrade documentation and also the chart needs updating because the CRDs which are deployed with a clean installation are outdated.
Hi @rust84 , what if I am performing helm install prometheus prometheus-community/kube-prometheus-stack
for the first time, how should I apply the updated CRDs?
It looks as though the following CRDs were updated. Reapplying them fixed for me.
and also
delete all crds worked for me, but this way should NOT be used in production.
I deleted all crds (mentioned above) and got somehow a different error message Error: failed pre-install: timed out waiting for the condition So it's still not quite working for me After deleting the crds from above comments, I still see some more. Should I delete them too?
Update: I re-ran the install command again and it worked
Helm Version
🞂🞂 helm version version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}
Kubernetes Version:
🞂🞂 kubectl version --short Client Version: v1.20.2 Server Version: v1.19.0
I deleted all crds (mentioned above) and additionally removed the remaining prometheus-operator-kubelet service as suggested. Still not working for me Error: failed pre-install: timed out waiting for the condition
You might want to try this:
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
Seems like this CRD changed with the new version too, but it is not listed in the changelog.
I actually did:
$ kubectl delete crd alertmanagers.monitoring.coreos.com
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
but very likely the second command will be enough.
@maresja1 thank you! It works like a charm now
I have not such CRD (clean install), though same error.
You shouldn't have to delete CRDs. Deleting CRDs deletes any defined resources that use that CRD and, in a cluster with more than one user, causes great consternation when you delete everybody's alerts and metrics.
CRDs have a version in order to provide a contract with the user. Had the CRD version been bumped, the chart templates could have tested against that version (.Capabilities.APIVersions.Has "prometheus.monitoring.coreos.com/v1"
) and enabled or disabled fields as appropriate for the contract. The solution for the future is to honor that contract and not go changing charts to insert fields when you have no way of knowing if the field is even there. Use proper API version management.
For cli users, you can work around this by just applying the CRDs in the charts crds/
directory using kubectl:
kubectl apply -f crds/
For the rest of us using gitops for continuous deployment, we're just out of luck.
You shouldn't have to delete CRDs. Deleting CRDs deletes any defined resources that use that CRD and, in a cluster with more than one user, causes great consternation when you delete everybody's alerts and metrics.
CRDs have a version in order to provide a contract with the user. Had the CRD version been bumped, the chart templates could have tested against that version (
.Capabilities.APIVersions.Has "prometheus.monitoring.coreos.com/v1"
) and enabled or disabled fields as appropriate for the contract. The solution for the future is to honor that contract and not go changing charts to insert fields when you have no way of knowing if the field is even there. Use proper API version management.For cli users, you can work around this by just applying the CRDs in the charts
crds/
directory using kubectl:kubectl apply -f crds/
For the rest of us using gitops for continuous deployment, we're just out of luck.
Thank you for this solution. Worked for me. My mistake was that I installed from deprecated helm repo. So reapplying crd fixed it!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
I ran into this issue today and workaround suggested by @maresja1 worked for me.
Ran into this issue when upgrading the rancher monitoring helm chart from 9.4.203 to 14.5.100. Applying the CRDs again (without deleting) as suggested by maresja1 made it upgrade fine.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
This issue is being automatically closed due to inactivity.
The fastest/easiest way I found to "clean" the remaining resources was to:
helm template prometheus-community/kube-prometheus-stack -f values.yml | kubectl delete -f -
Error: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]
I did follow @maresja1 & ran the suggested commands:
kubectl delete crd alertmanagers.monitoring.coreos.com customresourcedefinition.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" deleted
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
Then I again installed the Monitoring V2 from Rancher UI, It gave me the following error: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "shards" in com.coreos.monitoring.v1.Prometheus.spec]
I deleted the fields from the yaml file before installing : probeNamespaceSelector, probeSelector, shards. Then Installed again & got the error: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]
I deleted the Monitoring CRD and monitoring from Installed App section in Rancher UI and again Installed it after removing the probeNamespaceSelector & probeSelector fileds. Then I again installed the Monitoring App from Rancher version 2.6.13 and It got successfully installed.
SUCCESS: helm upgrade --install=true --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-100.2.0-up40.1.2.yaml --version=100.2.0+up40.1.2 --wait=true rancher-monitoring /home/shell/helm/rancher-monitoring-100.2.0-up40.1.2.tgz
Pro-Tip: Observe the Error Prompts and try to resolve that.
Describe the bug Excuse, when I install kube-prometheus-stack on azure kubernetes, I got this Error
Version of Helm and Kubernetes:
Helm Version:
Kubernetes Version:
Prequires Actually, I hava add any repo in helm where shows in readme.
So, can anyone give me some suggest? Very tks ~