[install prometheus-kube-stack] Error: unable to build kubernetes objects from release manifest

[ygbingo]

Describe the bug Excuse, when I install kube-prometheus-stack on azure kubernetes, I got this Error

$ helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"62876fc6d93e891aa7fbe19771e6a6c03773b0f7", GitTreeState:"clean", BuildDate:"2020-10-16T20:43:34Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Prequires Actually, I hava add any repo in helm where shows in readme.

$ helm search repo prometheus-community
NAME                                                    CHART VERSION   APP VERSION     DESCRIPTION
prometheus-community/alertmanager                       0.3.0           v0.21.0         The Alertmanager handles alerts sent by client ...
prometheus-community/kube-prometheus-stack              12.11.3         0.44.0          kube-prometheus-stack collects Kubernetes manif...
prometheus-community/prometheus                         13.0.3          2.22.1          Prometheus is a monitoring system and time seri...
prometheus-community/prometheus-adapter                 2.10.1          v0.8.2          A Helm chart for k8s prometheus adapter
prometheus-community/prometheus-blackbox-exporter       4.10.1          0.18.0          Prometheus Blackbox Exporter
prometheus-community/prometheus-cloudwatch-expo...      0.12.1          0.8.0           A Helm chart for prometheus cloudwatch-exporter
prometheus-community/prometheus-consul-exporter         0.4.0           0.4.0           A Helm chart for the Prometheus Consul Exporter
prometheus-community/prometheus-couchdb-exporter        0.2.0           1.0             A Helm chart to export the metrics from couchdb...
prometheus-community/prometheus-druid-exporter          0.9.0           v0.8.0          Druid exporter to monitor druid metrics with Pr...
prometheus-community/prometheus-elasticsearch-e...      4.0.1           1.1.0           Elasticsearch stats exporter for Prometheus
prometheus-community/prometheus-kafka-exporter          0.2.0           v1.2.0          A Helm chart to export the metrics from Kafka i...
prometheus-community/prometheus-mongodb-exporter        2.8.1           v0.10.0         A Prometheus exporter for MongoDB metrics
prometheus-community/prometheus-mysql-exporter          1.0.1           v0.12.1         A Helm chart for prometheus mysql exporter with...
prometheus-community/prometheus-nats-exporter           2.5.1           0.6.2           A Helm chart for prometheus-nats-exporter
prometheus-community/prometheus-node-exporter           1.12.0          1.0.1           A Helm chart for prometheus node-exporter
prometheus-community/prometheus-operator                9.3.2           0.38.1          DEPRECATED - This chart will be renamed. See ht...
prometheus-community/prometheus-postgres-exporter       1.6.0           0.8.0           A Helm chart for prometheus postgres-exporter
prometheus-community/prometheus-pushgateway             1.5.1           1.3.0           A Helm chart for prometheus pushgateway
prometheus-community/prometheus-rabbitmq-exporter       0.5.6           v0.29.0         Rabbitmq metrics exporter for prometheus
prometheus-community/prometheus-redis-exporter          4.0.0           1.11.1          Prometheus exporter for Redis metrics
prometheus-community/prometheus-snmp-exporter           0.1.1           0.19.0          Prometheus SNMP Exporter
prometheus-community/prometheus-stackdriver-exp...      1.6.1           0.6.0           Stackdriver exporter for Prometheus
prometheus-community/prometheus-statsd-exporter         0.2.0           0.18.0          A Helm chart for prometheus stats-exporter
prometheus-community/prometheus-to-sd                   0.3.1           0.5.2           Scrape metrics stored in prometheus format and ...

So, can anyone give me some suggest？ Very tks ~

[mrfelek]

Same here.

[eduardoriveror]

+1

[cheskayang]

having exactly the same issue, even with all default values.

I'm with

version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.13-gke.2600", GitCommit:"fc4bf3b03703b51c48ba123e8abc53b3051ba8a7", GitTreeState:"clean", BuildDate:"2020-11-11T09:20:10Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

---

update:

We have to follow the suggestions in the thread to delete all CRDs manually to resolve the problem. And yes, there was a small downtime we have to deal with.

context: we were migrating from the helm/stable chart to the new Prometheus chart, and was mainly following the migration guide documented here: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#redeploy-with-new-name-downtime

deleting crds manually is an extra step we have to take to make it work.

[ygbingo]

I resolved this problem after I delete all config that involes prometheus. Anyone else got this problem, maybe you have install prometheus or got error when install prometheus. You can check your config use this command:

$ kubectl get configmap --all-namespaces

And then, if you got some config's name like "prometheus-***". You could delete that configs and try again.

[rust84]

It looks as though the following CRDs were updated. Reapplying them fixed for me.

https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml

and also

https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml

[Doug-North]

Issue is from previous installations of other prometheus-relevant manifest files. As defined in the uninstallation guide, delete the following:

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com kubectl delete crd alertmanagers.monitoring.coreos.com kubectl delete crd podmonitors.monitoring.coreos.com kubectl delete crd probes.monitoring.coreos.com kubectl delete crd prometheuses.monitoring.coreos.com kubectl delete crd prometheusrules.monitoring.coreos.com kubectl delete crd servicemonitors.monitoring.coreos.com kubectl delete crd thanosrulers.monitoring.coreos.com

Dry-run still failed for me after but not the actual deployment... strange.

[desaintmartin]

Warning : deleting all CRDs will delete all instances of those CRDs (i.e all your ServiceMonitors, etc). Probably what you do NOT want to do in production.

[rust84]

Yes deleting the CRDs will result in deleting any servicemonitors and so on, which have previously been created by other charts.

If you read the helm docs here https://helm.sh/docs/chart_best_practices/custom_resource_definitions/ it explains that --dry-run and upgrading CRDs with helm is not supported at this time.

I believe the solution I posted above is the correct one. We need a note to be added to the upgrade documentation and also the chart needs updating because the CRDs which are deployed with a clean installation are outdated.

[yashrsharma44]

Hi @rust84 , what if I am performing helm install prometheus prometheus-community/kube-prometheus-stack for the first time, how should I apply the updated CRDs?

It looks as though the following CRDs were updated. Reapplying them fixed for me.

https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml

and also

https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml

[double12gzh]

delete all crds worked for me, but this way should NOT be used in production.

[nbrahimi]

I deleted all crds (mentioned above) and got somehow a different error message Error: failed pre-install: timed out waiting for the condition So it's still not quite working for me After deleting the crds from above comments, I still see some more. Should I delete them too?

Update: I re-ran the install command again and it worked

[mantans]

Helm Version

🞂🞂 helm version version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

Kubernetes Version:

🞂🞂 kubectl version --short Client Version: v1.20.2 Server Version: v1.19.0

I deleted all crds (mentioned above) and additionally removed the remaining prometheus-operator-kubelet service as suggested. Still not working for me Error: failed pre-install: timed out waiting for the condition

[maresja1]

You might want to try this:

kubectl apply -f  https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml

Seems like this CRD changed with the new version too, but it is not listed in the changelog.

I actually did:

$ kubectl delete crd alertmanagers.monitoring.coreos.com
$ kubectl apply -f  https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml

but very likely the second command will be enough.

[mantans]

@maresja1 thank you! It works like a charm now

[dennybaa]

I have not such CRD (clean install), though same error.

[joejulian]

You shouldn't have to delete CRDs. Deleting CRDs deletes any defined resources that use that CRD and, in a cluster with more than one user, causes great consternation when you delete everybody's alerts and metrics.

CRDs have a version in order to provide a contract with the user. Had the CRD version been bumped, the chart templates could have tested against that version (.Capabilities.APIVersions.Has "prometheus.monitoring.coreos.com/v1") and enabled or disabled fields as appropriate for the contract. The solution for the future is to honor that contract and not go changing charts to insert fields when you have no way of knowing if the field is even there. Use proper API version management.

For cli users, you can work around this by just applying the CRDs in the charts crds/ directory using kubectl:

kubectl apply -f crds/

For the rest of us using gitops for continuous deployment, we're just out of luck.

[seunroava]

You shouldn't have to delete CRDs. Deleting CRDs deletes any defined resources that use that CRD and, in a cluster with more than one user, causes great consternation when you delete everybody's alerts and metrics.

CRDs have a version in order to provide a contract with the user. Had the CRD version been bumped, the chart templates could have tested against that version (.Capabilities.APIVersions.Has "prometheus.monitoring.coreos.com/v1") and enabled or disabled fields as appropriate for the contract. The solution for the future is to honor that contract and not go changing charts to insert fields when you have no way of knowing if the field is even there. Use proper API version management.

For cli users, you can work around this by just applying the CRDs in the charts crds/ directory using kubectl:

kubectl apply -f crds/

For the rest of us using gitops for continuous deployment, we're just out of luck.

Thank you for this solution. Worked for me. My mistake was that I installed from deprecated helm repo. So reapplying crd fixed it!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[sdeoras]

I ran into this issue today and workaround suggested by @maresja1 worked for me.

[germanoeich]

Ran into this issue when upgrading the rancher monitoring helm chart from 9.4.203 to 14.5.100. Applying the CRDs again (without deleting) as suggested by maresja1 made it upgrade fine.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[dtourde]

The fastest/easiest way I found to "clean" the remaining resources was to:

helm template prometheus-community/kube-prometheus-stack -f values.yml | kubectl delete -f -

[believerHSP]

Error: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec, ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector" in com.coreos.monitoring.v1.Alertmanager.spec]

I did follow @maresja1 & ran the suggested commands:

kubectl delete crd alertmanagers.monitoring.coreos.com customresourcedefinition.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" deleted

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.45.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created

Then I again installed the Monitoring V2 from Rancher UI, It gave me the following error: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "shards" in com.coreos.monitoring.v1.Prometheus.spec]

I deleted the fields from the yaml file before installing : probeNamespaceSelector, probeSelector, shards. Then Installed again & got the error: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Prometheus.spec): unknown field "probeNamespaceSelector" in com.coreos.monitoring.v1.Prometheus.spec, ValidationError(Prometheus.spec): unknown field "probeSelector" in com.coreos.monitoring.v1.Prometheus.spec]

I deleted the Monitoring CRD and monitoring from Installed App section in Rancher UI and again Installed it after removing the probeNamespaceSelector & probeSelector fileds. Then I again installed the Monitoring App from Rancher version 2.6.13 and It got successfully installed.

SUCCESS: helm upgrade --install=true --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-100.2.0-up40.1.2.yaml --version=100.2.0+up40.1.2 --wait=true rancher-monitoring /home/shell/helm/rancher-monitoring-100.2.0-up40.1.2.tgz

Pro-Tip: Observe the Error Prompts and try to resolve that.