Closed peedrr closed 2 years ago
andrewgkew
commented
2 years ago I have just tested it with helm 3.6.0 and kubectl 1.22.1 (client) 1.20.7 (Server) and it works without a problem. I will upgrade my versions and try again, could be a newer version issue maybe...
andrewgkew
commented
2 years ago @peedrr what are you running your cluster on? A cloud provider or locally? Just looking your k8s server version is 1.22 which is currently not available on EKS at the moment, want to be able to replicate your issues.
peedrr
commented
2 years ago Hey @andrewgkew, thank you for looking into this.
I have just tried again (fresh cluster) and the issue seems to have gone away. The only thing I did differently this time was to add the repo to helm first, and then
helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack ...(rather than specifying the repo with the --repo flag). The stack then installed without a hitch. I am not sure if this contributed in some way to the 'fix', but for now, I think we can label this one helm-gremlin and consider it closed.
For completeness, I am using Linode's managed Kubernetes Engine (LKE).
Thanks again.
hackmare
commented
1 year ago I experienced the same issue (digitaloceans) in a help update. Race condition is clearly visible: "@hackmorgane@hmorgane@hmormormorgane@hmorgane@hackmar"
Fix: uninstall, re-insta..
helm upgrade kube-prom-stack prometheus-community/kube-prometheus-stack --version "${HELM_CHART_VERSION}" \
--namespace monitoring \
-f "04-setup-observability/assets/manifests/prom-stack-values-v${HELM_CHART_VERSION}.yaml"
Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/clusterrolebinding.yaml failed: Post "https://d35923a3-16a5-4d14-ae7f-ac68b194c100.k8s.ondigitalocean.com/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?fieldManmorgane@hmorgane@hackmare Kubernetes-Startmorgane@hackmare Kubernetesmorgane@hackmare Kubernetes-Startmorgane@hmorgane@hackmorgane@hmorgane@hmormormorgane@hmorgane@hackmarmorgane@hackmare Kubernetes-Starter-Kitmorgane@hackmarmorgane@hackmare Kmorgane@hackmarmorgane@hmorganmorganmormormormormorganmormormorgane@hackmare Kubernetes-Starter-Kit-Developers % HELM_CHART_VERSION="35.5.1"
UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/clusterrolebinding.yaml
helm upgrade kube-prom-stack prometheus-community/kube-prometheus-stack -f "04-setup-observability/assets/manifests/prom-stack-values-v35.5.1.yaml" --version 35.5.1 --namespace monitoring
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progressWorkaround after too much flailing: delete, re-install
helm delete kube-prom-stack --namespace monitoring
helm delete kube-prom-stack --namespace monitoring
release "kube-prom-stack" uninstalled
helm install kube-prom-stack prometheus-community/kube-prometheus-stack --version "${HELM_CHART_VERSION}" \
--namespace monitoring \
--create-namespace \
-f "04-setup-observability/assets/manifests/prom-stack-values-v${HELM_CHART_VERSION}.yaml"
NAME: kube-prom-stack
LAST DEPLOYED: Tue Jan 3 02:58:05 2023
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace monitoring get pods -l "release=kube-prom-stack"
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
morgane@hackmare Kubernetes-Starter-Kit-Developers % I hope this works for others...
OMARoun
commented
1 year ago Yes, I really appreciate it. Your solution has worked very well for me
bharath-dev-1
commented
1 year ago Worked for me as well, thanks.
nistiwar
commented
5 months ago I am facing similar issue with no clear indication as to why? Details:
$ helm version
version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"clean", GoVersion:"go1.21.9"}
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26+", GitVersion:"v1.26.15-dispatcher", GitCommit:"5490d28d307425a9b05773554bd5c037dbf3d492", GitTreeState:"clean", BuildDate:"2024-04-18T22:39:05Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.5-gke.1200", GitCommit:"4eb57372b0f0ac0023caca078161b75185febeef", GitTreeState:"clean", BuildDate:"2023-06-01T19:55:11Z", GoVersion:"go1.19.9 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
$ helm install prometheus-stack prometheus-community/kube-prometheus-stack -n prometheus-monitoring
Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
* timed out waiting for the condition
$ helm search repo prometheus-community/kube-prometheus-stack
NAME CHART VERSION APP VERSION DESCRIPTION
prometheus-community/kube-prometheus-stack 58.6.0 v0.73.2 kube-prometheus-stack collects Kubernetes manif...
Please note that the setup is a GKE Autopilot cluster
nistiwar
commented
5 months ago The above error, specifically this one, can be fixed by disabling prometheus admissionwebhook, But will fail further in installation for GKE Autopilot. Could not make it work so far by disabling these :
--set prometheusOperator.admissionWebhooks.enabled=false
--set nodeExporter.enabled=false
--set kube-prometheus-stack.coreDns.enabled=false
--set kube-prometheus-stack.kubeControllerManager.enabled=false
--set kube-prometheus-stack.kubeDns.enabled=false
--set kube-prometheus-stack.kubeEtcd.enabled=false
--set kube-prometheus-stack.kubeProxy.enabled=false
--set kube-prometheus-stack.kubeScheduler.enabled=falseStill continue to get the following errors:
$ helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --set prometheusOperator.admissionWebhooks.enabled=false --set nodeExporter.enabled=false --set kube-prometheus-stack.coreDns.enabled=false --set kube-prometheus-stack.kubeControllerManager.enabled=false --set kube-prometheus-stack.kubeDns.enabled=false --set kube-prometheus-stack.kubeEtcd.enabled=false --set kube-prometheus-stack.kubeProxy.enabled=false --set kube-prometheus-stack.kubeScheduler.enabled=false -n monitoring --create-namespace
Error: INSTALLATION FAILED: 5 errors occurred:
* services is forbidden: User "nishith.tiwari@abc.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied
* services is forbidden: User "nishith.tiwari@abc.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied
* services is forbidden: User "nishith.tiwari@abc.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied
* services is forbidden: User "nishith.tiwari@abc.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied
* services is forbidden: User "nishith.tiwari@abc.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied
Describe the bug a clear and concise description of what the bug is.
kube-prometheus-stack will not install. I am learning Kubernetes and had been following a tutorial video from Linode. The procedure for installing was previously working fine (I successfully installed 5 or 6 times over the past few days) but since yesterday evening, using a fresh cluster and the same commands as before, I cannot install
What's your helm version?
version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.17.3"}
What's your kubectl version?
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:42:41Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Which chart?
kube-prometheus-stack
What's the chart version?
27.1.0
What happened?
I ran the same
helm upgrade --installcommand as before (on a fresh k8s cluster)What you expected to happen?
Installation, as before
How to reproduce it?
Enter the changed values of values.yaml?
NONE
Enter the command that you execute and failing/misfunctioning.
Anything else we need to know?
I am new to kubernetes/helm and I am very open to this being some kind of simple user error, so please go easy on me!