Open AndreasMurk opened 7 months ago
I got this error because the pod couldn't write to the persistent storage location.
same issue here
repositories:
- name: prometheus-community
url: https://prometheus-community.github.io/helm-charts
releases:
- name: kube-prometheus-stack
namespace: monitoring
chart: prometheus-community/kube-prometheus-stack
version: 56.20.0
installed: true
values:
- values.yaml
I got this error because the pod couldn't write to the persistent storage location.
I can confirm this. I was setting grafana.containerSecurityContext.readOnlyRootFilesystem: true
, which was causing the problem.
Removing this fixed for me.
It seems the container only needs to write to /tmp
, so a better solution could be to mount only /tmp
as writable instead. But I haven't tested this yet.
Describe the bug a clear and concise description of what the bug is.
Hi!
I have deployed the
kube-prometheus-stack
using FluxCD with the latest 56.6.2 version.Prometheus along with Loki works fine. However, Grafana has some problems after a while.
It lasted approximately 60 minutes to start up fully until all migrations have been done. Then, whenever I make changes in the Dashboard (eg. adding a new data source) the pod fails. After inspecting the logs I have found these error messages:
The pod tries to restart but fails with the aformentioned bug. In Lens it always says:
Readiness probe failed: Get "http://192.168.1.247:3000/api/health": dial tcp 192.168.1.247:3000: connect: connection refused
What's your helm version?
3.14.0
What's your kubectl version?
1.29.1
Which chart?
kube-prometheus-stack
What's the chart version?
56.6.2
What happened?
Making changes in the Dashboard (eg. adding new data sources such as Loki) fails with the stated Python error.
What I have also encountered is that since the newest release, the Dashboard seems slower than with previous releases.
What you expected to happen?
Dashboard should correctly set the datasource
How to reproduce it?
values.yaml
Enter the changed values of values.yaml?
prometheus: ingress: enabled: true annotations: cert-manager.io/cluster-issuer: "letsencrypt-issuer" kubernetes.io/ingressClassName: nginx nginx.ingress.kubernetes.io/service-upstream: "true"
Enter the command that you execute and failing/misfunctioning.
helm install prometheus prometheus-community/kube-prometheus-stack --values values.yaml
Anything else we need to know?
No response