Closed McFuzz89 closed 4 years ago
read the same article, having the same issue. I think the resources need to be adjusted, for the stateful set "prometheus-k8s", I'm seeing Memory Usage 180.5Mi, Requested 450Mi, and Limit 50Mi. CPU is slightly over the requested & limit of 200m at 238.61m.
Exactly, this might be because the amount of memory Prometheus is using. If you are using a memory limited device like a RPi with 1 or 2GB RAM it might be the problem. I don't think there is a way to limit it's memory usage.
Closing as it's not a monitoring stack issue.
I'm using an 8gb pi, do you know if the script is setting the limits in the manifests? It's not a limit I set.
By default, Prometheus has a Resource request for 400Mi set in the upstream library (github.com/coreos/kube-prometheus/jsonnet/kube-prometheus/prometheus/prometheus.libsonnet). You might try to override it and test but it's out of scope here.
read the same article, having the same issue. I think the resources need to be adjusted, for the stateful set "prometheus-k8s", I'm seeing Memory Usage 180.5Mi, Requested 450Mi, and Limit 50Mi. CPU is slightly over the requested & limit of 200m at 238.61m.
How did you check the utilization? When I try to do kubectl top pod - it fails :(
edit: modified prometheus-prometheus.yaml to add some limits and that seems, at least for the moment, to have done the trick as the pod does not crash anymore...
resources:
requests:
memory: 400Mi
cpu: 300m
limits:
memory: 600Mi
cpu: 400m
@McFuzz89 I use https://infra.app/ for monitoring my homelab cluster
Yeah those increased limits were something I was hoping we could fix in this project for others, but seems like we should just be making them after the initial setup is done. I also made some changes to the generated files to enable anonymous mode in grafana for a kiosk in my office, and move the CPU temperature towards the top to fit. The grafana.ini file is base64 encoded in the grafana-configmap, but you can easily decode->modify->encode and run kubectl apply -f grafana-configmap.yaml
@rossmckelvie - thanks for the tip! Based on historical data, it seems prometheus crashes after about a day of operation - so come tomorrow evening my time, I should know if the changes worked.
@McFuzz89 any luck?
Hello!
Disclaimer: total kubernetes noob; followed Jeff Geerlings guide on deploying prometheus and grafana on a RPI4 cluster as outlined in his blog.
When I did my original deployment, I noticed an issue where grafana was showing no data in the dashboard. I thought it was my fault as I was messing around with things. Decided to do a fresh deployment by deleting the entire monitoring namespace (where everything for cluster monitoring was setup) and re-deploying.
Everything worked fine but several days ago, it started doing the same constant crash. Please see pod description below which seems to also provide the error associated with the crash (Exit Code 2). Would appreciate assistance in resolving or, if any more data is needed - please let me know! Thanks!!