search

carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 200 forks source link

Prometheus high on CPU #62

Closed FlomoN closed 4 years ago

FlomoN commented 4 years ago

Hey, I was wondering if there was a way or any steps one could take to reduce the CPU load of the Prometheus Pod. Maybe I'm also doing something wrong, not too sure.

I've got a K3s cluster with one RPI 4 4GB running as the master and 3 3B+ as workers. When I deploy the monitoring stack (following @geerlingguy 's tutorial) to my cluster most of the pods get scheduled to the master node.

In effect, the master node is constantly spiking from 20% CPU up to 50% (sometimes higher) CPU usage (I assume every time Prometheus is scraping), and in fact, Prometheus is the cause of almost all of this CPU usage.

geerlingguy commented 4 years ago

@FlomoN Fyi I don't schedule pods on the master node to keep it a little more happy; see my notes in the usage section: https://github.com/geerlingguy/turing-pi-cluster#usage

carlosedp commented 4 years ago

Keep the taint in the master node to avoid scheduling non-essential workloads on it. Other than that there's nothing much to be done. Maybe increase the scrapeInterval (search for similar issues) to ease-up Prom.