grafana / k8s-monitoring-helm

Apache License 2.0
228 stars 88 forks source link

Set default resource requests & limit in helm chart #753

Open nmehlei opened 2 months ago

nmehlei commented 2 months ago

Hello,

The helm chart is currently not setting resource requests & limits for the various deployments (e.g. alloy, kepler, opencost, prometheus-node-exporter, etc.). This can result in unbounded resource usage.

Should the chart set reasonable values here?

Alternatively, if for any reason this should not be set by default, then it would be useful to have an example on how to set these limits for each of the deployments within the chart values/parameters.

skl commented 2 months ago

At least for Alloy, the memory consumption is roughly proportional to the number of metric series (8-12 KiB RAM/series + whatever minimum is required by Alloy) - which in turn depends on the size of your cluster and the number of Alloy replicas in the StatefulSet.

OpenCost is likely to be affected by the size of the cluster, too.

The DaemonSets might be easier to predict, though.

I agree that it's a good idea to set a default and/or have an example 👍

RobFone commented 1 week ago

I would personally prefer to see guidance in the documentation. For example the information in the previous post about memory-consumed-per-series. Resources depend so much on the environment and the node size (for CPU resource where the unit is a proportion of a core) that choosing meaningful defaults feels virtually impossible. If a default memory limit is set too low then in some environments that could even lead to pods getting stuck restarting where they get terminated for being over their memory limit.