Closed zhoujoetan closed 4 months ago
/triage accepted /assign @rexagod
@zhoujoetan try excluding configmaps
and secrets
from the list of exported resources (--resources=
command line option).
At least for me it dropped initial memory usage from ~400Mib to 24Mib.
Both CLI and helm chart have them included by default.
In my case Helm charts (which store the manifests in secrets by defaults) history were the main culprit for the I could not confirm is pagination is used which may mitigate this issue.
Hope this helps.
kube-state-metrics version: v2.3.0
@zhoujoetan It seems you're on an outdated version that's no longer supported. Could you switch to one of the supported versions (preferably the latest release) and verify this issue still persists for you?
I have figured out the issue. We had a ton of configmaps objects that KSM read during startup time. Trimming those objects brought the memory usage back down. I am closing the issue now.
@zhoujoetan When you sam Configmap objects, you mean this cluste-rwide or was it something specific? Also when you say trimming was it like deleting unwanted CMs, or removing data from these CMs?
What happened: A few of our
kube-state-metrics
instance (single-instance, no sharding) recently had OOM issues after restart. The memory usage spiked up to 2.5GB (see attachment) for a few minutes before stabilized at 131 MB. We tried to increase the CPU limit from the default 0.1 to 1 or even 5, but it does not seem to help much.Here is the pprof profile I captured:
Looks like heap memory usage does not represent 100% of the
container_memory_usage_bytes
metric.What you expected to happen: memory usage to not spike 20x at restart
How to reproduce it (as minimally and precisely as possible): Kill/restart the KSM pod.
Anything else we need to know?:
Environment:
kubectl version
):v1.24.17-eks-8cb36c9