[kube-prometheus-stack] PVC is not being used for Prometheus statefulsets

[asamudrala1]

Describe the bug a clear and concise description of what the bug is.

Hi, I have used prometheus-snmp-exporter-5.1.0 version charts to deploy prometheus. Below attached is the config file where the pvc is included. I see the PVC getting created and mounting to path: /prometheus but it is not using it. The pod memory is growing to 4gb and failing all the pods -n that particular node due to insufficient memory. prometheus-statefulsets.txt

What's your helm version?

version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

What's your kubectl version?

Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.3-eks-adc7111

Which chart?

kube-prometheus-stack

What's the chart version?

prometheus-snmp-exporter-5.1.0

What happened?

Below attached is the config file where the pvc is included. I see the PVC getting created and mounting to path: /prometheus but it is not using it. The pod memory is growing to 4gb and failing all the pods -n that particular node due to insufficient memory.

What you expected to happen?

It is expected to used the /prometheus mount path and the pvc which is created to be used so the pod memory wont raise to 4GB and above

How to reproduce it?

No response

Enter the changed values of values.yaml?

• patch: | apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus-kube-prometheus-prometheus namespace: monitoring spec: tolerations:
  • key: "nodepool" operator: "Equal" value: "applications" effect: "NoSchedule" resources: limits: cpu: 300m memory: 1Gi requests: cpu: 150m memory: 512Mi storage: volumeClaimTemplate: spec: accessModes:
    • ReadWriteOnce resources: requests: storage: 50Gi storageClassName: ebs-sc

Enter the command that you execute and failing/misfunctioning.

I used kustomization to install prometheus through Argo CD

Anything else we need to know?

No response

[asamudrala1]

What I observed is /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 is causing the increasing in the memory usage in the pod. /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 need to be mounted to the PVC which is created.