v0.41.1 memory leak

[smoug25]

Describe the bug Pod memory consumption constantly irace until OOMKilled

To Reproduce Steps to reproduce the behavior:

1. Start mongodb exporter version 0.41.1 with args:

   --web.listen-address="9216" --collector.diagnosticdata # Enable collecting metrics from getDiagnosticData --collector.replicasetstatus # Enable collecting metrics from replSetGetStatus --collector.dbstats # Enable collecting metrics from dbStats --collector.dbstatsfreestorage # Enable collecting free space metrics from dbStats --collector.topmetrics # Enable collecting metrics from top admin command --collector.currentopmetrics # Enable collecting metrics currentop admin command --collector.indexstats # Enable collecting metrics from $indexStats --collector.collstats # Enable collecting metrics from $collStats --collector.fcv # Enable Feature Compatibility Version collector --collector.pbm # Enable collecting metrics from Percona Backup --mongodb.global-conn-pool --discovering-mode --compatible-mode

2. Wait until pod will be killed by kubernetes

Expected behavior

Pod mongodb exporter is not killed by Kubernetes

Logs dmesg.txt

Environment

• OS Linux
• environment k8s
• MongoDB version 7.0.14-8

[BupycHuk]

Hi @smoug25 could you provide logs from mongodb_exporter? in logs you attached I see no messages from mongodb_exporter itself. How often do you request metrics from mongodb_exporter? Could you provide full steps to reproduce including how do you scrape metrics from exporter

[smoug25]

Hi @BupycHuk Logs from mongo_exporter:

mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f-prometheus-molvmg6.txt

I have 3 node mongodb setup and run mongo_exporten for each mongodb node

Whis is mongo_exporter deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-mongodb-exporter
    app.kubernetes.io/version: 0.41.0
    argocd.argoproj.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
    helm.sh/chart: prometheus-mongodb-exporter-3.7.0
  name: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f-prometheus-mongodb
  namespace: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
      app.kubernetes.io/name: prometheus-mongodb-exporter
  template:
    metadata:
      annotations:
        checksum/config: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      labels:
        app.kubernetes.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
        app.kubernetes.io/name: prometheus-mongodb-exporter
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: statefulset.kubernetes.io/pod-name
                    operator: In
                    values:
                      - mongodb-prod-green-cluster-rs0-1
              topologyKey: kubernetes.io/hostname
      containers:
        - args:
            - '--web.listen-address=:9216'
            - '--collector.diagnosticdata'
            - '--collector.replicasetstatus'
            - '--collector.dbstats'
            - '--collector.dbstatsfreestorage'
            - '--collector.topmetrics'
            - '--collector.currentopmetrics'
            - '--collector.indexstats'
            - '--collector.collstats'
            - '--collector.fcv'
            - '--collector.pbm'
            - '--mongodb.global-conn-pool'
            - '--discovering-mode'
            - '--compatible-mode'
          env:
            - name: MONGODB_URI
              valueFrom:
                secretKeyRef:
                  key: uri
                  name: mongodb-rs0-1-monitoring-uri
          image: 'percona/mongodb_exporter:0.41.1'
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /
              port: metrics
            initialDelaySeconds: 10
          name: mongodb-exporter
          ports:
            - containerPort: 9216
              name: metrics
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /
              port: metrics
            initialDelaySeconds: 10
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 500m
              memory: 1024Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - all
            readOnlyRootFilesystem: true
            runAsGroup: 10000
            runAsNonRoot: true
            runAsUser: 10000
          volumeMounts: []
      imagePullSecrets: []
      nodeSelector:
        doks.digitalocean.com/node-pool: mongodb
      serviceAccountName: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f-prometheus-mongodb
      terminationGracePeriodSeconds: 30
      volumes: []

Whis is mongo_exporter service manifest:

apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-mongodb-exporter
    app.kubernetes.io/version: 0.41.0
    argocd.argoproj.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
    helm.sh/chart: prometheus-mongodb-exporter-3.7.0
  name: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f-prometheus-mongodb
  namespace: mongodb
spec:
  ports:
    - name: metrics
      port: 9216
      protocol: TCP
      targetPort: metrics
  selector:
    app.kubernetes.io/instance: mongodb-exp-rs0-1-prod-green-fra1-fd108fc69f
    app.kubernetes.io/name: prometheus-mongodb-exporter
  type: ClusterIP

Whis is mongo_exporter service monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: mongodb-metrics
    argocd.argoproj.io/instance: prometheus-resources-prod-green-fra1-fd108fc69f
  name: mongodb-metrics
  namespace: monitoring
spec:
  endpoints:
    - port: metrics
      relabelings:
        - action: labeldrop
          regex: instance
        - sourceLabels:
            - __meta_kubernetes_pod_node_name
          targetLabel: node_name
        - sourceLabels:
            - __meta_kubernetes_pod_node_name
          targetLabel: instance
  namespaceSelector:
    matchNames:
      - mongodb
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-mongodb-exporter

I scrape data every 5 sec.

[elafontaine]

I'm curious, but if you disable the PBM collector, is the memory leak still there ?

[smoug25]

I'm curious, but if you disable the PBM collector, is the memory leak still there ?

I'll check and back to you with answer

[smoug25]

@elafontaine Disabling pbm collector is normaliced memory consumption