saturation of pvc thanos receive

[BalighRezgui]

Hello !

I have a prometheus configured to send the metrics in thanos receive is stored in an s3 bucket with thanos store, everything works well I have metrics but the problem I have a saturation of PVC of thanos receive despite the fact that the metrics are stored in bucket s3.

the saturation path on the thanos receiver pods :

/var/thanos/receive/57236109-c0e3-4b53-8700-30232dbb073d

Do you have a solution for this?

What you expected to happen:

the metrics stored in the bucket without PVC saturation of thanos receive

Full logs to relevant components:

• receive args :

    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - thanos-receive
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - observatorium-xyz
              namespaces:
              - observatorium
              topologyKey: kubernetes.io/hostname
            weight: 100
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - thanos-receive
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - observatorium-xyz
              namespaces:
              - observatorium
              topologyKey: topology.kubernetes.io/zone
            weight: 100
      containers:
      - args:
        - receive
        - --log.level=info
        - --log.format=logfmt
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --receive.replication-factor=1
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=4d
        - --label=replica="$(NAME)"
        - --label=receive="true"
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --receive.local-endpoint=$(NAME).observatorium-xyz-thanos-receive-default.$(NAMESPACE).svc.cluster.local:10901
        - --receive.hashrings-file=/var/lib/thanos-receive/hashrings.json
        env:
        - name: NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: HOST_IP_ADDRESS
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos.yaml
              name: thanos-objectstorage
        image: quay.io/thanos/thanos:v0.24.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-receive
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 10902
          name: http
        - containerPort: 19291
          name: remote-write
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
        resources: {}
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/thanos/receive
          name: data
          readOnly: false
        - mountPath: /var/lib/thanos-receive
          name: hashring-config
      nodeSelector:
        kubernetes.io/os: linux
      securityContext: {}
      serviceAccountName: observatorium-xyz-thanos-receive
      terminationGracePeriodSeconds: 900
      volumes:
      - configMap:
          name: observatorium-xyz-thanos-receive-controller-tenants-generated
        name: hashring-config
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/component: database-write-hashring
        app.kubernetes.io/instance: observatorium-xyz
        app.kubernetes.io/name: thanos-receive
        app.kubernetes.io/part-of: observatorium
        controller.receive.thanos.io/hashring: default
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi

• hashrings.json

    [
      {
        "hashring": "default",
        "tenants": [

        ]
      }
    ]

• Prometheus args:

    spec:
      containers:
        - name: prometheus
          image: quay.io/prometheus/prometheus:latest
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: monitoring-msp-ca
              mountPath: /etc/prometheus/ca
            - name: prometheus-client-certs
              mountPath: /etc/prometheus/certs
            - name: prometheus-storage-volume
              mountPath: /prometheus/

• prometheus remote_write config :

    remote_write:
    - url: https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive
      remote_timeout: 30s
      tls_config:
        ca_file: /etc/prometheus/ca/tls.crt
        cert_file: /etc/prometheus/certs/tls.crt
        key_file: /etc/prometheus/certs/tls.key

• Receiver Logs: level=warn ts=2023-09-28T14:28:08.357778834Z caller=writer.go:139 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" numDropped=403 level=warn ts=2023-09-28T14:28:08.27034935Z caller=writer.go:139 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" numDropped=404 level=warn ts=2023-09-28T14:28:08.365816977Z caller=writer.go:139 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" numDropped=402 level=warn ts=2023-09-28T14:28:08.366437304Z caller=writer.go:139 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" numDropped=423 level=warn ts=2023-09-28T14:28:08.37529256Z caller=writer.go:139 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" level=info ts=2023-09-29T11:00:56.102529666Z caller=shipper.go:338 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="upload new block" id=01HBG9GVB2XH5MHHAMW78FQQ0H level=info ts=2023-09-29T11:01:42.236238599Z caller=head.go:979 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="WAL checkpoint complete" first=287 last=318 duration=53.323604649s level=info ts=2023-09-29T13:00:44.549333922Z caller=compact.go:518 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="write block" mint=1695981600057 maxt=1695988800000 ulid=01HBGGCJK0Z8D74JAYZTQ9FY2K duration=44.068442394s level=info ts=2023-09-29T13:00:46.641966711Z caller=head.go:810 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="Head GC completed" duration=2.078566563s level=info ts=2023-09-29T13:00:46.918423414Z caller=checkpoint.go:98 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="Creating checkpoint" from_segment=319 to_segment=350 mint=1695988800000 level=info ts=2023-09-29T13:00:56.083433627Z caller=shipper.go:338 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="upload new block" id=01HBGGCJK0Z8D74JAYZTQ9FY2K level=info ts=2023-09-29T13:01:26.165763324Z caller=head.go:979 component=receive component=multi-tsdb tenant=57236109-c0e3-4b53-8700-30232dbb073d msg="WAL checkpoint complete" first=319 last=350 duration=39.247924883s

• Prometheus logs : ts=2023-09-29T13:20:27.521Z caller=dedupe.go:112 component=remote level=warn remote_name=c8c068 url=https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 503 Service Unavailable: Timed out while waiting for a pending request to complete." ts=2023-09-29T13:21:36.181Z caller=dedupe.go:112 component=remote level=warn remote_name=c8c068 url=https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 503 Service Unavailable: Timed out while waiting for a pending request to complete." ts=2023-09-29T13:22:32.356Z caller=dedupe.go:112 component=remote level=warn remote_name=c8c068 url=https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 502 Bad Gateway: " ts=2023-09-29T13:22:40.861Z caller=dedupe.go:112 component=remote level=warn remote_name=c8c068 url=https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 503 Service Unavailable: Timed out while waiting for a pending request to complete." ts=2023-09-29T13:22:50.494Z caller=compact.go:523 level=info component=tsdb msg="write block" mint=1695992400057 maxt=1695993300000 ulid=01HBGHP1R2FHSG6HK83X1PB5S1 duration=11.003609169s ts=2023-09-29T13:22:51.071Z caller=head.go:1298 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=573.373072ms ts=2023-09-29T13:22:51.100Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=436 to_segment=439 mint=1695993300000 ts=2023-09-29T13:22:56.986Z caller=head.go:1266 level=info component=tsdb msg="WAL checkpoint complete" first=436 last=439 duration=5.886065902s ts=2023-09-29T13:23:42.103Z caller=dedupe.go:112 component=remote level=warn remote_name=c8c068 url=https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 503 Service Unavailable: Timed out while waiting for a pending request to complete." ts=2023-09-29T13:28:48.050Z caller=dedupe.go:112 component=remote level=warn remote_name=c8c068 url=https://observatorium-xyz-observatorium-api.msp-monitoring-stack.svc.cluster.local:8080/api/metrics/v1/default/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 503 Service Unavailable: Timed out while waiting for a pending

[MichaHoffmann]

How is Prometheus receiver configured? You can probably lower tsdb retention to save on storage space