remote write into to thanos-receive for a while will encountered decreased amounts of Prometheus

Hi Team

I'd like to follow up symptoms in our landscape. We adopted thanos-receive to achieve collected come from each kubernetes prometheus metrics for a while. Then we through metrics which count(count(up{app="prometheus"}) by (cluster)) to retrieve amounts of prometheus up in time interval. Some prometheus for a while would be encountered error via cli stern -n cic-system prometheus-prometheus-0 | grep 'component=remote level=error' msg prometheus-prometheus-0 prometheus ts=2023-04-21T03:32:34.904Z caller=dedupe.go:112 component=remote level=error remote_name=11063c url=http://thanos-receive.thanos.xxxxxxxxxx/api/v1/receive msg="non-recoverable error" count=800 exemplarCount=0 err="server returned HTTP status 409 Conflict: forwarding request to endpoint thanos-receive-2.thanos-receive.thanos.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = store locally for endpoint thanos-receive-2.thanos-receive.thanos.svc.cluster.local:10901: add 1 samples: duplicate sample for timestamp" caused the remote write metrics pending. We also updated remote_write in prometheus requests with HTTP headers to be multi-tsdb to receive local TSDB. Also each tenant folders be created. As far as i know should be resolved issue? right?

Through cli to amounts of tenants(prometheus by each cluster). We have 5 replicas of receive pod. The below as once get tenants counts

for idx in `seq 0 4`; do echo "-> List tenant folders: /var/thanos/receive in thanos-receive-${idx} \n\n"; k exec -it thanos-receive-$idx -n thanos -- sh -c 'ls /var/thanos/receive'; echo "\n thanos-receive-${idx} count: `k exec -it thanos-receive-$idx -n thanos -- sh -c 'ls /var/thanos/receive | wc -l'`"; done

...
...
...
thanos-receive-${idx} count: 56

For a while we through dashboard to get metrics count(count(up{app="prometheus"}) by (cluster)) will decreased counts

Expected results

Should be stability counts 56 not happend decreased. Unless some prometheus unhealthy.

Revision

thanos: v0.31.0
prometheus: v2.42.0

manifest of receive

- args:
  - receive
  - --grpc-address=0.0.0.0:10901
  - --http-address=0.0.0.0:10902
  - --remote-write.address=0.0.0.0:19291
  - --receive.replication-factor=1
  - --receive.hashrings-file-refresh-interval=1m
  - --receive.hashrings-algorithm=ketama
  - --objstore.config-file=/etc/thanos/objectstorage.yaml
  - --tsdb.path=/var/thanos/receive
  - --tsdb.retention=6h
  - --tsdb.out-of-order.time-window=2h
  - --receive.tenant-header=THANOS-TENANT
  - --receive-forward-timeout=5m
  - --label=receive_replica="$(NAME)"
  - --label=receive="true"
  - --tsdb.no-lockfile
  - --tsdb.allow-overlapping-blocks
  - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json
  - --receive.local-endpoint=$(NAME).thanos-receive.thanos.svc.cluster.local:10901

manifest of prometheus

  remoteWrite:
  - headers:
      THANOS-TENANT: cs-dev
    queueConfig:
      batchSendDeadline: 30s
      capacity: 5000
      maxBackoff: 5s
      maxSamplesPerSend: 800
      maxShards: 150
      minBackoff: 30ms
      minShards: 8
      retryOnRateLimit: true
    remoteTimeout: 30s
    url: http://thanos-receive.xxxxxxxxxxx/api/v1/receive

PVC information

$ k df-pv -n thanos 
 PV NAME                                                              PVC NAME               NAMESPACE  NODE NAME                                               POD NAME          VOLUME MOUNT NAME  SIZE   USED    AVAILABLE  %USED  IUSED  IFREE     %IUSED 
 pv-shoot--ede--cic-sre-prod-us-ece98c1c-672c-4cc1-90d7-ad0f77ffbe92  data-thanos-rule-0     thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z3-658fc-hn24n  thanos-rule-0     data               97Gi   40Ki    97Gi       0.00   17     6553583   0.00   
 pv-shoot--ede--cic-sre-prod-us-ff08f6b2-e354-4ab9-8a78-7423805d36e4  data-thanos-receive-2  thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z3-658fc-w2l4c  thanos-receive-2  data               786Gi  35Gi    750Gi      4.55   2193   52426607  0.00   
 pv-shoot--ede--cic-sre-prod-us-8e5cff00-0250-4631-b821-39460e7aae63  data-thanos-receive-4  thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z3-658fc-hbhc9  thanos-receive-4  data               786Gi  35Gi    751Gi      4.46   2222   52426578  0.00   
 pv-shoot--ede--cic-sre-prod-us-26e1e573-2d60-4335-a5d9-26155da22c9d  data-thanos-receive-3  thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z2-66db5-cxr9r  thanos-receive-3  data               786Gi  34Gi    751Gi      4.44   2221   52426579  0.00   
 pv-shoot--ede--cic-sre-prod-us-c17adc6f-e052-474d-bc95-a651c0e4bac4  data-thanos-compact-0  thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z2-66db5-8jld5  thanos-compact-0  data               491Gi  3Gi     487Gi      0.71   1412   32766588  0.00   
 pv-shoot--ede--cic-sre-prod-us-2bc18482-6178-4b2e-a0f2-42ea77f6051f  data-thanos-receive-1  thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z3-658fc-hn24n  thanos-receive-1  data               786Gi  35Gi    750Gi      4.56   2224   52426576  0.00   
 pv-shoot--ede--cic-sre-prod-us-8ac3549a-ec1a-4100-a604-0c2802c7080d  data-thanos-receive-0  thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z2-66db5-8jld5  thanos-receive-0  data               786Gi  36Gi    750Gi      4.59   2227   52426573  0.00   
 pv-shoot--ede--cic-sre-prod-us-c43713f4-7f17-4a57-a8a9-f8c3e41440fd  data-thanos-store-0    thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z3-658fc-hn24n  thanos-store-0    data               491Gi  1818Mi  489Gi      0.36   2642   32765358  0.01   
 pv-shoot--ede--cic-sre-prod-us-6710cc68-0e89-4d54-9249-2d9767267690  data-thanos-store-1    thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z2-66db5-cxr9r  thanos-store-1    data               491Gi  1818Mi  489Gi      0.36   2642   32765358  0.01   
 pv-shoot--ede--cic-sre-prod-us-d38485d9-2c8b-444d-8891-da9e29cc0f60  data-thanos-rule-1     thanos     shoot--ede--cic-sre-prod-us-sre-prod-us-z2-66db5-cxr9r  thanos-rule-1     data               97Gi   40Ki    97Gi       0.00   17     6553583   0.00

receive front-end ingress status were fine request success rate 100%

Experiencing a similar duplication issue.

ts=2024-06-28T07:06:10.839582748Z caller=writer.go:157 level=debug name=receive component=receive component=receive-writer tenant=default-tenant msg="Duplicate sample for timestamp" lset="{__name__=\"extra_kube_persistentvolumeclaim_labels\", container=\"kube-state-metrics\", endpoint=\"http\", exported_namespace=\"enviro-master-civil-finances\", instance=\"10.131.2.153:8080\", job=\"kube-state-metrics\", label_enviro_group_com_storage_key=\"m14-82-w19-nb\", label_enviro_group_com_storage_parent_id=\"0ba87ea0-868f-4284-8da9-c154ea4d5ade\", label_enviro_group_com_storage_type=\"master\", namespace=\"monitoring\", persistentvolumeclaim=\"9399b50d-20d9-45ad-931a-ed4c86ad1d44\", pod=\"extra-kube-state-metrics-6b8f9b7868-lxr72\", prometheus=\"openshift-user-workload-monitoring/user-workload\", prometheus_replica=\"prometheus-user-workload-1\", service=\"extra-kube-state-metrics\"}" value=NaN timestamp=1719558370533
ts=2024-06-28T07:06:10.840190991Z caller=writer.go:253 level=info name=receive component=receive component=receive-writer tenant=default-tenant msg="Error on ingesting samples with different value but same timestamp" numDropped=1
ts=2024-06-28T07:06:10.841305363Z caller=handler.go:584 level=debug name=receive component=receive component=receive-handler tenant=default-tenant msg="failed to handle request" err="add 1 samples: duplicate sample for timestamp"

I only encounter those for persistentvolumeclaims sent by kube-state-metrics. I run thanos receive in v 0.35.0, conf of which is:

          args:
            - receive
            - --label=receive_cluster="preprod"
            - --tsdb.path=/remote-write-data
            - --debug.name=receive
            - --log.level=debug
            - --grpc-address=0.0.0.0:19891
            - --http-address=0.0.0.0:18091
            - --remote-write.address=0.0.0.0:19291
            - --objstore.config-file=/etc/prometheus/objstore-config.yaml

thanos-io / thanos