query: Most recent values are not summed correctly for multiple receivers

[mfilocha]

Thanos, Prometheus and Golang version used: 0.30.1, golang: go1.19.4 (version 0.29 also affected)

Object Storage Provider: GCS bucket

What happened: We are using 150+ Prometheuses running on Kubernetes clusters and sending data via Remote Write endpoint to Thanos. We are running 5 instances of Receiver. For some data series, the most recent values (around last 2 hours) are not calculated properly - they are too low and fluctuating. These wrong values are only for data queried from receiver replicas endpoints. The same data provided by store gateway looks more consistent.

See the image below - data before 15:00 comes from store gateway, values after 15:00 are based on receivers endpoints.

What you expected to happen: Values returned by 5 receivers are the same as returned by store gateway. I don't expect node count to fluctuate - this is an error in calculation (summing) I suppose.

Anything else we need to know:

This is our stores configuration:

There is one more store gateway for another bucket with data older than 7 months, not shown on the screenshot.

[JayChanggithub]

@mfilocha I'd like to know when you collected 150+ Prometheuses metrics via thanos-rceivers. What's limit resources setup? Would you please kindly to share me for reference?. Also - --receive.replication-factor counts. As your screenshot should be have 5 pods of receiver.

[mfilocha]

@JayChanggithub - replication factor is default (1). All 5 replicas of receiver use around 115 GB of RAM in total.

And we don't send all default Kubernetes metrics, it's a heavily tuned subset.

[JayChanggithub]

We also adopted receiver of thanos as well and around 150 kubernetes cluster probably. As same your landscape. However i has been adjust the replication factor 1 to 3. I found the each pod of receiver consumed of memory higher than replication factor 1. I'am not sure what's limit of resources is enough. Just wanna to refer your setting as well. Thank you.

[fpetkovski]

@mfilocha do you have a hashrings file configured for your receivers?

[mfilocha]

@fpetkovski - No, we use the following configuration for the receiver:

- --objstore.config=$(OBJSTORE_CONFIG)
- --tsdb.retention=6h
- --receive.default-tenant-id=cluster
- --receive.local-endpoint="$(NAME).thanos-receive.$(NAMESPACE).svc.cluster.local:10901"
- --label=receive_replica="$(NAME)"

and for the querier:

- --http-address=0.0.0.0:9090
- --query.replica-label=receive_replica
- --store=dnssrv+_grpc._tcp.thanos-store
- --store=dnssrv+_grpc._tcp.thanos-receive
- --store=dnssrv+_grpc._tcp.thanos-rule
- --query.auto-downsampling
- --query.timeout=15m

[mfilocha]

@fpetkovski Any hints? Version 0.32.3 behaves the same.