Thanos Querier cannot query real time data less than 2 hours

[heartTorres]

Thanos Querier

Thanos, Prometheus and Golang version used: Thanos v0.15.0, rancher/prom-prometheus v2.17.2

Object Storage Provider: AWS S3

What happened: I have followed the set up in our kubernetes / rancher this blog to set up the thanos. https://rancher.com/blog/2020/prometheus-metric-federation.

• The thanos components are installed in ClusterA while the Prometheus is installed in Cluster B.
• Data blocks are successfully uploaded in S3 every 2 hours.
• Set the data source of Grafana Dashboard to use the Thanos Querier as data source.

Issue: When we query data with time range less than an hour, no results from the query. But if we query data 2 hours ago and beyond, data results were returned.

What you expected to happen: We expect to get real time data to be queried from Thanos Querier

How to reproduce it (as minimally and precisely as possible): Install based on this blog https://rancher.com/blog/2020/prometheus-metric-federation 2 clusters (1- for thanos components only, 1 - cluster with prometheus monitoring enabled)

Query Deployment Yaml:

      containers:
      - args:
        - query
        - --log.level=info
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --query.replica-label=prometheus_replica
        - --query.replica-label=rule_replica
        - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local
        image: quay.io/thanos/thanos:v0.15.0

Full logs to relevant components: Logs appear when we query with time range less than 2 hours. level=warn ts=2020-09-22T08:51:33.495068323Z caller=proxy.go:310 err="No StoreAPIs matched for this query" stores="store Addr: thanos-store.thanos.svc.cluster.local:10901 LabelSets: [name:\"receive\" value:\"true\" ,name:\"replica\" value:\"thanos-receive-0\" ,name:\"tenant_id\" value:\"default-tenant\" ] Mint: 1600684402983 Maxt: 1600762390077 filtered out"

Anything else we need to know:

[bwplotka]

From logs it looks like your Thanos deployment has access only to single Store that gives old [name:\"receive\" value:\"true\" ,name:\"replica\" value:\"thanos-receive-0\" ,name:\"tenant_id\" value:\"default-tenant\"]

Are you sure you connected it to the sidecar? To me it shows only Store Gateway.

Can you check querier UI and /stores page? It lists all StoreAPIs you have access to.

[heartTorres]

From logs it looks like your Thanos deployment has access only to single Store that gives old [name:\"receive\" value:\"true\" ,name:\"replica\" value:\"thanos-receive-0\" ,name:\"tenant_id\" value:\"default-tenant\"]

Are you sure you connected it to the sidecar? To me it shows only Store Gateway.

Can you check querier UI and /stores page? It lists all StoreAPIs you have access to.

Hi bwplotka, I was able to fixed this by adding the Thanos Receiver in my config in my Query Deployment. Thank you so much for the prompt response. I closed the ticket.

[pronny]

@heartTorres where did you add the Thanos Receiver? Can you please share an example. I'm having the same issue.

[heartTorres]

@heartTorres where did you add the Thanos Receiver? Can you please share an example. I'm having the same issue.

Hi Prony, I added it in my Thanos Querier, then it queries real time data from prometheus.

[erlatbud]

@pronny I also had this issue, and i fixed it by adding in the Thanos Receiver's StoreAPI endpoint to Thanos Query using this configuration

You can check the deployment manifest of Thanos Query to ensure that the endpoint(s) have been added in. This depends on the number of replicas you've configured for Thanos Receiver.

  args:
    - query
    - --log.level=info
    - --log.format=logfmt
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --query.replica-label=replica
    - --store.sd-files=/conf/sd/servicediscovery.yml
    - --endpoint=thanos-receive-0.thanos-receive-headless.monitoring.svc.cluster.local:10901     <---- StoreAPI endpoint
    - --endpoint=thanos-receive-1.thanos-receive-headless.monitoring.svc.cluster.local:10901      <---- StoreAPI endpoint
    - --endpoint=thanos-receive-2.thanos-receive-headless.monitoring.svc.cluster.local:10901      <---- StoreAPI endpoint

I thought I'd leave this comment here for anyone else that might stumble across this issue.