search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.03k stars 2.09k forks source link

Problem with receiver and store #7520

Open gerpion opened 3 months ago

gerpion commented 3 months ago

prometheus, version 2.45.5 (branch: HEAD, revision: 2b052add78646ff39d193dac84eae8855d11565a) build user: root@98598c5dfe5e build date: 20240502-08:58:53 go version: go1.21.9 platform: linux/amd64 tags: netgo,builtinassets,stringlabels

thanos, version 0.36.0-rc.0 (branch: HEAD, revision: c930d2ee6c113aa3b96991d42b498fccb61ad914) build user: root@fd311f2900e8 build date: 20240626-17:13:42 go version: go1.21.11 platform: linux/amd64 tags: netgo

Object Storage Provider:

What happened: The data are shown into grafana and sometimes no more data are incoming so the graph value are at 0. (i hope the explanation is clear). To compute the data and see them in grafana i have to stop the receiver and store services.

What you expected to happen: The data should constantly go in thanos and show in grafana.

How to reproduce it (as minimally and precisely as possible): To reproduce it, i just have to wait a few minute and no data are coming.

Full logs to relevant components:

Here is what happens suddenly, the graph stop to show data. thanos_no_data_inc

After restarting receiver and store services the data are here again. And it will do the same after a few minutes. thanos_after_restarting_services

Anything else we need to know:

gerpion commented 3 months ago

Here are the conf files :

thanos-receive.service [Unit] Description=Thanos Receive After=network.target

[Service] ExecStart=/usr/local/bin/thanos receive \ --http-address 0.0.0.0:10907 \ --grpc-address 0.0.0.0:10908 \ --remote-write.address 0.0.0.0:10901 \ --objstore.config-file /etc/minio/minio-config.yaml \ --tsdb.path /opt/thanos/receive \ --log.level=debug \ --label "receive_cluster=\"BDX-S1-THA-001\""

Restart=always

[Install] WantedBy=multi-user.target

thanos-store.service [Unit] Description=Thanos Store After=network.target

[Service] ExecStart=/usr/local/bin/thanos store \ --data-dir /opt/thanos/data/ \ --objstore.config-file /etc/minio/minio-config.yaml \ --http-address 0.0.0.0:10905 \ --grpc-address 0.0.0.0:10903 \ --index-cache-size 2GB \ --chunk-pool-size 5GB \ --log.level=debug Restart=always

[Install] WantedBy=multi-user.target

/etc/minio/minio-config.yaml type: S3 config: bucket: thanos endpoint: BDX-S1-THA-001:9000 access_key: xxxxx secret_key: yyyyy insecure: true signature_version2: false