store: disk usage continually increasing over time

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

https://thanos.io

Apache License 2.0

12.73k stars 2.04k forks source link

store: disk usage continually increasing over time #7473

Open pgier opened 1 week ago

pgier commented 1 week ago

Thanos, Prometheus and Golang version used: Thanos version: quay.io/thanos/thanos:v0.35.1 Golang: whatever version was used to build the quay image Prometheus: quay.io/prometheus/prometheus:v2.47.0

Object Storage Provider: AWS S3

What happened: Disk usage (k8s PV) continually increases, it's up to about 210 GB currently.

What you expected to happen: Disk usage would stabilize at a certain point

How to reproduce it (as minimally and precisely as possible): I'm not sure how to reproduce it other than install thanos with a store gateway and some metrics sources, and then periodically run queries.

Full logs to relevant components:

See attached log file.

Anything else we need to know:

thanos-store.log

pgier commented 1 week ago

mdraijer commented 1 week ago

Same here.

We have 9 different Thanos stacks running, with each a storegateway. Most of them have limited disk usage, as you would expect from the phrase "It acts primarily as an API gateway and therefore does not need significant amounts of local disk space", and also "It keeps a small amount of information about all remote blocks on local disk".

However, one of them has constantly increased in used disk space. Also after cleaning up and restarting, it starts filling up again. Have increased the disk to 150Gi now, whereas all other storegateway disks are 20Gi in size and 5%-30% filled.

Recently one of the other storegateway also started to fill up.

Why do some stores have so much local data and others not?

harry671003 commented 2 days ago

Each store-gateway downloads the index-header for the blocks its responsible for.

IIRC, Thanos by default doesn't have sharding enabled. Maybe you could try to shard the store-gateways so that not every store-gateway will download all the block index-headers.

I think there are two ways to shard store-gateways:

Sharding using relabelling - https://thanos.io/v0.14/thanos/sharding.md/#relabelling
Sharding using time-based partitioning - https://thanos.io/v0.14/components/store.md/#time-based-partitioning

pgier commented 1 day ago

@harry671003 I would think that would cause disk usage at startup, or when the overall number of metrics increases, but I'm seeing a gradual and consistent increase in disk usage, maybe 1-2 GB per week.