search

thanos-io / kube-thanos

Kubernetes specific configuration for deploying Thanos.
Apache License 2.0
512 stars 176 forks source link

Thanos-store: Store Pod is not able to load certain blocks from object storage #320

Open epetry30 opened 3 months ago

epetry30 commented 3 months ago

Thanos, Prometheus and Golang version used: quay.io/thanos/thanos:v0.31.0

Object Storage Provider: own Object Storage with S3 endpoint

What happened: At some point after a restart of a Thanos-store pod several blocks are no able to be loaded from our object storage , it is not a huge number but it still results in data loss What you expected to happen: The Thanos-store pod should be able to load all of the blocks.

Full logs to relevant components: level=warn ts=2024-03-27T13:56:45.548274508Z caller=bucket.go:652 msg="loading block failed" elapsed=3.549132ms id=01GNH44X841X3PA2YF67AE4CVB err="create index header reader: write index header: new index reader: get object attributes of 01GNH44X841X3PA2YF67AE4CVB/index: The specified key does not exist." level=warn ts=2024-03-27T13:56:45.548551713Z caller=bucket.go:652 msg="loading block failed" elapsed=3.520884ms id=01HG5MJHV6536D5NKFMN6Z3HD7 err="create index header reader: write index header: new index reader: get object attributes of 01HG5MJHV6536D5NKFMN6Z3HD7/index: The specified key does not exist." level=warn ts=2024-03-27T13:56:45.548600296Z caller=bucket.go:652 msg="loading block failed" elapsed=2.923082ms id=01HG695QEDVAGJ9MCPYEFM1B1E err="create index header reader: write index header: new index reader: get object attributes of 01HG695QEDVAGJ9MCPYEFM1B1E/index: The specified key does not exist." level=warn ts=2024-03-27T13:56:45.548964003Z caller=bucket.go:652 msg="loading block failed" elapsed=3.079476ms id=01GTJ8VHZJ70EXBHDF7R9A1MZ2 err="create index header reader: write index header: new index reader: get object attributes of 01GTJ8VHZJ70EXBHDF7R9A1MZ2/index: The specified key does not exist." level=warn ts=2024-03-27T13:56:45.549556296Z caller=bucket.go:652 msg="loading block failed" elapsed=3.047629ms id=01HFV43YJX0SY19SSJNJ4Q05WE err="create index header reader: write index header: new index reader: get object attributes of 01HFV43YJX0SY19SSJNJ4Q05WE/index: The specified key does not exist." level=warn ts=2024-03-27T13:56:45.55066445Z caller=bucket.go:652 msg="loading block failed" elapsed=3.244963ms id=01H7J41MEM1EZGEAY9Y695CM1T err="create index header reader: write index header: new index reader: get object attributes of 01H7J41MEM1EZGEAY9Y695CM1T/index: The specified key does not exist." level=warn ts=2024-03-27T13:56:45.566415912Z caller=bucket.go:652 msg="loading block failed" elapsed=21.854809ms id=01H5V67MFXA5PZG4VCF1SW5REM err="create index header reader: write index header: new index reader: get object attributes of 01H5V67MFXA5PZG4VCF1SW5REM/index: The specified key does not exist."

epetry30 commented 3 months ago

Missing blocks are in /var/thanos/store/meta-syncer/ directory but not in /var/thanos/store so they have no index reference