search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.06k stars 2.09k forks source link

Sidecar fails to upload compacted blocks. #6569

Open esapozhnikov-wish opened 1 year ago

esapozhnikov-wish commented 1 year ago

Thanos, Prometheus and Golang version used: thanos : 0.31.0 go: go1.19.7 Promethues: 2.32.1

Object Storage Provider: S3

What happened:
Uploaded already compacted blocks from the Prometheus node using '-shipper.upload-compacted' flag successfully. Ended up having some issues and deleted all the blocks from S3 using the tools bucket cleanup. On Prom node deleted the "thanos.shipper.json" file and tried to run the upload of compacted blocks again.

Command used to upload the blocks: /usr/local/bin/thanos sidecar --tsdb.path /prometheus/metrics --prometheus.url "http://localhost:9090" --objstore.config-file /etc/thanos/bucket_config.yml --shipper.upload-compacted

But I am getting following error now:

level=info ts=2023-07-28T16:19:51.552532693Z caller=sidecar.go:201 msg="successfully loaded prometheus external labels" external_labels="{_shard_id=\"0\", node_id=\"002\", shard_node_id=\"001\", wish_environment=\"legacy\", zone=\"us-west-1c\"}" level=info ts=2023-07-28T16:20:23.542135946Z caller=shipper.go:208 msg="gathering all existing blocks from the remote bucket for check" id=01GRF60FSMR9SQG5X2GD8HKD44 level=warn ts=2023-07-28T16:20:24.400567385Z caller=sidecar.go:347 err="Found overlap or error during sync, cannot upload compacted block, details: get all block meta.: meta.json bkt get for 01H6CZD8H3FGW0M3B6EWCKNA75: The specified key does not exist." uploaded=0 level=info ts=2023-07-28T16:20:53.538785008Z caller=shipper.go:208 msg="gathering all existing blocks from the remote bucket for check" id=01GRF60FSMR9SQG5X2GD8HKD44 level=warn ts=2023-07-28T16:20:54.294146151Z caller=sidecar.go:347 err="Found overlap or error during sync, cannot upload compacted block, details: get all block meta.: meta.json bkt get for 01H6CZD8H3FGW0M3B6EWCKNA75: The specified key does not exist." uploaded=0 level=info ts=2023-07-28T16:21:23.541173627Z caller=shipper.go:208 msg="gathering all existing blocks from the remote bucket for check" id=01GRF60FSMR9SQG5X2GD8HKD44

I checked the bucket for any blocks with the same zone external label but there was nothing: `root@thanos-compactor-c:/etc/thanos/store/downsample# thanos tools bucket inspect --objstore.config-file /etc/thanos/bucket_config.yml -l zone=\"us-west-1c\" level=info ts=2023-07-28T16:11:06.28602282Z caller=factory.go:52 msg="loading bucket configuration" level=info ts=2023-07-28T16:11:06.480545107Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=193.824295ms duration_ms=193 cached=67 returned=67 partial=5 ULID FROM UNTIL RANGE UNTIL-DOWN #SERIES #SAMPLES #CHUNKS COMP-LEVEL COMP-FAILED LABELS RESOLUTION SOURCE

level=info ts=2023-07-28T16:11:06.481017187Z caller=main.go:164 msg=exiting`

What you expected to happen: Expected the already compacted blocks to be uploaded to S3 again

How to reproduce it (as minimally and precisely as possible): Upload compacted blocks Delete the compacted blocks Delete thanos.shipper.json Try uploading the blocks again using following command: /usr/local/bin/thanos sidecar --tsdb.path /prometheus/metrics --prometheus.url "http://localhost:9090" --objstore.config-file /etc/thanos/bucket_config.yml --shipper.upload-compacted Full logs to relevant components:

Anything else we need to know:

douglascamata commented 1 year ago

Seems like you have a block (`01H6CZD8H3FGW0M3B6EWCKNA75) in your S3 bucket that doesn't have a meta.json file for it?