search

grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.18k stars 537 forks source link

mimirtool backfill - compactor: can't check block state/block validation failed #9032

Open litetex opened 3 months ago

litetex commented 3 months ago

Describe the bug

When running mimirtool backfill I get caller=block_upload.go:207 level=warn component=compactor user=tdr feature="block upload" block=01J4WNXPS4GSR8AMK9A4732704 operation="start block upload" request_id=17ec82ff283a34ff msg="can't check block state" response="block validation failed" status=400

Setting the log level to debug doesn't help at all.

It would be nice to know WHY the validation failed so that I can diagnose and address the problem...

To Reproduce

Sorry but I can't paste my complete setup here.

Basically:

  1. Start mimir
  2. Run backfill for prometheus: mimirtool backfill --address=http://mimir-backend:9009 --id=<redacted> /data/prometheus/01J4WNXPS4GSR8AMK9A4732704 ...

Expected behavior

See an error message that says what caused the block validation error so that I can fix it.

Environment

irrelevant Mimir version: 2.13.0

Additional Context

Same problem was already encountered in https://github.com/grafana/mimir/issues/8164 but no efforts were made to improve the situation :/

Likely these pieces of code should rethrow the error and not just "swallow" it: https://github.com/grafana/mimir/blob/3872ccbbb4b2d54f2bf535659db37af5e5f8eac9/pkg/compactor/block_upload.go#L766-L768 https://github.com/grafana/mimir/blob/3872ccbbb4b2d54f2bf535659db37af5e5f8eac9/pkg/compactor/block_upload.go#L733-L735

litetex commented 3 months ago

Also note that compactor_block_upload_validation_enabled: false doesn't help... The error is still the same