search

grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.17k stars 537 forks source link

422 Unprocessable Entity, body: \"block time range crosses boundary of configured compactor time range (1d) #9910

Open mifinn85 opened 2 weeks ago

mifinn85 commented 2 weeks ago

Hi,

We are trying to upload TSDB blocks from Prometheus to Mimir but it keeps failing on time range set to 1d, older block files are not being uploaded.

Where exactly we need to change this setting from 1d to a different value.

Helm Release:

  chart: mimir-distributed

  version: 5.6.0-weekly.316

Error:

/prometheus $ ./mimirtool backfill --address=http://mimir-compactor.monitoring.svc.cluster.local:8080 --id=anonymous /prometheus/01J9HCHX3VV8W28445J8G7DGEC INFO[0000] Backfilling blocks=/prometheus/01J9HCHX3VV8W28445J8G7DGEC user=anonymous INFO[0000] making request to start block upload block=01J9HCHX3VV8W28445J8G7DGEC file=meta.json path=/prometheus/01J9HCHX3VV8W28445J8G7DGEC ERRO[0000] response body="block time range crosses boundary of configured compactor time range (1d)\n" status="422 Unprocessable Entity" ERRO[0000] failed uploading block error="request to start block upload failed: POST request to http://mimir-compactor.monitoring.svc.cluster.local:8080/api/v1/upload/block /01J9HCHX3VV8W28445J8G7DGEC/start failed: server returned HTTP status: 422 Unprocessable Entity, body: \"block time range crosses boundary of configured compactor time range (1d)\n\"" path=/pro metheus/01J9HCHX3VV8W28445J8G7DGEC INFO[0000] finished uploading blocks already_exists=0 failed=1 succeeded=0 mimirtool: error: blocks failed to upload 1 block(s), try --help

mifinn85 commented 1 week ago

Would really appreciate some help here please.

andyasp commented 6 days ago

Hello,

Where exactly we need to change this setting from 1d to a different value.

I would not suggest changing the -compactor.block-ranges setting. It's best to leave that at the default.

ERRO[0000] response body="block time range crosses boundary of configured compactor time range (1d)\n" status="422 Unprocessable Entity"

The error you are encountering is due to an uploaded block not aligning to the time boundaries that are expected by Mimir. Mimir, by default, expects at most 24-hour blocks aligned by day. For example, if you try to upload a block that spans noon of one day to noon of the following day you would hit this error. This validation was added in https://github.com/grafana/mimir/pull/9524. Ensuring this alignment helps avoid encountering performance issues later on in the query path.

The solution is to use a tool in the Mimir repository, splitblocks, to split the original block into multiple blocks that have an expected alignment. Then those blocks can be uploaded instead. It would certainly be nice if the backfill did this for you, but that hasn't been implemented.