Constant compactor warnings due to deletion delay

[andrejshapal]

Thanos, Prometheus and Golang version used: We bump to latest Thanos on every Monday

Object Storage Provider: gc bucket

What happened: I was searching for resolution of my issue and found a lot of similar issues on git and here in slack. It is very scary none of the issues had a proper answer or solution. Basically, we run compactor with args:

    - args:
        - compact
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--http-address=0.0.0.0:10902'
        - '--data-dir=/data'
        - '--retention.resolution-raw=164h'
        - '--retention.resolution-5m=164h'
        - '--retention.resolution-1h=164h'
        - '--consistency-delay=30m'
        - '--objstore.config-file=/conf/objstore.yml'
        - '--compact.concurrency=1'
        - '--downsampling.disable'
        - '--bucket-web-label=cluster'
        - '--compact.skip-block-with-out-of-order-chunks'
        - |
          --selector.relabel-config=
            - action: keep
              source_labels:
                - cluster
              regex: 'ci-k8s'
        - '--wait'

and we have noticed spam in logs like:

2024-02-16 00:31:30.153 
ts=2024-02-15T22:31:30.153276892Z caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"
2024-02-16 00:31:30.153 
ts=2024-02-15T22:31:30.153246614Z caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
2024-02-16 00:31:30.153 
ts=2024-02-15T22:31:30.153233803Z caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
2024-02-16 00:31:30.153 
ts=2024-02-15T22:31:30.153217266Z caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
2024-02-16 00:31:30.153 
ts=2024-02-15T22:31:30.153132042Z caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=12.574082672s duration_ms=12574 cached=1135 returned=50 partial=0
2024-02-16 00:31:17.579 
ts=2024-02-15T22:31:17.579080936Z caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
2024-02-16 00:31:17.579 
ts=2024-02-15T22:31:17.579036132Z caller=retention.go:47 level=info msg="optional retention apply done"
2024-02-16 00:31:17.579 
ts=2024-02-15T22:31:17.57901602Z caller=block.go:185 level=warn msg="requested to mark for deletion, but file already exists; this should not happen; investigate" err="file 01GYBSC0YF8Z0K1WG4QARTDZZ7/deletion-mark.json already exists in bucket"
2024-02-16 00:31:17.561 
ts=2024-02-15T22:31:17.561785129Z caller=retention.go:41 level=info msg="applying retention: marking block for deletion" id=01GYBSC0YF8Z0K1WG4QARTDZZ7 maxTime="2023-04-19 00:00:00 +0000 UTC"

The compactor goes through the same blocks in frequent loop trying to mark them and fire warning.

What you expected to happen: My theory is that the default deletion delay is 48h. And wait period is 5m. Therefore compactor runs every 5 minutes and tries to mark the blocks which were marked before but not deleted due to 48h delay. If this is correct, then the warning should not be fired if the deletion mark was created < deletion delay. And maybe the default wait-period should be increased (to 48h?). I don't know if it is really needed to run compaction each 5m and run over the same blocks...

How to reproduce it (as minimally and precisely as possible): It should be reproducible if there are any marked for deletion blocks.

Full logs to relevant components:

Anything else we need to know: I also think this leads to increased expenses, no? Not sure how it works, but compactor probably downloads some data from bucket and this is paid operation?

Environment:

• k8s
• deploying with bitnami helm

[PantelisK]

Our environments seem to be similar, and we also face the same issue. Does your thanos_blocks_meta_synced report a lot of duplicates as well?

This is an image from bitnami's bucketweb that shows "duplicate" blocks with different resolution

[andrejshapal]

@PantelisK Different resolution is expected behaviour when downsampling is turned on. In our case it is:

• '--downsampling.disable' Because we don't wan't to consume bucket space. Downsampling main purpose is to improve performance when querieng long data ranges.

[PantelisK]

@andrejshapal thanks for the clarification. I though the duplicate was a problem :)