Open LERUfic opened 10 months ago
@LERUfic May I know which version of Prometheus are you using? I am wondering if it is the same issue as https://github.com/thanos-io/thanos/issues/6723
sure @yeya24 I use prometheus 2.42.0
root@vm# ./prometheus --version
prometheus, version 2.42.0 (branch: HEAD, revision: 225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)
build user: root@c67d48967507
build date: 20230201-07:53:32
go version: go1.19.5
platform: linux/amd64
PS: opss wrong account
This seems like a bug in Thanos https://github.com/thanos-io/thanos/blob/main/pkg/compact/compact.go#L1394. The error cause doesn't work properly to identify the original error is out of chunk error, causing fail to ignore OOO chunks issue.
I will create an issue on Thanos side.
I see thank you for the response. For now I marked the chunks with no-compact-mark.json
using thanos tools.
type: GCS
config:
bucket: <redacted>
prefix: prometheus-data-prd
thanos tools bucket mark --id=<block> --marker=no-compact-mark.json --objstore.config-file=thanos.yaml --details=OOO --log.level=debug
And seems like there's no error for now but the compaction still has not finished yet. We will monitor this for a while.
Describe the bug Due to out-of-order chunk, the compactor is not performing compaction as expected. Although I have added the
skip_blocks_with_out_of_order_chunks_enabled: true
configuration, the block is not being marked as non-compact.To Reproduce Steps to reproduce the behavior:
Expected behavior I expect the compaction process is running smoothly and even the out-of-order happening it's should be skipped because of the
skip_blocks_with_out_of_order_chunks_enabled: true
config.Environment:
Additional Context My compactor config
Runtime Config
Metrics
Error logs