search

grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.14k stars 531 forks source link

Flaky TestMultitenantCompactor_DeleteLocalSyncFiles #619

Open dimitarvdimitrov opened 2 years ago

dimitarvdimitrov commented 2 years ago 
--- FAIL: TestMultitenantCompactor_DeleteLocalSyncFiles (6.44s)
    compactor_test.go:1694: 
            Error Trace:    compactor_test.go:1694
            Error:          Should not be zero, but was 0
            Test:           TestMultitenantCompactor_DeleteLocalSyncFiles

happens on commit 5679fba6b760f129c7e2277975328b0e4d67f64a

Example failed run https://github.com/grafana/mimir/runs/4546071431?check_suite_focus=true

dimitarvdimitrov commented 11 months ago

another failure 

--- FAIL: TestMultitenantCompactor_DeleteLocalSyncFiles (6.38s)
    compactor_test.go:2001: 
            Error Trace:    /__w/mimir/mimir/pkg/compactor/compactor_test.go:2001
            Error:          Should not be zero, but was 0
            Test:           TestMultitenantCompactor_DeleteLocalSyncFiles
pracucci commented 4 months ago

another failure

--- FAIL: TestMultitenantCompactor_DeleteLocalSyncFiles (6.16s)
    compactor_test.go:2003: 
            Error Trace:    /__w/mimir/mimir/pkg/compactor/compactor_test.go:2003
            Error:          Should not be zero, but was 0
            Test:           TestMultitenantCompactor_DeleteLocalSyncFiles
dimitarvdimitrov commented 1 month ago

failed again

Details

``` --- FAIL: TestMultitenantCompactor_DeleteLocalSyncFiles (6.52s) compactor_test.go:2003: Error Trace: /__w/mimir/mimir/pkg/compactor/compactor_test.go:2003 Error: Should not be zero, but was 0 Test: TestMultitenantCompactor_DeleteLocalSyncFiles level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1727454973 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=3 existing=0 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=3 partial_errors=0 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=4.41063ms level=info component=cleaner run_id=1727454973 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.765765ms duration_ms=6 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01J8T5E3371XJ7X9YKGYFP39ZR (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01J8T5E352EHG2HKMAB9KCB4HN (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01J8T5E36SY8R934WJYKPR1G01 (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=3 blocks=01J8T5E3371XJ7X9YKGYFP39ZR,01J8T5E352EHG2HKMAB9KCB4HN,01J8T5E36SY8R934WJYKPR1G01 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E352EHG2HKMAB9KCB4HN/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E3371XJ7X9YKGYFP39ZR/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E36SY8R934WJYKPR1G01/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=3 blocks=01J8T5E3371XJ7X9YKGYFP39ZR,01J8T5E352EHG2HKMAB9KCB4HN,01J8T5E36SY8R934WJYKPR1G01 duration=4.628968ms duration_ms=4 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01J8T5E3DJN4DN5F3CBM9RFN6X sources="[01J8T5E352EHG2HKMAB9KCB4HN 01J8T5E3371XJ7X9YKGYFP39ZR 01J8T5E36SY8R934WJYKPR1G01]" duration=131.604695ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01J8T5E3DJG5R9A4YVSJF78ZJH sources="[01J8T5E352EHG2HKMAB9KCB4HN 01J8T5E3371XJ7X9YKGYFP39ZR 01J8T5E36SY8R934WJYKPR1G01]" duration=131.702958ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E3DJN4DN5F3CBM9RFN6X 01J8T5E3DJG5R9A4YVSJF78ZJH]" block_count=3 blocks=01J8T5E3371XJ7X9YKGYFP39ZR,01J8T5E352EHG2HKMAB9KCB4HN,01J8T5E36SY8R934WJYKPR1G01 duration=131.845986ms duration_ms=131 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E3DJG5R9A4YVSJF78ZJH/chunks/000001 dst=01J8T5E3DJG5R9A4YVSJF78ZJH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E3DJN4DN5F3CBM9RFN6X/chunks/000001 dst=01J8T5E3DJN4DN5F3CBM9RFN6X/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E3DJG5R9A4YVSJF78ZJH/index dst=01J8T5E3DJG5R9A4YVSJF78ZJH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E3DJN4DN5F3CBM9RFN6X/index dst=01J8T5E3DJN4DN5F3CBM9RFN6X/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4044587089/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E3DJG5R9A4YVSJF78ZJH duration=1.915002ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E3DJN4DN5F3CBM9RFN6X duration=1.862043ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=5.048943ms duration_ms=5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E3371XJ7X9YKGYFP39ZR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E3371XJ7X9YKGYFP39ZR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E352EHG2HKMAB9KCB4HN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E352EHG2HKMAB9KCB4HN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E36SY8R934WJYKPR1G01 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E36SY8R934WJYKPR1G01 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=144.899112ms duration_ms=144 block_count=3 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=5.06788ms duration_ms=5 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.0[64](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:65)474ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454973 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=1 existing=0 level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=1 partial_errors=0 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.61422ms duration_ms=4 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=cleaner run_id=1727454973 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=5.277251ms level=info component=cleaner run_id=1727454973 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01J8T5E3QYB5MKRHFTWMMAQ4N3 (min time: 1970-01-01 03:59:59.999 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=1 blocks=01J8T5E3QYB5MKRHFTWMMAQ4N3 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E3QYB5MKRHFTWMMAQ4N3/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=1 blocks=01J8T5E3QYB5MKRHFTWMMAQ4N3 duration=1.989582ms duration_ms=1 level=info component=compactor msg="compact blocks" count=1 mint=14399999 maxt=14400000 ulid=01J8T5E3XDY0XGWAFRFQSXZSKH sources=[01J8T5E3QYB5MKRHFTWMMAQ4N3] duration=88.607625ms shard=1_of_2 level=info component=compactor msg="compact blocks resulted in empty block" count=1 sources=[01J8T5E3QYB5MKRHFTWMMAQ4N3] duration=88.720326ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E3XDY0XGWAFRFQSXZSKH 00000000000000000000000000]" block_count=1 blocks=01J8T5E3QYB5MKRHFTWMMAQ4N3 duration=88.820894ms duration_ms=88 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction produced an empty block" shard_id=2_of_2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage2462273109/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E3XDY0XGWAFRFQSXZSKH/chunks/000001 dst=01J8T5E3XDY0XGWAFRFQSXZSKH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage2462273109/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage2462273109/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E3XDY0XGWAFRFQSXZSKH/index dst=01J8T5E3XDY0XGWAFRFQSXZSKH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage2462273109/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E3XDY0XGWAFRFQSXZSKH duration=1.794637ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=9.998183ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E3QYB5MKRHFTWMMAQ4N3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E3QYB5MKRHFTWMMAQ4N3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=108.632643ms duration_ms=108 block_count=1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=7.090442ms duration_ms=7 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.532262ms duration_ms=6 cached=1 returned=1 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454974 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=8.019837ms duration_ms=8 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--0-7200000 job="stage: merge, range start: 0, range end: 7200000, shard: , blocks: 01J8T5E43RNGFP2D9PQKD0ZHDG (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01J8T5E46RMR8N1J4719SEXBSA (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E43RNGFP2D9PQKD0ZHDG,01J8T5E46RMR8N1J4719SEXBSA level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E46RMR8N1J4719SEXBSA/meta.json level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=19.30569ms level=info component=cleaner run_id=1727454974 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E43RNGFP2D9PQKD0ZHDG/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E43RNGFP2D9PQKD0ZHDG,01J8T5E46RMR8N1J4719SEXBSA duration=10.622769ms duration_ms=10 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01J8T5E49JMN3MBWGYX2XSPZ0C sources="[01J8T5E43RNGFP2D9PQKD0ZHDG 01J8T5E46RMR8N1J4719SEXBSA]" duration=50.035729ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E49JMN3MBWGYX2XSPZ0C] block_count=2 blocks=01J8T5E43RNGFP2D9PQKD0ZHDG,01J8T5E46RMR8N1J4719SEXBSA duration=50.192523ms duration_ms=50 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02914642819/001/compact/0@17241709254077376921-merge--0-7200000/01J8T5E49JMN3MBWGYX2XSPZ0C/chunks/000001" dst=01J8T5E49JMN3MBWGYX2XSPZ0C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02914642819/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02914642819/001/compact/0@17241709254077376921-merge--0-7200000/01J8T5E49JMN3MBWGYX2XSPZ0C/index" dst=01J8T5E49JMN3MBWGYX2XSPZ0C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_02914642819/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E49JMN3MBWGYX2XSPZ0C duration=5.885648ms duration_ms=5 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.4[65](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:66)595ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E43RNGFP2D9PQKD0ZHDG level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E43RNGFP2D9PQKD0ZHDG level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E46RMR8N1J4719SEXBSA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E46RMR8N1J4719SEXBSA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=71.[66](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:67)701ms duration_ms=71 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=5.941784ms duration_ms=5 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.953214ms duration_ms=3 cached=1 returned=1 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=cleaner run_id=1727454974 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=cleaner run_id=1727454974 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=12.760838ms level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454974 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.310996ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01J8T5E4F0MXMVV377XGT0Q3ZM (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01J8T5E4HEHCCKBMZFM46T0GA7 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E4F0MXMVV377XGT0Q3ZM,01J8T5E4HEHCCKBMZFM46T0GA7 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E4F0MXMVV377XGT0Q3ZM/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E4HEHCCKBMZFM46T0GA7/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E4F0MXMVV377XGT0Q3ZM,01J8T5E4HEHCCKBMZFM46T0GA7 duration=7.407414ms duration_ms=7 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E4S64BCHV3PR48230QWH sources="[01J8T5E4HEHCCKBMZFM46T0GA7 01J8T5E4F0MXMVV377XGT0Q3ZM]" duration=163.791045ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E4S60F4S81A6ZT14DCEM sources="[01J8T5E4HEHCCKBMZFM46T0GA7 01J8T5E4F0MXMVV377XGT0Q3ZM]" duration=163.930857ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E4S64BCHV3PR48230QWH 01J8T5E4S60F4S81A6ZT14DCEM]" block_count=2 blocks=01J8T5E4F0MXMVV377XGT0Q3ZM,01J8T5E4HEHCCKBMZFM46T0GA7 duration=164.181014ms duration_ms=164 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E4S60F4S81A6ZT14DCEM/chunks/000001 dst=01J8T5E4S60F4S81A6ZT14DCEM/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E4S60F4S81A6ZT14DCEM/index dst=01J8T5E4S60F4S81A6ZT14DCEM/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E4S60F4S81A6ZT14DCEM duration=2.041869ms duration_ms=2 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E4S64BCHV3PR48230QWH/chunks/000001 dst=01J8T5E4S64BCHV3PR48230QWH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E4S64BCHV3PR48230QWH/index dst=01J8T5E4S64BCHV3PR48230QWH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorcompaction_on_blocks_containing_native_histograms3235710925/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E4S64BCHV3PR48230QWH duration=1.784268ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=8.848783ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E4F0MXMVV377XGT0Q3ZM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E4F0MXMVV377XGT0Q3ZM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E4HEHCCKBMZFM46T0GA7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E4HEHCCKBMZFM46T0GA7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=182.896351ms duration_ms=182 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.344205ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.285518ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1727454975 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=2.130986ms duration_ms=2 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01J8T5E53V3BGXN543SB4D9XTA (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01J8T5E55PZCVD0Y1AAGR2Y591 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E55PZCVD0Y1AAGR2Y591,01J8T5E53V3BGXN543SB4D9XTA level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=5.070814ms level=info component=cleaner run_id=1727454975 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E55PZCVD0Y1AAGR2Y591/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E53V3BGXN543SB4D9XTA/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E55PZCVD0Y1AAGR2Y591,01J8T5E53V3BGXN543SB4D9XTA duration=3.700849ms duration_ms=3 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E5C32V0QQ87DXKDZP7Z7 sources="[01J8T5E55PZCVD0Y1AAGR2Y591 01J8T5E53V3BGXN543SB4D9XTA]" duration=134.220372ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E5C3Q4NNNGP4W1DK1EBC sources="[01J8T5E55PZCVD0Y1AAGR2Y591 01J8T5E53V3BGXN543SB4D9XTA]" duration=134.311082ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E5C32V0QQ87DXKDZP7Z7 01J8T5E5C3Q4NNNGP4W1DK1EBC]" block_count=2 blocks=01J8T5E55PZCVD0Y1AAGR2Y591,01J8T5E53V3BGXN543SB4D9XTA duration=134.430174ms duration_ms=134 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E5C3Q4NNNGP4W1DK1EBC/chunks/000001 dst=01J8T5E5C3Q4NNNGP4W1DK1EBC/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E5C32V0QQ87DXKDZP7Z7/chunks/000001 dst=01J8T5E5C32V0QQ87DXKDZP7Z7/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E5C3Q4NNNGP4W1DK1EBC/index dst=01J8T5E5C3Q4NNNGP4W1DK1EBC/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E5C32V0QQ87DXKDZP7Z7/index dst=01J8T5E5C32V0QQ87DXKDZP7Z7/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split866083122/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E5C3Q4NNNGP4W1DK1EBC duration=1.939899ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E5C32V0QQ87DXKDZP7Z7 duration=1.768909ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=4.35439ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E55PZCVD0Y1AAGR2Y591 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E55PZCVD0Y1AAGR2Y591 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E53V3BGXN543SB4D9XTA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E53V3BGXN543SB4D9XTA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=144.66[67](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:68)13ms duration_ms=144 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=2.[68](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:69)5444ms duration_ms=2 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.47087ms duration_ms=3 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454975 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.637686ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@1724170925407737[69](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:70)21-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01J8T5E5J0FY0BSF3YNMY8CQXC (min time: 19[70](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:71)-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01J8T5E5N3EAMPCYAP1QW2ZWAY (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E5J0FY0BSF3YNMY8CQXC,01J8T5E5N3EAMPCYAP1QW2ZWAY level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E5J0FY0BSF3YNMY8CQXC/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E5N3EAMPCYAP1QW2ZWAY/meta.json level=info component=cleaner run_id=1727454975 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=31.59829ms level=info component=cleaner run_id=1727454975 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E5J0FY0BSF3YNMY8CQXC,01J8T5E5N3EAMPCYAP1QW2ZWAY duration=7.557943ms duration_ms=7 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01J8T5E5Q121V3G966JJWQ39XE sources="[01J8T5E5N3EAMPCYAP1QW2ZWAY 01J8T5E5J0FY0BSF3YNMY8CQXC]" duration=115.777478ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01J8T5E5Q1XGFEYTHWXK7H8M22 sources="[01J8T5E5N3EAMPCYAP1QW2ZWAY 01J8T5E5J0FY0BSF3YNMY8CQXC]" duration=115.873978ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E5Q121V3G966JJWQ39XE 01J8T5E5Q1XGFEYTHWXK7H8M22]" block_count=2 blocks=01J8T5E5J0FY0BSF3YNMY8CQXC,01J8T5E5N3EAMPCYAP1QW2ZWAY duration=116.034798ms duration_ms=116 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E5Q1XGFEYTHWXK7H8M22/chunks/000001 dst=01J8T5E5Q1XGFEYTHWXK7H8M22/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E5Q1XGFEYTHWXK7H8M22/index dst=01J8T5E5Q1XGFEYTHWXK7H8M22/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E5Q121V3G966JJWQ39XE/chunks/000001 dst=01J8T5E5Q121V3G966JJWQ39XE/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E5Q1XGFEYTHWXK7H8M22 duration=3.326429ms duration_ms=3 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E5Q121V3G966JJWQ39XE/index dst=01J8T5E5Q121V3G966JJWQ39XE/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split3213369274/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E5Q121V3G966JJWQ39XE duration=2.0[71](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:72)204ms duration_ms=2 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@1[72](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:73)41709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=9.047944ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@1724170925407[73](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:74)76921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E5J0FY0BSF3YNMY8CQXC level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E5J0FY0BSF3YNMY8CQXC level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E5N3EAMPCYAP1QW2ZWAY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E5N3EAMPCYAP1QW2ZWAY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction job succeeded" duration=135.39143ms duration_ms=135 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.11591ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.590707ms duration_ms=3 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=172[74](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:75)54976 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.283865ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01J8T5E5WNPBDZDQBDQ66XTY7E (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01J8T5E5YAMFKZDEYYR6X5G8HS (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=6.660719ms level=info component=compactor user=user-1 msg="start of compactions" level=info component=cleaner run_id=1727454976 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E5WNPBDZDQBDQ66XTY7E,01J8T5E5YAMFKZDEYYR6X5G8HS level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E5WNPBDZDQBDQ66XTY7E/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E5YAMFKZDEYYR6X5G8HS/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E5WNPBDZDQBDQ66XTY7E,01J8T5E5YAMFKZDEYYR6X5G8HS duration=2.71945ms duration_ms=2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01J8T5E6568WJ57WZHMWX6GWH2 sources="[01J8T5E5WNPBDZDQBDQ66XTY7E 01J8T5E5YAMFKZDEYYR6X5G8HS]" duration=46.843018ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01J8T5E6569QBQZMMWECGJNVBX sources="[01J8T5E5WNPBDZDQBDQ66XTY7E 01J8T5E5YAMFKZDEYYR6X5G8HS]" duration=46.994ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E6568WJ57WZHMWX6GWH2 01J8T5E6569QBQZMMWECGJNVBX]" block_count=2 blocks=01J8T5E5WNPBDZDQBDQ66XTY7E,01J8T5E5YAMFKZDEYYR6X5G8HS duration=47.185649ms duration_ms=47 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6568WJ57WZHMWX6GWH2/chunks/000001 dst=01J8T5E6568WJ57WZHMWX6GWH2/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6568WJ57WZHMWX6GWH2/index dst=01J8T5E6568WJ57WZHMWX6GWH2/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E6568WJ57WZHMWX6GWH2 duration=1.956141ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6569QBQZMMWECGJNVBX/chunks/000001 dst=01J8T5E6569QBQZMMWECGJNVBX/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6569QBQZMMWECGJNVBX/index dst=01J8T5E6569QBQZMMWECGJNVBX/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split948583102/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E6569QBQZMMWECGJNVBX duration=2.027823ms duration_ms=2 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=6.086806ms duration_ms=6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E5WNPBDZDQBDQ66XTY7E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E5WNPBDZDQBDQ66XTY7E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E5YAMFKZDEYYR6X5G8HS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E5YAMFKZDEYYR6X5G8HS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=58.706301ms duration_ms=58 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.984359ms duration_ms=3 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.216984ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1727454976 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454976 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=5.013267ms level=info component=cleaner run_id=1727454976 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=5.456706ms duration_ms=5 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01J8T5E6BAXWPHJGJ2M440ER7Z (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01J8T5E6FNRVBKSCVDREB7K01T (min time: 1970-01-01 00:07:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E6BAXWPHJGJ2M440ER7Z,01J8T5E6FNRVBKSCVDREB7K01T level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E6FNRVBKSCVDREB7K01T/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E6BAXWPHJGJ2M440ER7Z/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E6BAXWPHJGJ2M440ER7Z,01J8T5E6FNRVBKSCVDREB7K01T duration=9.701376ms duration_ms=9 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01J8T5E6J43ACNHFATMK2R4SE0 sources="[01J8T5E6BAXWPHJGJ2M440ER7Z 01J8T5E6FNRVBKSCVDREB7K01T]" duration=93.489456ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01J8T5E6J4Y3AHXF74BCBR0XXE sources="[01J8T5E6BAXWPHJGJ2M440ER7Z 01J8T5E6FNRVBKSCVDREB7K01T]" duration=93.602086ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E6J43ACNHFATMK2R4SE0 01J8T5E6J4Y3AHXF74BCBR0XXE]" block_count=2 blocks=01J8T5E6BAXWPHJGJ2M440ER7Z,01J8T5E6FNRVBKSCVDREB7K01T duration=93.760652ms duration_ms=93 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6J4Y3AHXF74BCBR0XXE/chunks/000001 dst=01J8T5E6J4Y3AHXF74BCBR0XXE/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6J4Y3AHXF74BCBR0XXE/index dst=01J8T5E6J4Y3AHXF74BCBR0XXE/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E6J4Y3AHXF74BCBR0XXE duration=1.793564ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6J43ACNHFATMK2R4SE0/chunks/000001 dst=01J8T5E6J43ACNHFATMK2R4SE0/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E6J43ACNHFATMK2R4SE0/index dst=01J8T5E6J43ACNHFATMK2R4SE0/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1515032291/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E6J43ACNHFATMK2R4SE0 duration=8.635793ms duration_ms=8 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=12.081318ms duration_ms=12 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E6BAXWPHJGJ2M440ER7Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E6BAXWPHJGJ2M440ER7Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E6FNRVBKSCVDREB7K01T level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E6FNRVBKSCVDREB7K01T level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=118.225633ms duration_ms=118 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=9.521971ms duration_ms=9 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.320142ms duration_ms=6 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1727454977 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=10 existing=0 level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=10 partial_errors=0 level=info component=cleaner run_id=1727454977 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=10.3[75](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:76)991ms level=info component=cleaner run_id=1727454977 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=12.133753ms duration_ms=12 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@172417092540773[76](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:77)921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01J8T5E70ZN62GBVSN9CRHJGP5 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01J8T5E73NXBN0MRGP0MZ8KZD3 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@172417092540[77](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:78)376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01J8T5E72BVY4F0MKC52CJ8M8E (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01J8T5E756YDGZB5G7ZFNTB3ZD (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job="stage: split, range start: 28800000, range end: 36000000, shard: 1_of_1, blocks: 01J8T5E76W8P8B1NKCCKRM3RCQ (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job="stage: split, range start: 50400000, range end: 57600000, shard: 1_of_1, blocks: 01J8T5E[78](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:79)NA2CF87NHYQ30378G (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01J8T5E6XTB3X5DXYYPSB82XA6 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01J8T5E6ZBAXWWDCAR07KT0FV7 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E6ZBAXWWDCAR07KT0FV7,01J8T5E6XTB3X5DXYYPSB82XA6 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E6ZBAXWWDCAR07KT0FV7/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E6XTB3X5DXYYPSB82XA6/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E6ZBAXWWDCAR07KT0FV7,01J8T5E6XTB3X5DXYYPSB82XA6 duration=2.806497ms duration_ms=2 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E7FZVHXP06NJ74AQR210 sources="[01J8T5E6ZBAXWWDCAR07KT0FV7 01J8T5E6XTB3X5DXYYPSB82XA6]" duration=42.641066ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E7FZ14GVBXY2WH2H8DM3 sources="[01J8T5E6ZBAXWWDCAR07KT0FV7 01J8T5E6XTB3X5DXYYPSB82XA6]" duration=42.729301ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E7FZVHXP06NJ74AQR210 01J8T5E7FZ14GVBXY2WH2H8DM3]" block_count=2 blocks=01J8T5E6ZBAXWWDCAR07KT0FV7,01J8T5E6XTB3X5DXYYPSB82XA6 duration=42.818818ms duration_ms=42 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges[79](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:80)3854752/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E7FZ14GVBXY2WH2H8DM3/chunks/000001 dst=01J8T5E7FZ14GVBXY2WH2H8DM3/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E7FZVHXP06NJ74AQR210/chunks/000001 dst=01J8T5E7FZVHXP06NJ74AQR210/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E7FZ14GVBXY2WH2H8DM3/index dst=01J8T5E7FZ14GVBXY2WH2H8DM3/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5E7FZVHXP06NJ74AQR210/index dst=01J8T5E7FZVHXP06NJ74AQR210/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7FZ14GVBXY2WH2H8DM3 duration=1.85983ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7FZVHXP06NJ74AQR210 duration=1.867674ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=4.401931ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E6ZBAXWWDCAR07KT0FV7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E6ZBAXWWDCAR07KT0FV7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E6XTB3X5DXYYPSB82XA6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E6XTB3X5DXYYPSB82XA6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=52.247651ms duration_ms=52 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28[80](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:81)0000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=1 blocks=01J8T5E76W8P8B1NKCCKRM3RCQ level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E76W8P8B1NKCCKRM3RCQ/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=1 blocks=01J8T5E76W8P8B1NKCCKRM3RCQ duration=2.053121ms duration_ms=2 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01J8T5E7HKSPAKNK0XSQZEEC3C sources=[01J8T5E76W8P8B1NKCCKRM3RCQ] duration=153.556724ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01J8T5E7HKN0D9ZVZYH852XQJZ sources=[01J8T5E76W8P8B1NKCCKRM3RCQ] duration=153.625212ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E7HKSPAKNK0XSQZEEC3C 01J8T5E7HKN0D9ZVZYH852XQJZ]" block_count=1 blocks=01J8T5E76W8P8B1NKCCKRM3RCQ duration=153.727523ms duration_ms=153 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01J8T5E7HKN0D9ZVZYH852XQJZ/chunks/000001 dst=01J8T5E7HKN0D9ZVZYH852XQJZ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01J8T5E7HKSPAKNK0XSQZEEC3C/chunks/000001 dst=01J8T5E7HKSPAKNK0XSQZEEC3C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01J8T5E7HKSPAKNK0XSQZEEC3C/index dst=01J8T5E7HKSPAKNK0XSQZEEC3C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01J8T5E7HKN0D9ZVZYH852XQJZ/index dst=01J8T5E7HKN0D9ZVZYH852XQJZ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7HKSPAKNK0XSQZEEC3C duration=1.[81](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:82)0618ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7HKN0D9ZVZYH852XQJZ duration=1.949437ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=4.345079ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E76W8P8B1NKCCKRM3RCQ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E76W8P8B1NKCCKRM3RCQ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job_type=split minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction job succeeded" duration=161.330166ms duration_ms=161 block_count=1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=1 blocks=01J8T5E78NA2CF87NHYQ30378G level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E78NA2CF87NHYQ30378G/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=1 blocks=01J8T5E78NA2CF87NHYQ30378G duration=2.055144ms duration_ms=2 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01J8T5E7PMG8VG241M13RQARRA sources=[01J8T5E78NA2CF87NHYQ30378G] duration=145.027054ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01J8T5E7PMBBVZDVKBY9RB5JM5 sources=[01J8T5E78NA2CF87NHYQ30378G] duration=145.115018ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E7PMG8VG241M13RQARRA 01J8T5E7PMBBVZDVKBY9RB5JM5]" block_count=1 blocks=01J8T5E78NA2CF87NHYQ30378G duration=145.230665ms duration_ms=145 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01J8T5E7PMG8VG241M13RQARRA/chunks/000001 dst=01J8T5E7PMG8VG241M13RQARRA/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01J8T5E7PMG8VG241M13RQARRA/index dst=01J8T5E7PMG8VG241M13RQARRA/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7PMG8VG241M13RQARRA duration=1.873596ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01J8T5E7PMBBVZDVKBY9RB5JM5/chunks/000001 dst=01J8T5E7PMBBVZDVKBY9RB5JM5/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01J8T5E7PMBBVZDVKBY9RB5JM5/index dst=01J8T5E7PMBBVZDVKBY9RB5JM5/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7PMBBVZDVKBY9RB5JM5 duration=1.92399ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=8.30123ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E78NA2CF87NHYQ30378G level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E78NA2CF87NHYQ30378G level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job_type=split minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=157.181593ms duration_ms=157 block_count=1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E70ZN62GBVSN9CRHJGP5,01J8T5E73NXBN0MRGP0MZ8KZD3 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E70ZN62GBVSN9CRHJGP5/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E73NXBN0MRGP0MZ8KZD3/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E70ZN62GBVSN9CRHJGP5,01J8T5E73NXBN0MRGP0MZ8KZD3 duration=5.640702ms duration_ms=5 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01J8T5E7VQZJ7XQETXTD051CZB sources="[01J8T5E70ZN62GBVSN9CRHJGP5 01J8T5E73NXBN0MRGP0MZ8KZD3]" duration=20.275001ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E7VQZJ7XQETXTD051CZB] block_count=2 blocks=01J8T5E70ZN62GBVSN9CRHJGP5,01J8T5E73NXBN0MRGP0MZ8KZD3 duration=20.417326ms duration_ms=20 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01J8T5E7VQZJ7XQETXTD051CZB/chunks/000001 dst=01J8T5E7VQZJ7XQETXTD051CZB/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01J8T5E7VQZJ7XQETXTD051CZB/index dst=01J8T5E7VQZJ7XQETXTD051CZB/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7VQZJ7XQETXTD051CZB duration=1.910003ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=5.199006ms duration_ms=5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E70ZN62GBVSN9CRHJGP5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E70ZN62GBVSN9CRHJGP5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E73NXBN0MRGP0MZ8KZD3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E73NXBN0MRGP0MZ8KZD3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=33.709223ms duration_ms=33 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E72BVY4F0MKC52CJ8M8E,01J8T5E756YDGZB5G7ZFNTB3ZD level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E72BVY4F0MKC52CJ8M8E/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E756YDGZB5G7ZFNTB3ZD/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E72BVY4F0MKC52CJ8M8E,01J8T5E756YDGZB5G7ZFNTB3ZD duration=3.003579ms duration_ms=3 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01J8T5E7WPYNTYQ4JCY7HH075P sources="[01J8T5E72BVY4F0MKC52CJ8M8E 01J8T5E756YDGZB5G7ZFNTB3ZD]" duration=16.572183ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E7WPYNTYQ4JCY7HH075P] block_count=2 blocks=01J8T5E72BVY4F0MKC52CJ8M8E,01J8T5E756YDGZB5G7ZFNTB3ZD duration=16.68807ms duration_ms=16 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01J8T5E7WPYNTYQ4JCY7HH075P/chunks/000001 dst=01J8T5E7WPYNTYQ4JCY7HH075P/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01J8T5E7WPYNTYQ4JCY7HH075P/index dst=01J8T5E7WPYNTYQ4JCY7HH075P/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7WPYNTYQ4JCY7HH075P duration=1.421439ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=3.542177ms duration_ms=3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E72BVY4F0MKC52CJ8M8E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E72BVY4F0MKC52CJ8M8E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E756YDGZB5G7ZFNTB3ZD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E756YDGZB5G7ZFNTB3ZD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=25.24011ms duration_ms=25 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=9.11[82](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:83)4ms duration_ms=9 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 1_of_2, blocks: 01J8T5E6SCZ6E355XXFMV6RA6C (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01J8T5E7FZVHXP06NJ74AQR210 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 2_of_2, blocks: 01J8T5E6V53SMSZHB4DAZAWAMS (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01J8T5E7FZ14GVBXY2WH2H8DM3 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 1_of_2, blocks: 01J8T5E7HKSPAKNK0XSQZEEC3C (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01J8T5E7PMG8VG241M13RQARRA (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 2_of_2, blocks: 01J8T5E7HKN0D9ZVZYH852XQJZ (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01J8T5E7PMBBVZDVKBY9RB5JM5 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E6SCZ6E355XXFMV6RA6C,01J8T5E7FZVHXP06NJ74AQR210 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E6SCZ6E355XXFMV6RA6C/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7FZVHXP06NJ74AQR210/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E6SCZ6E355XXFMV6RA6C,01J8T5E7FZVHXP06NJ74AQR210 duration=2.819475ms duration_ms=2 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01J8T5E7XTYJQATZHFXBPVF505 sources="[01J8T5E7FZVHXP06NJ74AQR210 01J8T5E6SCZ6E355XXFMV6RA6C]" duration=28.012604ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E7XTYJQATZHFXBPVF505] block_count=2 blocks=01J8T5E6SCZ6E355XXFMV6RA6C,01J8T5E7FZVHXP06NJ74AQR210 duration=28.180008ms duration_ms=28 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01J8T5E7XTYJQATZHFXBPVF505/chunks/000001 dst=01J8T5E7XTYJQATZHFXBPVF505/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01J8T5E7XTYJQATZHFXBPVF505/index dst=01J8T5E7XTYJQATZHFXBPVF505/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7XTYJQATZHFXBPVF505 duration=2.118272ms duration_ms=2 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=4.802404ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E6SCZ6E355XXFMV6RA6C level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E6SCZ6E355XXFMV6RA6C level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7FZVHXP06NJ74AQR210 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7FZVHXP06NJ74AQR210 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=38.50[83](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:84)8ms duration_ms=38 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E6V53SMSZHB4DAZAWAMS,01J8T5E7FZ14GVBXY2WH2H8DM3 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E6V53SMSZHB4DAZAWAMS/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7FZ14GVBXY2WH2H8DM3/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E6V53SMSZHB4DAZAWAMS,01J8T5E7FZ14GVBXY2WH2H8DM3 duration=4.089756ms duration_ms=4 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01J8T5E7Z3YZH7CRX16K8S6E9H sources="[01J8T5E7FZ14GVBXY2WH2H8DM3 01J8T5E6V53SMSZHB4DAZAWAMS]" duration=18.807245ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E7Z3YZH7CRX16K8S6E9H] block_count=2 blocks=01J8T5E6V53SMSZHB4DAZAWAMS,01J8T5E7FZ14GVBXY2WH2H8DM3 duration=19.042725ms duration_ms=19 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01J8T5E7Z3YZH7CRX16K8S6E9H/chunks/000001 dst=01J8T5E7Z3YZH7CRX16K8S6E9H/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01J8T5E7Z3YZH7CRX16K8S6E9H/index dst=01J8T5E7Z3YZH7CRX16K8S6E9H/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E7Z3YZH7CRX16K8S6E9H duration=2.204655ms duration_ms=2 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=4.863498ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E6V53SMSZHB4DAZAWAMS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E6V53SMSZHB4DAZAWAMS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7FZ14GVBXY2WH2H8DM3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7FZ14GVBXY2WH2H8DM3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=30.767705ms duration_ms=30 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E7HKSPAKNK0XSQZEEC3C,01J8T5E7PMG8VG241M13RQARRA level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7PMG8VG241M13RQARRA/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7HKSPAKNK0XSQZEEC3C/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E7HKSPAKNK0XSQZEEC3C,01J8T5E7PMG8VG241M13RQARRA duration=2.508573ms duration_ms=2 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01J8T5E800WB8HRT0W5KAKWR27 sources="[01J8T5E7HKSPAKNK0XSQZEEC3C 01J8T5E7PMG8VG241M13RQARRA]" duration=20.824288ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E800WB8HRT0W5KAKWR27] block_count=2 blocks=01J8T5E7HKSPAKNK0XSQZEEC3C,01J8T5E7PMG8VG241M13RQARRA duration=21.062664ms duration_ms=21 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01J8T5E800WB8HRT0W5KAKWR27/chunks/000001 dst=01J8T5E800WB8HRT0W5KAKWR27/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01J8T5E800WB8HRT0W5KAKWR27/index dst=01J8T5E800WB8HRT0W5KAKWR27/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E800WB8HRT0W5KAKWR27 duration=2.062188ms duration_ms=2 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=4.403217ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7HKSPAKNK0XSQZEEC3C level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7HKSPAKNK0XSQZEEC3C level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7PMG8VG241M13RQARRA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7PMG8VG241M13RQARRA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=30.712547ms duration_ms=30 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E7HKN0D9ZVZYH852XQJZ,01J8T5E7PMBBVZDVKBY9RB5JM5 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7PMBBVZDVKBY9RB5JM5/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7HKN0D9ZVZYH852XQJZ/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E7HKN0D9ZVZYH852XQJZ,01J8T5E7PMBBVZDVKBY9RB5JM5 duration=3.974976ms duration_ms=3 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01J8T5E811G3095QRHZB01WXE1 sources="[01J8T5E7HKN0D9ZVZYH852XQJZ 01J8T5E7PMBBVZDVKBY9RB5JM5]" duration=20.968614ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E811G3095QRHZB01WXE1] block_count=2 blocks=01J8T5E7HKN0D9ZVZYH852XQJZ,01J8T5E7PMBBVZDVKBY9RB5JM5 duration=21.173318ms duration_ms=21 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01J8T5E811G3095QRHZB01WXE1/chunks/000001 dst=01J8T5E811G3095QRHZB01WXE1/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01J8T5E811G3095QRHZB01WXE1/index dst=01J8T5E811G3095QRHZB01WXE1/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E811G3095QRHZB01WXE1 duration=1.9671ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=4.507002ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7HKN0D9ZVZYH852XQJZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7HKN0D9ZVZYH852XQJZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7PMBBVZDVKBY9RB5JM5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7PMBBVZDVKBY9RB5JM5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=32.647613ms duration_ms=32 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=7.922131ms duration_ms=7 cached=6 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01J8T5E7VQZJ7XQETXTD051CZB (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC),01J8T5E7XTYJQATZHFXBPVF505 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01J8T5E7WPYNTYQ4JCY7HH075P (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC),01J8T5E7Z3YZH7CRX16K8S6E9H (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E7XTYJQATZHFXBPVF505,01J8T5E7VQZJ7XQETXTD051CZB level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7XTYJQATZHFXBPVF505/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7VQZJ7XQETXTD051CZB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E7XTYJQATZHFXBPVF505,01J8T5E7VQZJ7XQETXTD051CZB duration=3.752248ms duration_ms=3 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01J8T5E82CTC6NQDF54QD0HB8J sources="[01J8T5E7XTYJQATZHFXBPVF505 01J8T5E7VQZJ7XQETXTD051CZB]" duration=23.440723ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E82CTC6NQDF54QD0HB8J] block_count=2 blocks=01J8T5E7XTYJQATZHFXBPVF505,01J8T5E7VQZJ7XQETXTD051CZB duration=23.601263ms duration_ms=23 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01J8T5E82CTC6NQDF54QD0HB8J/chunks/000001 dst=01J8T5E82CTC6NQDF54QD0HB8J/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01J8T5E82CTC6NQDF54QD0HB8J/index dst=01J8T5E82CTC6NQDF54QD0HB8J/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E82CTC6NQDF54QD0HB8J duration=1.9[84](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:85)612ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.567062ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7XTYJQATZHFXBPVF505 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7XTYJQATZHFXBPVF505 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7VQZJ7XQETXTD051CZB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7VQZJ7XQETXTD051CZB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=38.350719ms duration_ms=38 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E7Z3YZH7CRX16K8S6E9H,01J8T5E7WPYNTYQ4JCY7HH075P level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7Z3YZH7CRX16K8S6E9H/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E7WPYNTYQ4JCY7HH075P/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E7Z3YZH7CRX16K8S6E9H,01J8T5E7WPYNTYQ4JCY7HH075P duration=3.245489ms duration_ms=3 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01J8T5E83J4F138S6PP7KG7V1G sources="[01J8T5E7WPYNTYQ4JCY7HH075P 01J8T5E7Z3YZH7CRX16K8S6E9H]" duration=25.2[85](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:86)624ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E83J4F138S6PP7KG7V1G] block_count=2 blocks=01J8T5E7Z3YZH7CRX16K8S6E9H,01J8T5E7WPYNTYQ4JCY7HH075P duration=25.453959ms duration_ms=25 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01J8T5E83J4F138S6PP7KG7V1G/chunks/000001 dst=01J8T5E83J4F138S6PP7KG7V1G/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01J8T5E83J4F138S6PP7KG7V1G/index dst=01J8T5E83J4F138S6PP7KG7V1G/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges793854752/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E83J4F138S6PP7KG7V1G duration=1.740656ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=5.172817ms duration_ms=5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7Z3YZH7CRX16K8S6E9H level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7Z3YZH7CRX16K8S6E9H level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E7WPYNTYQ4JCY7HH075P level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E7WPYNTYQ4JCY7HH075P level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=36.310136ms duration_ms=36 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=6.989726ms duration_ms=6 cached=4 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=8.670992ms duration_ms=8 cached=4 returned=4 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=cleaner run_id=1727454978 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=3 existing=0 level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=3 partial_errors=0 level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454978 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=8.970322ms level=info component=cleaner run_id=1727454978 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=7.781688ms duration_ms=7 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01J8T5E8A0FT638MR8DT5K99NT (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01J8T5E8C3MW77ZA02BN4EZEV1 (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01J8T5E8E3AVWMGV9XKNN2G5B2 (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=3 blocks=01J8T5E8A0FT638MR8DT5K99NT,01J8T5E8C3MW77ZA02BN4EZEV1,01J8T5E8E3AVWMGV9XKNN2G5B2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E8E3AVWMGV9XKNN2G5B2/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E8C3MW77ZA02BN4EZEV1/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E8A0FT638MR8DT5K99NT/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=3 blocks=01J8T5E8A0FT638MR8DT5K99NT,01J8T5E8C3MW77ZA02BN4EZEV1,01J8T5E8E3AVWMGV9XKNN2G5B2 duration=8.414261ms duration_ms=8 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01J8T5E8FV2EDNPSMY8PNA3D7C sources="[01J8T5E8C3MW77ZA02BN4EZEV1 01J8T5E8E3AVWMGV9XKNN2G5B2 01J8T5E8A0FT638MR8DT5K99NT]" duration=240.995049ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01J8T5E8FVR1J4SAENWH9CQHD8 sources="[01J8T5E8C3MW77ZA02BN4EZEV1 01J8T5E8E3AVWMGV9XKNN2G5B2 01J8T5E8A0FT638MR8DT5K99NT]" duration=241.245448ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5E8FV2EDNPSMY8PNA3D7C 01J8T5E8FVR1J4SAENWH9CQHD8]" block_count=3 blocks=01J8T5E8A0FT638MR8DT5K99NT,01J8T5E8C3MW77ZA02BN4EZEV1,01J8T5E8E3AVWMGV9XKNN2G5B2 duration=241.484215ms duration_ms=241 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E8FV2EDNPSMY8PNA3D7C/chunks/000001 dst=01J8T5E8FV2EDNPSMY8PNA3D7C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E8FVR1J4SAENWH9CQHD8/chunks/000001 dst=01J8T5E8FVR1J4SAENWH9CQHD8/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E8FVR1J4SAENWH9CQHD8/index dst=01J8T5E8FVR1J4SAENWH9CQHD8/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E8FVR1J4SAENWH9CQHD8 duration=1.74842ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01J8T5E8FV2EDNPSMY8PNA3D7C/index dst=01J8T5E8FV2EDNPSMY8PNA3D7C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together2272059814/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E8FV2EDNPSMY8PNA3D7C duration=4.385425ms duration_ms=4 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=8.75554ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E8A0FT638MR8DT5K99NT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E8A0FT638MR8DT5K99NT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E8C3MW77ZA02BN4EZEV1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E8C3MW77ZA02BN4EZEV1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E8E3AVWMGV9XKNN2G5B2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E8E3AVWMGV9XKNN2G5B2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job_type=split minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=262.416084ms duration_ms=262 block_count=3 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.140837ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=10.069649ms duration_ms=10 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454979 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=10 existing=0 level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=10 partial_errors=0 level=info component=cleaner run_id=1727454979 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=12.284716ms level=info component=cleaner run_id=1727454979 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=16.104768ms duration_ms=16 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01J8T5E965A96QZ7AW38G1YPSH (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01J8T5E9AFZRQ4VK53BFCEY9EF (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01J8T5E98KSVZKJN9DN0VTKDCF (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01J8T5E9CBK57AGFJ9JNNZB266 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: , blocks: 01J8T5E9E61ECZDS99WSY0ZA1K (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01J8T5E9FZ47YRAQZ9CY039AZ9 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--7200000-14400000 job="stage: merge, range start: 7200000, range end: 14400000, shard: , blocks: 01J8T5E8Z7DZP5M583ESZFJ2BY (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01J8T5E9134EGY6VTFNXA6R0K7 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E8Z7DZP5M583ESZFJ2BY,01J8T5E9134EGY6VTFNXA6R0K7 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E8Z7DZP5M583ESZFJ2BY/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9134EGY6VTFNXA6R0K7/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E8Z7DZP5M583ESZFJ2BY,01J8T5E9134EGY6VTFNXA6R0K7 duration=5.723658ms duration_ms=5 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5E9HSCXV4ZXH85TCDGTGB sources="[01J8T5E8Z7DZP5M583ESZFJ2BY 01J8T5E9134EGY6VTFNXA6R0K7]" duration=22.740373ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E9HSCXV4ZXH85TCDGTGB] block_count=2 blocks=01J8T5E8Z7DZP5M583ESZFJ2BY,01J8T5E9134EGY6VTFNXA6R0K7 duration=22.959803ms duration_ms=22 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge--7200000-14400000/01J8T5E9HSCXV4ZXH85TCDGTGB/chunks/000001" dst=01J8T5E9HSCXV4ZXH85TCDGTGB/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge--7200000-14400000/01J8T5E9HSCXV4ZXH85TCDGTGB/index" dst=01J8T5E9HSCXV4ZXH85TCDGTGB/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E9HSCXV4ZXH85TCDGTGB duration=2.047353ms duration_ms=2 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=5.277066ms duration_ms=5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E8Z7DZP5M583ESZFJ2BY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E8Z7DZP5M583ESZFJ2BY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9134EGY6VTFNXA6R0K7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9134EGY6VTFNXA6R0K7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 job_type=merge minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=36.762749ms duration_ms=36 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E965A96QZ7AW38G1YPSH,01J8T5E9AFZRQ4VK53BFCEY9EF level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E965A96QZ7AW38G1YPSH/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9AFZRQ4VK53BFCEY9EF/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E965A96QZ7AW38G1YPSH,01J8T5E9AFZRQ4VK53BFCEY9EF duration=2.76845ms duration_ms=2 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01J8T5E9JVR634R51F5GGXMETP sources="[01J8T5E965A96QZ7AW38G1YPSH 01J8T5E9AFZRQ4VK53BFCEY9EF]" duration=81.28829ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E9JVR634R51F5GGXMETP] block_count=2 blocks=01J8T5E965A96QZ7AW38G1YPSH,01J8T5E9AFZRQ4VK53BFCEY9EF duration=81.508803ms duration_ms=81 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01J8T5E9JVR634R51F5GGXMETP/chunks/000001" dst=01J8T5E9JVR634R51F5GGXMETP/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01J8T5E9JVR634R51F5GGXMETP/index" dst=01J8T5E9JVR634R51F5GGXMETP/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E9JVR634R51F5GGXMETP duration=2.102273ms duration_ms=2 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=9.610429ms duration_ms=9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E965A96QZ7AW38G1YPSH level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E965A96QZ7AW38G1YPSH level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9AFZRQ4VK53BFCEY9EF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9AFZRQ4VK53BFCEY9EF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=96.459943ms duration_ms=96 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E98KSVZKJN9DN0VTKDCF,01J8T5E9CBK57AGFJ9JNNZB266 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E98KSVZKJN9DN0VTKDCF/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9CBK57AGFJ9JNNZB266/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E98KSVZKJN9DN0VTKDCF,01J8T5E9CBK57AGFJ9JNNZB266 duration=9.3809ms duration_ms=9 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01J8T5E9P952J9GDZ4X1C6DFJ6 sources="[01J8T5E98KSVZKJN9DN0VTKDCF 01J8T5E9CBK57AGFJ9JNNZB266]" duration=45.96991ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E9P952J9GDZ4X1C6DFJ6] block_count=2 blocks=01J8T5E98KSVZKJN9DN0VTKDCF,01J8T5E9CBK57AGFJ9JNNZB266 duration=46.153573ms duration_ms=46 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01J8T5E9P952J9GDZ4X1C6DFJ6/chunks/000001" dst=01J8T5E9P952J9GDZ4X1C6DFJ6/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01J8T5E9P952J9GDZ4X1C6DFJ6/index" dst=01J8T5E9P952J9GDZ4X1C6DFJ6/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E9P952J9GDZ4X1C6DFJ6 duration=4.404109ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=7.134982ms duration_ms=7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E98KSVZKJN9DN0VTKDCF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E98KSVZKJN9DN0VTKDCF level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9CBK57AGFJ9JNNZB266 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9CBK57AGFJ9JNNZB266 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job_type=merge minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=65.131481ms duration_ms=65 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E9E61ECZDS99WSY0ZA1K,01J8T5E9FZ47YRAQZ9CY039AZ9 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9E61ECZDS99WSY0ZA1K/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9FZ47YRAQZ9CY039AZ9/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E9E61ECZDS99WSY0ZA1K,01J8T5E9FZ47YRAQZ9CY039AZ9 duration=6.892585ms duration_ms=6 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01J8T5E9RAA021CSXEFSQ8ESHH sources="[01J8T5E9E61ECZDS99WSY0ZA1K 01J8T5E9FZ47YRAQZ9CY039AZ9]" duration=20.733328ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E9RAA021CSXEFSQ8ESHH] block_count=2 blocks=01J8T5E9E61ECZDS99WSY0ZA1K,01J8T5E9FZ47YRAQZ9CY039AZ9 duration=20.915739ms duration_ms=20 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge--28800000-57600000/01J8T5E9RAA021CSXEFSQ8ESHH/chunks/000001" dst=01J8T5E9RAA021CSXEFSQ8ESHH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge--28800000-57600000/01J8T5E9RAA021CSXEFSQ8ESHH/index" dst=01J8T5E9RAA021CSXEFSQ8ESHH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E9RAA021CSXEFSQ8ESHH duration=2.060034ms duration_ms=2 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=4.569198ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9E61ECZDS99WSY0ZA1K level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9E61ECZDS99WSY0ZA1K level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9FZ47YRAQZ9CY039AZ9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9FZ47YRAQZ9CY039AZ9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 job_type=merge minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=34.929777ms duration_ms=34 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=7.88[86](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:87)05ms duration_ms=7 cached=6 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01J8T5E8VANDCFC6T7GDCVF6JV (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01J8T5E9JVR634R51F5GGXMETP (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01J8T5E8XBRCCGSDWT7C76X6Z6 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01J8T5E9P952J9GDZ4X1C6DFJ6 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E8VANDCFC6T7GDCVF6JV,01J8T5E9JVR634R51F5GGXMETP level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E8VANDCFC6T7GDCVF6JV/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9JVR634R51F5GGXMETP/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E8VANDCFC6T7GDCVF6JV,01J8T5E9JVR634R51F5GGXMETP duration=34.542185ms duration_ms=34 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01J8T5E9TYY99GVSYVV5WSDBK4 sources="[01J8T5E8VANDCFC6T7GDCVF6JV 01J8T5E9JVR634R51F5GGXMETP]" duration=53.064389ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E9TYY99GVSYVV5WSDBK4] block_count=2 blocks=01J8T5E8VANDCFC6T7GDCVF6JV,01J8T5E9JVR634R51F5GGXMETP duration=53.302435ms duration_ms=53 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01J8T5E9TYY99GVSYVV5WSDBK4/chunks/000001" dst=01J8T5E9TYY99GVSYVV5WSDBK4/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01J8T5E9TYY99GVSYVV5WSDBK4/index" dst=01J8T5E9TYY99GVSYVV5WSDBK4/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E9TYY99GVSYVV5WSDBK4 duration=9.568992ms duration_ms=9 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=18.141249ms duration_ms=18 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E8VANDCFC6T7GDCVF6JV level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E8VANDCFC6T7GDCVF6JV level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9JVR634R51F5GGXMETP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9JVR634R51F5GGXMETP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=108.599948ms duration_ms=108 block_count=2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5E8XBRCCGSDWT7C76X6Z6,01J8T5E9P952J9GDZ4X1C6DFJ6 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E9P952J9GDZ4X1C6DFJ6/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5E8XBRCCGSDWT7C76X6Z6/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5E8XBRCCGSDWT7C76X6Z6,01J8T5E9P952J9GDZ4X1C6DFJ6 duration=9.997897ms duration_ms=9 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01J8T5E9XF17V9GRT81X3GPT2Z sources="[01J8T5E9P952J9GDZ4X1C6DFJ6 01J8T5E8XBRCCGSDWT7C76X6Z6]" duration=40.718206ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5E9XF17V9GRT81X3GPT2Z] block_count=2 blocks=01J8T5E8XBRCCGSDWT7C76X6Z6,01J8T5E9P952J9GDZ4X1C6DFJ6 duration=40.902209ms duration_ms=40 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01J8T5E9XF17V9GRT81X3GPT2Z/chunks/000001" dst=01J8T5E9XF17V9GRT81X3GPT2Z/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01J8T5E9XF17V9GRT81X3GPT2Z/index" dst=01J8T5E9XF17V9GRT81X3GPT2Z/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0425547198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5E9XF17V9GRT81X3GPT2Z duration=4.9709ms duration_ms=4 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=8.966855ms duration_ms=8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E8XBRCCGSDWT7C76X6Z6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E8XBRCCGSDWT7C76X6Z6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5E9P952J9GDZ4X1C6DFJ6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5E9P952J9GDZ4X1C6DFJ6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job_type=merge minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=65.264059ms duration_ms=65 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=9.647198ms duration_ms=9 cached=4 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=10.732579ms duration_ms=10 cached=4 returned=4 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=cleaner run_id=1727454980 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=2 existing=0 level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=2 partial_errors=0 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.550382ms duration_ms=3 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01J8T5EA1QTA468NDWWJAP7V4X (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01J8T5EA3NYETNZQYSPWYPSJ64 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5EA1QTA468NDWWJAP7V4X,01J8T5EA3NYETNZQYSPWYPSJ64 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EA1QTA468NDWWJAP7V4X/meta.json level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=11.521129ms level=info component=cleaner run_id=1727454980 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EA3NYETNZQYSPWYPSJ64/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5EA1QTA468NDWWJAP7V4X,01J8T5EA3NYETNZQYSPWYPSJ64 duration=5.456127ms duration_ms=5 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5EA58K0SQ37KZJKB49A86 sources="[01J8T5EA1QTA468NDWWJAP7V4X 01J8T5EA3NYETNZQYSPWYPSJ64]" duration=174.616392ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01J8T5EA58VREFZ56YC6Q567FR sources="[01J8T5EA1QTA468NDWWJAP7V4X 01J8T5EA3NYETNZQYSPWYPSJ64]" duration=174.726809ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5EA58K0SQ37KZJKB49A86 01J8T5EA58VREFZ56YC6Q567FR]" block_count=2 blocks=01J8T5EA1QTA468NDWWJAP7V4X,01J8T5EA3NYETNZQYSPWYPSJ64 duration=174.892759ms duration_ms=174 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EA58K0SQ37KZJKB49A86/chunks/000001 dst=01J8T5EA58K0SQ37KZJKB49A86/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EA58K0SQ37KZJKB49A86/index dst=01J8T5EA58K0SQ37KZJKB49A86/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5EA58K0SQ37KZJKB49A86 duration=3.429737ms duration_ms=3 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EA58VREFZ56YC6Q567FR/chunks/000001 dst=01J8T5EA58VREFZ56YC6Q567FR/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EA58VREFZ56YC6Q567FR/index dst=01J8T5EA58VREFZ56YC6Q567FR/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split2329284346/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5EA58VREFZ56YC6Q567FR duration=1.859388ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=14.970136ms duration_ms=14 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EA1QTA468NDWWJAP7V4X level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EA1QTA468NDWWJAP7V4X level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EA3NYETNZQYSPWYPSJ64 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EA3NYETNZQYSPWYPSJ64 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=203.055866ms duration_ms=203 block_count=2 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=7.[87](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:88)8124ms duration_ms=7 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.736651ms duration_ms=3 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=e62aa77bf43f ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=cleaner run_id=1727454980 task=clean_up_users msg="started blocks cleanup and maintenance" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="fetched existing bucket index" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="listed deletion markers" count=0 level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="updated deletion markers for recently marked blocks" count=0 total_deletion_markers=0 level=info component=compactor user=user-1 msg="start sync of metas" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="listed all blocks in storage" newly_discovered=1 existing=0 level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="fetched blocks metas for newly discovered blocks" total_blocks=1 partial_errors=0 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.673671ms duration_ms=3 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01J8T5EAFZFA3ZY41KVVHBR2RB (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=cleaner run_id=1727454980 task=clean_up_users user=user-1 msg="completed blocks cleanup and maintenance" duration=4.371999ms level=info component=compactor user=user-1 msg="start of compactions" level=info component=cleaner run_id=1727454980 task=clean_up_users msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=1 blocks=01J8T5EAFZFA3ZY41KVVHBR2RB level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EAFZFA3ZY41KVVHBR2RB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=1 blocks=01J8T5EAFZFA3ZY41KVVHBR2RB duration=2.674523ms duration_ms=2 level=info component=compactor msg="compact blocks" count=1 mint=7200000 maxt=14400000 ulid=01J8T5EANY7W9V8730F8MF4RBA sources=[01J8T5EAFZFA3ZY41KVVHBR2RB] duration=42.227045ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=7200000 maxt=14400000 ulid=01J8T5EANY4A10M70ZEZPA5HKR sources=[01J8T5EAFZFA3ZY41KVVHBR2RB] duration=42.311844ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new_block_count=2 new_blocks="[01J8T5EANY7W9V8730F8MF4RBA 01J8T5EANY4A10M70ZEZPA5HKR]" block_count=1 blocks=01J8T5EAFZFA3ZY41KVVHBR2RB duration=42.401131ms duration_ms=42 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EANY4A10M70ZEZPA5HKR/chunks/000001 dst=01J8T5EANY4A10M70ZEZPA5HKR/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EANY7W9V8730F8MF4RBA/chunks/000001 dst=01J8T5EANY7W9V8730F8MF4RBA/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EANY4A10M70ZEZPA5HKR/index dst=01J8T5EANY4A10M70ZEZPA5HKR/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01J8T5EANY7W9V8730F8MF4RBA/index dst=01J8T5EANY7W9V8730F8MF4RBA/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldGuaranteeSeriesShardingConsistencyOverTheTime1963648484/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5EANY4A10M70ZEZPA5HKR duration=1.711101ms duration_ms=1 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01J8T5EANY7W9V8730F8MF4RBA duration=1.65157ms duration_ms=1 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=4.189328ms duration_ms=4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EAFZFA3ZY41KVVHBR2RB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EAFZFA3ZY41KVVHBR2RB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job_type=split minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=50.731145ms duration_ms=50 block_count=1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.MetaFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.MetaFetcher msg="successfully synchronized block metadata" duration=3.606177ms duration_ms=3 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.MetaFetcher msg="fetching meta data" concurrency=1 level=info component=block.MetaFetcher msg="successfully synchronized block metadata" duration=4.074833ms duration_ms=4 cached=2 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket=inmem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2876173640/001" level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2876173640/001" level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2876173640/001" level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem2876173640/001" level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01J8T5ECYG7NQ1WP0YEMKCTHDJ (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01J8T5ED052TDJ5S0YE0PYDNAQ (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01J8T5ED116MSNSHM0MNS5BDXG (min time: 1970-01-01 00:00:01 +0000 UTC, max time: 1970-01-01 00:00:02 +0000 UTC)" level=debug msg="grouper found a compactable blocks group" groupKey=124@8084862141041806226-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01J8T5ED5WAQ8T39B15MVE4ETC (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01J8T5ED90Z9KBN5RAQEP224BW (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5ED90Z9KBN5RAQEP224BW,01J8T5ED5WAQ8T39B15MVE4ETC level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=3 blocks=01J8T5ECYG7NQ1WP0YEMKCTHDJ,01J8T5ED116MSNSHM0MNS5BDXG,01J8T5ED052TDJ5S0YE0PYDNAQ level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5ED90Z9KBN5RAQEP224BW/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5ECYG7NQ1WP0YEMKCTHDJ/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5ED052TDJ5S0YE0PYDNAQ/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5ED5WAQ8T39B15MVE4ETC/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5ED116MSNSHM0MNS5BDXG/meta.json level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=3 blocks=01J8T5ECYG7NQ1WP0YEMKCTHDJ,01J8T5ED116MSNSHM0MNS5BDXG,01J8T5ED052TDJ5S0YE0PYDNAQ duration=2.808434ms duration_ms=2 level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5ED90Z9KBN5RAQEP224BW,01J8T5ED5WAQ8T39B15MVE4ETC duration=3.283031ms duration_ms=3 level=info msg="compact blocks" count=2 mint=0 maxt=3000 ulid=01J8T5EDDXY6828S4XDFWTY562 sources="[01J8T5ED90Z9KBN5RAQEP224BW 01J8T5ED5WAQ8T39B15MVE4ETC]" duration=16.320282ms shard=1_of_1 level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5EDDXY6828S4XDFWTY562] block_count=2 blocks=01J8T5ED90Z9KBN5RAQEP224BW,01J8T5ED5WAQ8T39B15MVE4ETC duration=16.550275ms duration_ms=16 level=info msg="compact blocks" count=3 mint=500 maxt=3000 ulid=01J8T5EDDXKHP50RMEHWX93R9R sources="[01J8T5ECYG7NQ1WP0YEMKCTHDJ 01J8T5ED116MSNSHM0MNS5BDXG 01J8T5ED052TDJ5S0YE0PYDNAQ]" duration=16.[88](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:89)4177ms shard=1_of_1 level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5EDDXKHP50RMEHWX93R9R] block_count=3 blocks=01J8T5ECYG7NQ1WP0YEMKCTHDJ,01J8T5ED116MSNSHM0MNS5BDXG,01J8T5ED052TDJ5S0YE0PYDNAQ duration=17.06768ms duration_ms=17 level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem3379[89](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:90)990/001/124@8084862141041806226-merge--0-3000/01J8T5EDDXY6828S4XDFWTY562/chunks/000001 dst=01J8T5EDDXY6828S4XDFWTY562/chunks/000001 bucket=inmem level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem3379899[90](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:91)/001/124@8084862141041806226-merge--0-3000/01J8T5EDDXY6828S4XDFWTY562/index dst=01J8T5EDDXY6828S4XDFWTY562/index bucket=inmem level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01J8T5EDDXY6828S4XDFWTY562 duration=922.015µs duration_ms=0 external_labels="{e1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=2.472576ms duration_ms=2 level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5ED90Z9KBN5RAQEP224BW level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5ED90Z9KBN5RAQEP224BW level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5ED5WAQ8T39B15MVE4ETC level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5ED5WAQ8T39B15MVE4ETC level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=23.922347ms duration_ms=23 block_count=2 level=debug groupKey=124@1425157[91](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:92)50387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem337989990/001/124@14251579150387240703-merge--0-3000/01J8T5EDDXKHP50RMEHWX93R9R/chunks/000001 dst=01J8T5EDDXKHP50RMEHWX93R9R/chunks/000001 bucket=inmem level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem337989990/001/124@14251579150387240703-merge--0-3000/01J8T5EDDXKHP50RMEHWX[93](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:94)R9R/index dst=01J8T5EDDXKHP50RMEHWX93R9R/index bucket=inmem level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01J8T5EDDXKHP50RMEHWX93R9R duration=1.138873ms duration_ms=1 external_labels="{e1=\"1/weird\"}" level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=3.798717ms duration_ms=3 level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5ECYG7NQ1WP0YEMKCTHDJ level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5ECYG7NQ1WP0YEMKCTHDJ level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5ED116MSNSHM0MNS5BDXG level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5ED116MSNSHM0MNS5BDXG level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5ED052TDJ5S0YE0PYDNAQ level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5ED052TDJ5S0YE0PYDNAQ level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=25.601966ms duration_ms=25 block_count=3 level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01J8T5EDF1KYS[94](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:95)2TTASCV1FRD (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01J8T5EDGCZASV72FV0TGT69B7 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01J8T5EDH812MQXQDTEVD2CWRK (min time: 1970-01-01 00:00:01 +0000 UTC, max time: 1970-01-01 00:00:02 +0000 UTC)" level=debug msg="grouper found a compactable blocks group" groupKey=124@8084862141041806226-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01J8T5EDMPYQZC531TM8T06THV (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01J8T5EDREFPMD4AW5NNGB0BNM (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=2 blocks=01J8T5EDREFPMD4AW5NNGB0BNM,01J8T5EDMPYQZC531TM8T06THV level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" block_count=3 blocks=01J8T5EDF1KYS942TTASCV1FRD,01J8T5EDH812MQXQDTEVD2CWRK,01J8T5EDGCZASV72FV0TGT69B7 level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EDH812MQXQDTEVD2CWRK/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EDREFPMD4AW5NNGB0BNM/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EDMPYQZC531TM8T06THV/meta.json level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=2 blocks=01J8T5EDREFPMD4AW5NNGB0BNM,01J8T5EDMPYQZC531TM8T06THV duration=1.28458ms duration_ms=1 level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EDF1KYS942TTASCV1FRD/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01J8T5EDGCZASV72FV0TGT69B7/meta.json level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" block_count=3 blocks=01J8T5EDF1KYS942TTASCV1FRD,01J8T5EDH812MQXQDTEVD2CWRK,01J8T5EDGCZASV72FV0TGT69B7 duration=2.24481ms duration_ms=2 level=info msg="compact blocks" count=2 mint=0 maxt=3000 ulid=01J8T5EDTMKJWR50TKH2844CKR sources="[01J8T5EDREFPMD4AW5NNGB0BNM 01J8T5EDMPYQZC531TM8T06THV]" duration=15.701763ms shard=1_of_1 level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5EDTMKJWR50TKH2844CKR] block_count=2 blocks=01J8T5EDREFPMD4AW5NNGB0BNM,01J8T5EDMPYQZC531TM8T06THV duration=15.857684ms duration_ms=15 level=info msg="compact blocks" count=3 mint=500 maxt=3000 ulid=01J8T5EDTQ1TFMP4YTHCDT897T sources="[01J8T5EDF1KYS942TTASCV1FRD 01J8T5EDH812MQXQDTEVD2CWRK 01J8T5EDGCZASV72FV0TGT69B7]" duration=17.4904[96](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:97)ms shard=1_of_1 level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new_block_count=1 new_blocks=[01J8T5EDTQ1TFMP4YTHCDT8[97](https://github.com/grafana/mimir/actions/runs/11074144077/job/30772935005?pr=9454#step:8:98)T] block_count=3 blocks=01J8T5EDF1KYS942TTASCV1FRD,01J8T5EDH812MQXQDTEVD2CWRK,01J8T5EDGCZASV72FV0TGT69B7 duration=17.687004ms duration_ms=17 level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem709123543/002/124@8084862141041806226-merge--0-3000/01J8T5EDTMKJWR50TKH2844CKR/chunks/000001 dst=01J8T5EDTMKJWR50TKH2844CKR/chunks/000001 bucket="fs: /tmp/TestGroupCompactE2Efilesystem709123543/001" level=debug groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem709123543/002/124@8084862141041806226-merge--0-3000/01J8T5EDTMKJWR50TKH2844CKR/index dst=01J8T5EDTMKJWR50TKH2844CKR/index bucket="fs: /tmp/TestGroupCompactE2Efilesystem709123543/001" level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01J8T5EDTMKJWR50TKH2844CKR duration=1.477214ms duration_ms=1 external_labels="{e1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=3.842629ms duration_ms=3 level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EDREFPMD4AW5NNGB0BNM level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EDREFPMD4AW5NNGB0BNM level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EDMPYQZC531TM8T06THV level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EDMPYQZC531TM8T06THV level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem709123543/002/124@14251579150387240703-merge--0-3000/01J8T5EDTQ1TFMP4YTHCDT897T/chunks/000001 dst=01J8T5EDTQ1TFMP4YTHCDT897T/chunks/000001 bucket="fs: /tmp/TestGroupCompactE2Efilesystem709123543/001" level=info groupKey=124@8084862141041806226-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=23.390494ms duration_ms=23 block_count=2 level=debug groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem709123543/002/124@14251579150387240703-merge--0-3000/01J8T5EDTQ1TFMP4YTHCDT897T/index dst=01J8T5EDTQ1TFMP4YTHCDT897T/index bucket="fs: /tmp/TestGroupCompactE2Efilesystem709123543/001" level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01J8T5EDTQ1TFMP4YTHCDT897T duration=1.189045ms duration_ms=1 external_labels="{e1=\"1/weird\"}" level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=3.390891ms duration_ms=3 level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EDF1KYS942TTASCV1FRD level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EDF1KYS942TTASCV1FRD level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EDH812MQXQDTEVD2CWRK level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EDH812MQXQDTEVD2CWRK level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01J8T5EDGCZASV72FV0TGT69B7 level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01J8T5EDGCZASV72FV0TGT69B7 level=info groupKey=124@14251579150387240703-merge--0-3000 job_type=merge minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=25.674194ms duration_ms=25 block_count=3 level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" ```