search

grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.18k stars 538 forks source link

Race observed in `TestMultiDimensionalQueueAlgorithmSlowConsumerEffects` test. #9253

Open pstibrany opened 2 months ago

pstibrany commented 2 months ago

Race observed in TestMultiDimensionalQueueAlgorithmSlowConsumerEffects test.

==================
WARNING: DATA RACE
Write at 0x00c0003ba0b8 by goroutine 2879:
  github.com/grafana/mimir/pkg/scheduler/queue.(*tenantQuerierAssignments).setup()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_querier_assignment.go:423 +0xc4
  github.com/grafana/mimir/pkg/scheduler/queue.(*MultiQueuingAlgorithmTreeQueue).Dequeue()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue.go:85 +0x115
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:485 +0xc9b
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Previous read at 0x00c0003ba0b8 by goroutine 2880:
  github.com/grafana/mimir/pkg/scheduler/queue.(*queueBroker).dequeueRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/tenant_queues.go:152 +0x146
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).trySendNextRequestForQuerier()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:422 +0x76
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).dispatcherLoop()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:340 +0x364
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting.gowrap1()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:289 +0x33

Goroutine 2879 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:422 +0x1e04
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 2880 (running) created at:
  github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting()
      /__w/mimir/mimir/pkg/scheduler/queue/queue.go:289 +0x8d
  github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1()
      /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:442 +0x486
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44
==================
dimitarvdimitrov commented 2 months ago

another one

Details

``` ================== WARNING: DATA RACE Write at 0x00c000539ad8 by goroutine 4103: github.com/grafana/mimir/pkg/scheduler/queue.(*tenantQuerierAssignments).setup() /__w/mimir/mimir/pkg/scheduler/queue/tenant_querier_assignment.go:237 +0xc4 github.com/grafana/mimir/pkg/scheduler/queue.(*MultiQueuingAlgorithmTreeQueue).Dequeue() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue.go:85 +0x115 github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:485 +0xc9b testing.tRunner() /usr/local/go/src/testing/testing.go:1690 +0x226 testing.(*T).Run.gowrap1() /usr/local/go/src/testing/testing.go:1743 +0x44 Previous read at 0x00c000539ad8 by goroutine 4104: github.com/grafana/mimir/pkg/scheduler/queue.(*queueBroker).dequeueRequestForQuerier() /__w/mimir/mimir/pkg/scheduler/queue/tenant_queues.go:156 +0x153 github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).trySendNextRequestForQuerier() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:392 +0x[76](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:77) github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).dispatcherLoop() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:310 +0x364 github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting.gowrap1() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x33 Goroutine 4103 (running) created at: testing.(*T).Run() /usr/local/go/src/testing/testing.go:1743 +0x825 github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:422 +0x1d84 testing.tRunner() /usr/local/go/src/testing/testing.go:1690 +0x226 testing.(*T).Run.gowrap1() /usr/local/go/src/testing/testing.go:1743 +0x44 Goroutine 4104 (running) created at: github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x8d github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:442 +0x486 testing.tRunner() /usr/local/go/src/testing/testing.go:1690 +0x226 testing.(*T).Run.gowrap1() /usr/local/go/src/testing/testing.go:1743 +0x44 ================== --- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects (15.47s) --- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects/tree:_worker-queue_prioritization_->_tenant-querier_tree,_2_tenants,_first_with_25pct_slow_queries,_second_with_75pct_slow_queries (0.67s) multi_queuing_algorithm_tree_queue_benchmark_test.go:4[78](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:79): tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0290 stddev: 0.01 store-gateway: mean: 0.2595 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:4[79](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:80): tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0329 stddev: 0.01 tenant-1: mean: 0.0169 stddev: 0.00] testing.go:1399: race detected during execution of test multi_queuing_algorithm_tree_queue_benchmark_test.go:492: Results by query component: multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1128 stddev: 0.02 store-gateway: mean: 0.0282 stddev: 0.04] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1191 stddev: 0.02 store-gateway: mean: 0.0270 stddev: 0.04] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0283 stddev: 0.01 store-gateway: mean: 0.0323 stddev: 0.03] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2079 stddev: 0.05 store-gateway: mean: 0.0963 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2359 stddev: 0.07 store-gateway: mean: 0.1067 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.0247 stddev: 0.01 store-gateway: mean: 0.1124 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.3968 stddev: 0.17 store-gateway: mean: 0.2492 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.3963 stddev: 0.18 store-gateway: mean: 0.2500 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.0200 stddev: 0.00 store-gateway: mean: 0.2642 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2599 stddev: 0.16 store-gateway: mean: 0.4161 stddev: 0.25] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2574 stddev: 0.17 store-gateway: mean: 0.4107 stddev: 0.26] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.0139 stddev: 0.00 store-gateway: mean: 0.4102 stddev: 0.25] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.1076 stddev: 0.08 store-gateway: mean: 0.4847 stddev: 0.30] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0822 stddev: 0.07 store-gateway: mean: 0.5012 stddev: 0.31] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0103 stddev: 0.00 store-gateway: mean: 0.4971 stddev: 0.31] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1291 stddev: 0.04 store-gateway: mean: 0.2398 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1437 stddev: 0.04 store-gateway: mean: 0.2519 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.0200 stddev: 0.00 store-gateway: mean: 0.2717 stddev: 0.18] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2099 stddev: 0.08 store-gateway: mean: 0.2400 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2298 stddev: 0.09 store-gateway: mean: 0.2440 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0290 stddev: 0.01 store-gateway: mean: 0.2595 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3658 stddev: 0.17 store-gateway: mean: 0.2202 stddev: 0.14] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3548 stddev: 0.15 store-gateway: mean: 0.2205 stddev: 0.14] multi_queuing_algorithm_tree_queue_benchmark_test.go:494: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.0257 stddev: 0.00 store-gateway: mean: 0.2188 stddev: 0.14] multi_queuing_algorithm_tree_queue_benchmark_test.go:497: Results for ingester-only queries by tenant ID: multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1128 stddev: 0.02] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1191 stddev: 0.02] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0283 stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2079 stddev: 0.05] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2359 stddev: 0.07] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.0247 stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3968 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3963 stddev: 0.18] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0200 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2599 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2574 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0139 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1076 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0822 stddev: 0.07] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0103 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.13[80](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:81) stddev: 0.02 tenant-1: mean: 0.0652 stddev: 0.05] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1456 stddev: 0.04 tenant-1: mean: 0.1286 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0209 stddev: 0.00 tenant-1: mean: 0.0119 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2139 stddev: 0.07 tenant-1: mean: 0.1968 stddev: 0.10] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2294 stddev: 0.07 tenant-1: mean: 0.2309 stddev: 0.13] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0329 stddev: 0.01 tenant-1: mean: 0.0169 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3527 stddev: 0.17 tenant-1: mean: 0.3802 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.32[88](https://github.com/grafana/mimir/actions/runs/10964505773/job/30448297525#step:8:89) stddev: 0.16 tenant-1: mean: 0.3794 stddev: 0.14] multi_queuing_algorithm_tree_queue_benchmark_test.go:499: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0261 stddev: 0.00 tenant-1: mean: 0.0252 stddev: 0.00] FAIL ```

@chencs @francoposa does anything come to mind?

dimitarvdimitrov commented 2 months ago

this failed again, I think the failure is the same - GH action run

Details

``` ================== WARNING: DATA RACE Read at 0x00c0001bc758 by goroutine 2880: github.com/grafana/mimir/pkg/scheduler/queue.(*queueBroker).dequeueRequestForQuerier() /__w/mimir/mimir/pkg/scheduler/queue/queue_broker.go:154 +0x153 github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).trySendNextRequestForQuerier() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:392 +0x[76](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:77) github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).dispatcherLoop() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:310 +0x364 github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting.gowrap1() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x33 Previous write at 0x00c0001bc758 by goroutine 2879: github.com/grafana/mimir/pkg/scheduler/queue.(*tenantQuerierAssignments).setup() /__w/mimir/mimir/pkg/scheduler/queue/tenant_querier_assignment.go:239 +0xc4 github.com/grafana/mimir/pkg/scheduler/queue.(*MultiQueuingAlgorithmTreeQueue).Dequeue() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue.go:90 +0x134 github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:491 +0xd06 testing.tRunner() /usr/local/go/src/testing/testing.go:1690 +0x226 testing.(*T).Run.gowrap1() /usr/local/go/src/testing/testing.go:1743 +0x44 Goroutine 2880 (running) created at: github.com/grafana/mimir/pkg/scheduler/queue.(*RequestQueue).starting() /__w/mimir/mimir/pkg/scheduler/queue/queue.go:259 +0x8d github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects.func1() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:448 +0x4d6 testing.tRunner() /usr/local/go/src/testing/testing.go:1690 +0x226 testing.(*T).Run.gowrap1() /usr/local/go/src/testing/testing.go:1743 +0x44 Goroutine 2879 (running) created at: testing.(*T).Run() /usr/local/go/src/testing/testing.go:1743 +0x825 github.com/grafana/mimir/pkg/scheduler/queue.TestMultiDimensionalQueueAlgorithmSlowConsumerEffects() /__w/mimir/mimir/pkg/scheduler/queue/multi_queuing_algorithm_tree_queue_benchmark_test.go:428 +0x21[77](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:78) testing.tRunner() /usr/local/go/src/testing/testing.go:1690 +0x226 testing.(*T).Run.gowrap1() /usr/local/go/src/testing/testing.go:1743 +0x44 ================== --- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects (16.65s) --- FAIL: TestMultiDimensionalQueueAlgorithmSlowConsumerEffects/tree:_worker-queue_prioritization_->_tenant-querier_tree,_1_tenant,_90pct_slow_queries (1.17s) multi_queuing_algorithm_tree_queue_benchmark_test.go:484: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0182 stddev: 0.01 store-gateway: mean: 0.5084 stddev: 0.30] multi_queuing_algorithm_tree_queue_benchmark_test.go:485: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0182 stddev: 0.01] testing.go:1399: race detected during execution of test multi_queuing_algorithm_tree_queue_benchmark_test.go:498: Results by query component: multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1237 stddev: 0.02 store-gateway: mean: 0.0285 stddev: 0.03] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.1186 stddev: 0.02 store-gateway: mean: 0.0154 stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue: [ingester: mean: 0.0471 stddev: 0.01 store-gateway: mean: 0.0506 stddev: 0.03] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2933 stddev: 0.08 store-gateway: mean: 0.1186 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.2339 stddev: 0.06 store-gateway: mean: 0.1142 stddev: 0.07] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue: [ingester: mean: 0.0398 stddev: 0.01 store-gateway: mean: 0.1497 stddev: 0.09] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.4435 stddev: 0.21 store-gateway: mean: 0.2[80](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:81)6 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.4431 stddev: 0.21 store-gateway: mean: 0.2725 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue: [ingester: mean: 0.0288 stddev: 0.00 store-gateway: mean: 0.2867 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2571 stddev: 0.16 store-gateway: mean: 0.4390 stddev: 0.26] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.2957 stddev: 0.19 store-gateway: mean: 0.4087 stddev: 0.25] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue: [ingester: mean: 0.0205 stddev: 0.00 store-gateway: mean: 0.4313 stddev: 0.26] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0902 stddev: 0.07 store-gateway: mean: 0.5085 stddev: 0.31] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0844 stddev: 0.06 store-gateway: mean: 0.5138 stddev: 0.31] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue: [ingester: mean: 0.0182 stddev: 0.01 store-gateway: mean: 0.5084 stddev: 0.30] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1370 stddev: 0.04 store-gateway: mean: 0.24[81](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:82) stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.1542 stddev: 0.04 store-gateway: mean: 0.2623 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue: [ingester: mean: 0.0388 stddev: 0.01 store-gateway: mean: 0.2857 stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2835 stddev: 0.11 store-gateway: mean: 0.2540 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.2316 stddev: 0.09 store-gateway: mean: 0.2447 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue: [ingester: mean: 0.0264 stddev: 0.01 store-gateway: mean: 0.2565 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3802 stddev: 0.18 store-gateway: mean: 0.2350 stddev: 0.15] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.3842 stddev: 0.18 store-gateway: mean: 0.2300 stddev: 0.14] multi_queuing_algorithm_tree_queue_benchmark_test.go:500: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue: [ingester: mean: 0.0322 stddev: 0.00 store-gateway: mean: 0.2353 stddev: 0.14] multi_queuing_algorithm_tree_queue_benchmark_test.go:503: Results for ingester-only queries by tenant ID: multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1237 stddev: 0.02] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.1186 stddev: 0.02] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 10pct slow queries: seconds in queue:[tenant-0: mean: 0.0471 stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2933 stddev: 0.08] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.2339 stddev: 0.06] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 25pct slow queries: seconds in queue:[tenant-0: mean: 0.0398 stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.4435 stddev: 0.21] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.4431 stddev: 0.21] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0288 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2571 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2957 stddev: 0.19] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0205 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0902 stddev: 0.07] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0844 stddev: 0.06] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 1 tenant, 90pct slow queries: seconds in queue:[tenant-0: mean: 0.01[82](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:83) stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1466 stddev: 0.03 tenant-1: mean: 0.0440 stddev: 0.04] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.1568 stddev: 0.04 tenant-1: mean: 0.1338 stddev: 0.07] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 10pct slow queries, second with 90pct slow queries: seconds in queue:[tenant-0: mean: 0.0404 stddev: 0.00 tenant-1: mean: 0.0239 stddev: 0.01] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.3130 stddev: 0.08 tenant-1: mean: 0.1952 stddev: 0.12] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.2260 stddev: 0.07 tenant-1: mean: 0.2480 stddev: 0.13] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 25pct slow queries, second with 75pct slow queries: seconds in queue:[tenant-0: mean: 0.0280 stddev: 0.00 tenant-1: mean: 0.0219 stddev: 0.00] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: tenant-querier -> query component round-robin tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3606 stddev: 0.18 tenant-1: mean: 0.39[85](https://github.com/grafana/mimir/actions/runs/11083284447/job/30797331436?pr=9461#step:8:86) stddev: 0.17] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: query component round-robin -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.3597 stddev: 0.19 tenant-1: mean: 0.4072 stddev: 0.16] multi_queuing_algorithm_tree_queue_benchmark_test.go:505: tree: worker-queue prioritization -> tenant-querier tree, 2 tenants, first with 50pct slow queries, second with 50pct slow queries: seconds in queue:[tenant-0: mean: 0.0315 stddev: 0.00 tenant-1: mean: 0.0331 stddev: 0.00] FAIL ```