thanos-io / promql-engine

Multi-threaded PromQL engine implementation based on the Volcano paper.
Apache License 2.0
141 stars 54 forks source link

Optimize queries on histogram statistics #453

Closed fpetkovski closed 4 months ago

fpetkovski commented 4 months ago

This commit optimizes queries on native histograms when we only need to get the histogram count or sum value.

The approach uses an optimizer which can set a flag in the vector selector to hint to scanners that only histogram stats are needed for the query. When scanners detect this flag, they would use histogram decoders which return histogram objects with populated values for sum, count and the counter reset hint, completely dropping buckets and spans.

Benchmarks

benchstat benchmarks/main.out benchmarks/new.out                                                                                                      Filips-MBP-2.fritz.box: Thu May 16 14:03:09 2024

name                                                   old time/op    new time/op    delta
NativeHistograms/selector-11                             82.1ms ± 1%    83.0ms ± 1%   +1.02%  (p=0.032 n=5+5)
NativeHistograms/sum-11                                   127ms ± 1%     128ms ± 3%     ~     (p=0.222 n=5+5)
NativeHistograms/rate-11                                  112ms ± 2%     116ms ± 8%     ~     (p=0.095 n=5+5)
NativeHistograms/sum_rate-11                              148ms ± 1%     149ms ± 2%     ~     (p=0.548 n=5+5)
NativeHistograms/histogram_sum-11                         304ms ± 2%     293ms ± 1%   -3.73%  (p=0.016 n=5+4)
NativeHistograms/histogram_count-11                       306ms ± 2%     295ms ± 2%   -3.57%  (p=0.008 n=5+5)
NativeHistograms/histogram_count_with_sum_and_rate-11     149ms ± 1%      71ms ± 2%  -52.53%  (p=0.016 n=4+5)
NativeHistograms/histogram_quantile-11                    135ms ± 2%     136ms ± 4%     ~     (p=1.000 n=5+5)
NativeHistograms/histogram_scalar_binop-11                208ms ± 1%     216ms ± 6%   +4.07%  (p=0.008 n=5+5)

name                                                   old alloc/op   new alloc/op   delta
NativeHistograms/selector-11                              435MB ± 0%     435MB ± 0%     ~     (p=0.310 n=5+5)
NativeHistograms/sum-11                                   417MB ± 0%     417MB ± 0%     ~     (p=0.310 n=5+5)
NativeHistograms/rate-11                                  389MB ± 0%     389MB ± 0%     ~     (p=0.548 n=5+5)
NativeHistograms/sum_rate-11                              370MB ± 0%     370MB ± 0%     ~     (p=0.222 n=5+5)
NativeHistograms/histogram_sum-11                         437MB ± 0%     346MB ± 0%  -20.74%  (p=0.016 n=4+5)
NativeHistograms/histogram_count-11                       437MB ± 0%     346MB ± 0%  -20.74%  (p=0.008 n=5+5)
NativeHistograms/histogram_count_with_sum_and_rate-11     370MB ± 0%     226MB ± 0%  -39.03%  (p=0.016 n=4+5)
NativeHistograms/histogram_quantile-11                    432MB ± 0%     432MB ± 0%     ~     (p=0.190 n=5+4)
NativeHistograms/histogram_scalar_binop-11                610MB ± 0%     610MB ± 0%     ~     (p=0.841 n=5+5)

name                                                   old allocs/op  new allocs/op  delta
NativeHistograms/selector-11                              5.20M ± 0%     5.20M ± 0%     ~     (p=0.151 n=5+5)
NativeHistograms/sum-11                                   5.19M ± 0%     5.19M ± 0%     ~     (p=0.278 n=5+5)
NativeHistograms/rate-11                                  7.56M ± 0%     7.56M ± 0%     ~     (p=0.730 n=5+5)
NativeHistograms/sum_rate-11                              7.55M ± 0%     7.55M ± 0%     ~     (p=0.595 n=5+5)
NativeHistograms/histogram_sum-11                         5.21M ± 0%     2.36M ± 0%  -54.68%  (p=0.008 n=5+5)
NativeHistograms/histogram_count-11                       5.21M ± 0%     2.36M ± 0%  -54.68%  (p=0.008 n=5+5)
NativeHistograms/histogram_count_with_sum_and_rate-11     7.55M ± 0%     1.66M ± 0%  -78.09%  (p=0.016 n=4+5)
NativeHistograms/histogram_quantile-11                    5.23M ± 0%     5.23M ± 0%     ~     (p=0.286 n=5+4)
NativeHistograms/histogram_scalar_binop-11                8.85M ± 0%     8.85M ± 0%     ~     (p=1.000 n=5+5)