thanos-io / promql-engine

Multi-threaded PromQL engine implementation based on the Volcano paper.
Apache License 2.0
141 stars 54 forks source link

Adding filter to vector scanner #391

Closed harsh-ps-2003 closed 7 months ago

harsh-ps-2003 commented 9 months ago

Fixes #338

Changes Adding a filtering function to the vector scanner which would take in a step vector and produce a new step vector with filtered samples to optimize the query execution process.

Verification Added a test case sum(kube_pod_info == 1)

Tests are passing locally using make test

harsh-ps-2003 commented 9 months ago

@fpetkovski the make tests are passing locally :

❯ make test
>> running unit tests (without cache)
?       github.com/thanos-io/promql-engine/api  [no test files]
ok      github.com/thanos-io/promql-engine/engine       589.694s
?       github.com/thanos-io/promql-engine/execution    [no test files]
?       github.com/thanos-io/promql-engine/execution/aggregate  [no test files]
?       github.com/thanos-io/promql-engine/execution/binary     [no test files]
?       github.com/thanos-io/promql-engine/execution/exchange   [no test files]
?       github.com/thanos-io/promql-engine/execution/function   [no test files]
?       github.com/thanos-io/promql-engine/execution/model      [no test files]
?       github.com/thanos-io/promql-engine/execution/noop       [no test files]
?       github.com/thanos-io/promql-engine/execution/remote     [no test files]
?       github.com/thanos-io/promql-engine/execution/parse      [no test files]
?       github.com/thanos-io/promql-engine/execution/step_invariant     [no test files]
?       github.com/thanos-io/promql-engine/execution/scan       [no test files]
?       github.com/thanos-io/promql-engine/execution/unary      [no test files]
?       github.com/thanos-io/promql-engine/execution/warnings   [no test files]
?       github.com/thanos-io/promql-engine/extlabels    [no test files]
ok      github.com/thanos-io/promql-engine/execution/storage    (cached)
?       github.com/thanos-io/promql-engine/query        [no test files]
?       github.com/thanos-io/promql-engine/worker       [no test files]
ok      github.com/thanos-io/promql-engine/logicalplan  (cached)
ok      github.com/thanos-io/promql-engine/ringbuffer   (cached)

but make lint is failing! Please point out if I have messed up something!

MichaHoffmann commented 9 months ago

I think we need to add a "FilterFunc" to logicalplan.VectorSelector and then pass it down from execute.go into the constructor of the scan.VectorScan operator. The logical filter should be populated by an optimizer that also removes the binary expression (which is the ultimate benefit ~ no binary operator on top of the vector scan anymore). I think that would be the way to implement the feature.

Edit: that is probably a lot of information! If you have any questions or i can help somehow, let me know!

harsh-ps-2003 commented 9 months ago

@MichaHoffmann thanks for the guidance! I am kinda noob to PromQL engine, I am trying to solve this just for getting some exposure!