search

thanos-io / promql-engine

Multi-threaded PromQL engine implementation based on the Volcano paper.
Apache License 2.0
133 stars 52 forks source link

Allow injecting custom scanners #407

Closed fpetkovski closed 4 months ago

fpetkovski commented 4 months ago

The current implementation of the engine is coupled to the Prometheus storage model. The input is a SeriesSet which gets iterated on one series at a time. In cases where data is pulled from a different storage, it first has to be transformed into the Prometheus model before it can be queried. This transformation can be resource intensive if the data is not sorted by series and time.

This commit allows extending the engine with custom scanners which take in the logical node and return an execution operator. With this extension point we should get decoupling from Prometheus and can use the engine more efficiently with different storages.

yeya24 commented 4 months ago

It is an interesting idea. I wonder what storage you are using rather than Prometheus TSDB.

MichaHoffmann commented 4 months ago

It is an interesting idea. I wonder what storage you are using rather than Prometheus TSDB.

It can enable us down the road to get away from Select limitations in thanos.

fpetkovski commented 4 months ago

It is an interesting idea. I wonder what storage you are using rather than Prometheus TSDB.

We are starting to explore storing data in parquet files, possibly by even using https://github.com/polarsignals/frostdb. Using ingesting into something like ClickHouse is also an interesting direction.