infiniflow / infinity

The AI-native database built for LLM applications, providing incredibly fast hybrid search of dense vector, sparse vector, tensor (multi-vector), and full-text
https://infiniflow.org
Apache License 2.0
2.47k stars 268 forks source link

[Question]: If it's possible to design a new dsl for optimizer rules, so that we can add new optimizer by just add new rule files #1286

Open xiaoma20082008 opened 4 months ago

xiaoma20082008 commented 4 months ago

Describe your problem

As we know that almost all optimizers are based on rules, just like too functions: match and replace, means that if we match the expression, then we can replace the old expression to a new expression.

so can we

  1. design a new dsl only for optimizer rule
  2. write a tiny compiler and transform the dsl to rule(eg: c++ code)
  3. then we don't need write new optimizer rules, just write the dsl code

here is an example: if we has a Match And Replace function for Filter:

Match(Filter(AND, lhs = $1, rhs = true))  

Replace(Filter(AND, lhs = $1, rhs = true)) => $1

Then the sql expression

a = 'tom' AND b = true

can rewrite to the following directly:

a = 'tom'
JinHai-CN commented 4 months ago

Using a DSL to define RBO rules is a very cool approach. But, considering the road map of Infinity, we won't start the development of this feature, recently.