apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.15k stars 416 forks source link

Using substrait-core to convert SparkPlan and Expression #531

Open baibaichen opened 1 year ago

baibaichen commented 1 year ago

Catalyst Expression needs to inherit ExpressionTransformer to be converted to Substrait porto expression. This was the main problem of the current design.

With substrait-java, we can have an better way to do it.

FelixYBW commented 1 year ago

Some background here:

With this modification, we tried to generate expression tree and substrait plan from a logic plan tree directly, instead of the whole stage code gen style.