Catalyst Expression needs to inherit ExpressionTransformer to be converted to Substrait porto expression. This was the main problem of the current design.
With substrait-java, we can have an better way to do it.
When we started Gluten projects, substrait-java is still not started. So we use the java API from substrait's protobuf to create the substrait plan. Later substrait-java is created and bring in some high level API we can use to create substrait plan from java code.
In current design we reuse Spark's whole stage code gen design to transform Spark's logic plan to substrait plan. Each operator has an transformer operator which includes "doTransform" interface. The WholeStageCodegenTransformer calls the doTransform interface of each operator to generate the final substrait plan.
Expression is very similar design
With this modification, we tried to generate expression tree and substrait plan from a logic plan tree directly, instead of the whole stage code gen style.
Catalyst
Expression
needs to inheritExpressionTransformer
to be converted to Substrait porto expression. This was the main problem of the current design.With substrait-java, we can have an better way to do it.