[VL][CH] Gluten columnar rules + RAS integration

[zhztheplayer]

Description

To achieve one of the original purpose of ras: to make Gluten's rule simpler, and to make the rule list cleaner, some kind of migrations on our current rule code will be suggested.

The following is what I could assume:

RAS=on vs RAS=off

I'd raise a new approach around current entrance of RAS. Will be inclined to having two individual source code files (or folders) for RAS=on and RAS=off, to store the rule list of the each way. By doing this, it would be easier for developers to know what and how RAS actually does.

Worth noting that the migration would not remove rules from RAS=off as of now. We only doing code refactors to make RAS=on cover more columnar optimizations in RAS=off.

The gap of CH backend

CH backend doesn't currently support RAS=on. We should eliminate the following gaps as far as I know to make them compatible:

• In CH backend there is some other plan nodes that are coupled with others, for example AFAIK the broadcast exchange doesn't work with C2R / R2C.

There is a PR https://github.com/apache/incubator-gluten/pull/5101 to verify CH backend and TPC-H, we can make it pass asap.

Rules to migrate

Ideally we'll migrate most of columnar rules that are not 100% heuristic. The following is a initial list and since I may not be the writer of these rules, so please correct me if I get anything wrong.

1. FallbackOnANSIMode Will keep this in heuristics since it's a global switch.
2. FallbackMultiCodegens
   • choice 1: In RAS, tune cost model to slightly increase Velox's BHJ to larger than Vanilla Spark's BHJ. Then optimizer will fallback consecutive Gluten BHJs when it sees some.
   • choice 2: Keep as heuristic
3. PlanOneRowRelation Will keep this in heuristics since it's a global switch.
4. FallbackEmptySchemaRelation Will keep this in heuristics since it's a global switch.
5. MergeTwoPhasesHashBaseAggregate (CH only)
   • choice 1: Could add a RAS rule to merge aggregates.
   • choice 2: Keep as heuristic
6. Spark rewrite rules: RewriteIn, RewriteMultiChildrenCount, RewriteCollect, RewriteTypedImperativeAggregate, PullOutPreProject, PullOutPostProject: Move them to RAS. RAS would naturally do such "tentative" transformations better than RBO. RAS could enumerate the possible plans then try to find the one that is executable and have lowest cost. For example, if the transformation yields too many of C2Rs/R2Cs, or just an inexecutable plan, optimizer will not choose it.
7. AddTransformHintRule In RAS, merge this main validation rule together with transformation rule. Which means, RAS should validate the plan before it does each tentative transformation.
8. FallbackBloomFilterAggIfNeeded
   • choice 1: In RAS, add an independent physical property to indicate if the bloom filter buffer data is vanilla or Gluten/Velox. Then optimizer would only choose plans that have consistent buffer data format.
   • choice 2: Keep as heuristic
9. ImplementFilter, ImplementAggregate, ImplementExchange, ImplementJoin, ImplementOthers Can be moved to RAS, to do transformation.
10. RemoveNativeWriteFilesSortAndProject The "remove sort and project" part can be perfectly done by RAS by property requirements. The other Empty2Null looks like some kind of validation that can be moved into RAS's transformation rule. Confirmation required.
11. RewriteTransformer RAS could directly use the plugged rules, ideally.
12. EnsureLocalSortRequirements It's RAS's natural job. Though we just need to add ordering property to property model.
13. CollapseProjectExecTransformer RAS can use that rule.
14. genExtendedColumnarTransformRules (Velox only) Currently only flushable agg rule. Can move to RAS's agg transformation rule.
15. GlutenConfig.getConf.extendedColumnarTransformRules Keep it at the end of heuristic rule list to allow user defined rewrites. Will not add to RAS.
16. ExpandFallbackPolicy (whole stage fallback) This can be done by RAS by setting more reasonable costs to C2R / R2C. Then optimizer would consider the cost of a plan with a lot of C2Rs / R2Cs higher then will not choose it.
17. InsertTransitions, TransformPostOverrides, InsertColumnarToColumnarTransitions, RemoveGlutenTableCacheColumnarToRow RAS doesn't need these rules. C2Rs / R2Cs can be added via property enforcement.
18. RemoveTopmostColumnarToRow Since this one is for compatible with Spark, will keep it in heuristics.
19. genExtendedColumnarPostRules Probably keep in heuristics.
20. ColumnarCollapseTransformStages Probably keep in heuristics.
21. extendedColumnarRules Keep in heuristics.
22. GlutenFallbackReporter Keep in heuristics.
23. RemoveTransformHintRule Keep in heuristics.

Removal of RAS=off

This can be one of the final goal if all goes well as expected. But we will not take this action in short term.

Note

This topic would not only be related to code quality and maintenance. After we have a cleaner rule list and plan node definitions that are fully compatible to RAS, then we can decide whether to move on from RAS's current responsibility (fallback processing) to more advanced optimizations that could be powered by RAS.

[zhouyuan]

to make Gluten's rule simpler, and to make the rule list cleaner Are you suggesting to remove all the rules and replace them with the CBO framework?

thanks, -yuan