Join support

[viirya]

This is an umbrella ticket for adding Join support to Comet. In Spark, there are basically three types of Join operators: BroadcastJoin, HashJoin, SortMergeJoin. In DataFusion, two Join operators are supported: HashJoin, SortMergeJoin (experimental).

We are going to delegate Spark Join operators to correspond DataFusion Join operators. We will also go to improve DataFusion Join operators if needed.

SortMergeJoin

• [x] Relax join keys constraint from Column to any physical expression for physical join operators https://github.com/apache/arrow-datafusion/pull/8991
• [x] Support join filter for SortMergeJoin https://github.com/apache/arrow-datafusion/pull/9080
• [ ] Add spilling in SortMergeJoin https://github.com/apache/arrow-datafusion/issues/9359
• [x] Support SortMergeJoin operator - #177 ...

HashJoin

• [x] Support HashJoin operator - #193

BroadcastJoin

• [x] Add CometBroadcastExchangeExec support broadcasting the result of Comet native operator - https://github.com/apache/arrow-datafusion-comet/issues/81
• [x] Support BroadcastHashJoinExec #202
• [x] Enable Comet broadcast by default #212

Other Join operators (Cross Join)

• [ ] Support CartesianProductExec - #199
• [ ] Support BroadcastNestedLoopJoinExec - #198

[mwiewior]

Hey @viirya - it's really awesome ! Do you happen to know when approx these BHJ and shuffle components will be open sourced?

[viirya]

Hi @mwiewior

I will open source shuffle components soon. For join support, it is in development now. We already have working code but get some test failure on Spark tests. Once these test failures are resolved, we will open source it.

[viirya]

For shuffle, we just remove it from the initial open source code. So the code is ready, we only need to add it back to Comet.

[mwiewior]

Great news ! Thx !