Support sort merge join with a join condition

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator

https://datafusion.apache.org/comet

Apache License 2.0

823 stars 163 forks source link

Support sort merge join with a join condition #398

Open viirya opened 6 months ago

viirya commented 6 months ago

What is the problem the feature request solves?

Currently SMJ with join condition is not supported by Comet and falls back to Spark. The feature was added into DataFusion but we've not incorporated it yet. We should do it to add the support.

Describe the potential solution

No response

Additional context

No response

comphead commented 5 months ago

@viirya can I take it as I spent some time with SMJ?

andygrove commented 2 months ago

Re-opening this since we discovered that there are some correctness and performance issues and currently have this feature disabled by default

andygrove commented 2 weeks ago

There is an existing test SortMergeJoin with join filter that we need to enable as part of closing this issue. It was ignored in https://github.com/apache/datafusion-comet/pull/1057

comphead commented 2 weeks ago

Thanks @andygrove I'll take if from now

comphead commented 3 days ago

Waiting for DataFusion 44.0.0