Improve performance of TPC-H q16

andygrove commented 2 weeks ago

What is the problem the feature request solves?

Comet is currently slower than Spark for TPC-H q16.

This query does not run fully native yet.

Issues

BuildRight with LeftAnti is not supported (https://github.com/apache/datafusion-comet/issues/457)
BroadcastExchange is not supported (confusing error, needs improving)
BroadcastHashJoin disabled because not all child plans are native
SortMergeJoin disabled because not all child plans are native
Disabling coalesce partitions (so that we can enable Comet shuffle) has quite a negative impact on performance - https://github.com/apache/datafusion-comet/issues/387

Describe the potential solution

No response

Additional context

No response

andygrove commented 2 weeks ago

As an experiment, I removed our check where we fall back to Spark for BuildRight with LeftAnti so that the query runs natively (possibly with incorrect results) and performance increased significantly and matched Spark (but was not faster). I no longer saw the "BroadcastExchange is not supported" error.

andygrove commented 2 weeks ago

At the end of the query we are performing a RowToColumnar to move to Comet just for a query stage that performs a sort and then perform a ColumnarToRow. This likely explains why we currently make the query slower than Spark.

apache / datafusion-comet