Open andygrove opened 4 months ago
BroadcastExchange
should be supported, I think. We have CometBroadcastExchange
.
We don't need to support AQEShuffleRead
. It is a shuffle reader wrapper in Spark. It calls wrapped shuffle's execute
or executeColumnar
depending on it is columnar or not.
We don't need to support Execute CreateViewCommand
too. It is a command exec operator.
Also CommandResult
, which is only used to hold data from a command. CommandResult
and Execute CreateViewCommand
are not query execution operators.
Also
CommandResult
, which is only used to hold data from a command.CommandResult
andExecute CreateViewCommand
are not query execution operators.
Thanks. I saw those from the CREATE VIEW
in q15 but I see from the Spark UI that the SELECT part of this query is already fully native. I have removed those from the list.
BroadcastExchange
should be supported, I think. We haveCometBroadcastExchange
.
BroadcastExchange is not supported
is the information that Comet provides for q8. I think part of this epic will be making these messages more informative.
For Sort merge join with a join condition
, I added the support to DataFusion for a while but we've not incorporated the feature in Comet yet. I opened #398 to track it and I will work on it once #250 is merged and #248 is done.
BroadcastExchange is not supported
is the information that Comet provides for q8. I think part of this epic will be making these messages more informative.
I will take a look at q8 and see why it is not enabled there.
I will take a look at q8 and see why it is not enabled there.
The error BroadcastExchange is not supported
really means BroadcastExchange is not supported because the child operators are not supported
Please disable spark.comet.exec.broadcast.enabled
which should not be used in normal query: https://github.com/apache/datafusion-comet/issues/408#issuecomment-2104818958
What is the problem the feature request solves?
We currently fall back to Spark for parts of TPC-H. This epic is to track work to support these features in the context of the TPC-H queries so that we can start to get some benchmark results.
Status
Updated 5/30/2024 based on PR with BuildRight support and SMJ enabled.
Original configs used (ignore this now)
Used Comet as of commit hash
bc6b2cda3efd2b0c6c48f932ce19da46456bcbd5
.Configs used:
Describe the potential solution
No response
Additional context
No response