Open adesh-rao opened 1 week ago
It's a known issue. The issue is Gluten/Velox broadcast raw table instead of hash table. So each task needs to build its own hash table from raw data. The solution is to build hashtable from driver or one task then broadcast. But the problem now is Velox doesn't support it yet.
we plan to fix it in next year.
Add the impact TPC-DS queries: q24a, q24b, q11, q65, q69, q85 as a record.
Backend
VL (Velox)
Bug description
Setup: Running TPCDS benchmark on spark with gluten.
With increasing broadcast hash join threshold, the performance of TPCDS benchmark degrades (config:
spark.sql.autoBroadcastJoinThreshold
).We have jobs with BHJ threshold set to 200MB in vanilla spark. Such jobs are not showing any gains with Spark+Gluten.
Spark version
Spark-3.5.x
Spark configurations
spark.sql.autoBroadcastJoinThreshold
System information
No response
Relevant logs
No response