zhouyuan
commented
1 year ago @liucheng-potato thanks for reporting, have you got a chance to check which stage is problematic on Q14?
based on your configurations, the performance could be improved on below on both shuffle write and read:
- using samller shuffle partitions, e.g., 1x or 2x core number. As gluten is using hash based shuffle, using smaller shuffle partition would help to improve the performance
e.g., in your configuration, --conf spark.default.parallelism=1000 seems too big for scale factor 100.
- I assume AQE is enabled in your tests, there's a new fix on AQE batch fetch which should improve the performance on shuffle reading:
--conf spark.gluten.sql.columnar.shuffle.writeEOS=false
Please check if this could help to improve the performance and report back if there are further issues
thanks, -yuan
liucheng-potato
rshkv
Backend
VL (Velox)
Bug description
We did a TPC-H benchmark and found that the time consumption of some SQL increased: q5, q7, q8, q9, q12, q14, q15, q16, especially q14. Did I make a mistake somewhere?
gluten: build from branch main data-size：100 G spark-submit \ --class com.test.RunBenchmark \ --deploy-mode cluster \ --driver-memory 10g \ --executor-cores 4 \ --executor-memory 6G \ --num-executors 20 \ --conf spark.network.timeout=600s \ --conf spark.gluten.loadLibFromJar=true \ --conf spark.plugins=io.glutenproject.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.gluten.sql.columnar.shuffle.codec=lz4 \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.dynamicAllocation.enabled=false \ --conf spark.default.parallelism=1000 \ --conf spark.kubernetes.container.image=spark-imagesxxx \ /path/gluten-1.0-SNAPSHOT-jar-with-dependencies.jar
Spark version
Spark-3.2.x
Spark configurations
spark-submit \ --class com.test.RunBenchmark \ --deploy-mode cluster \ --driver-memory 10g \ --executor-cores 4 \ --executor-memory 6G \ --num-executors 20 \ --conf spark.network.timeout=600s \ --conf spark.gluten.loadLibFromJar=true \ --conf spark.plugins=io.glutenproject.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=2g \ --conf spark.gluten.sql.columnar.shuffle.codec=lz4 \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.dynamicAllocation.enabled=false \ --conf spark.default.parallelism=1000 \ --conf spark.kubernetes.container.image=spark-imagesxxx \ /path/gluten-1.0-SNAPSHOT-jar-with-dependencies.jar
System information
Velox System Info v0.0.2 Commit: 172820c771c02ea3da08ec496d33ad1f416ce183 CMake Version: 3.16.3 System: Linux-4.14.238-125.422.amzn1.x86_64 Arch: x86_64 C++ Compiler: /usr/bin/c++ C++ Compiler Version: 9.4.0 C Compiler: /usr/bin/cc C Compiler Version: 9.4.0 CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt
Relevant logs
No response