tgravescs
commented
3 years ago the issue here is that you have more then just Rapids using the GPU, you see Xorg and gnome-shell also using it (normal terminal graphics related things). So basically at spark.rapids.memory.gpu.allocFraction=0.9, you are over allocated the GPU memory and you run out. Rapids tries to use 90% but other processes are using 10+% already.
When you change it to 0.8 then Rapids will try to use less which leaves room for your normal graphics related processes and you don't run out of memory.
revans2
abellina
wjxiz1992
jlowe
sameerz
Describe the bug I was running an LHA query on my PC in standalone mode. When set
spark.rapids.memory.gpu.allocFraction=0.9, there's error ofcudaErrorMemoryAllocation: out of memory:But set
spark.rapids.memory.gpu.allocFraction=0.8, the error went away.Steps/Code to reproduce bug It's an LHA query so I only describe the main steps here:
Expected behavior A query should run successfully with a higher memory allocation fraction if it is able to pass with a lower number.
Environment details (please complete the following information)
Set path for source table data
T2_PATH=/home/allxu/tmp/q4/t2 T3_PATH=/home/allxu/tmp/q4/t3 T4_PATH=/home/allxu/tmp/q4/t4
Set path for your query output
Here your query result(dataset) will be saved to the 'output' folder under your current working directory
OUT=output
Set your Spark master IP:Port
MASTER=*****
Remove existed query result data first
Or you cannot run this query successfully
rm -rf $OUT
$SPARK_HOME/bin/spark-submit --master $MASTER \ --driver-memory ${DRIVE_MEMORY}G \ --executor-memory ${EXECUTOR_MEMORY}G \ --executor-cores $EXECUTOR_CORES \ --num-executors $NUM_EXECUTOR \ --conf spark.executor.extraClassPath=${SPARK_CUDF_JAR}:${SPARK_RAPIDS_PLUGIN_JAR} \ --conf spark.driver.extraClassPath=${SPARK_CUDF_JAR}:${SPARK_RAPIDS_PLUGIN_JAR} \ --conf spark.driver.extraJavaOptions=-Dai.rapids.cudf.nvtx.enabled=true \ --conf spark.executor.extraJavaOptions='-Dai.rapids.cudf.prefer-pinned=true -Dai.rapids.cudf.nvtx.enabled=true' \ --conf spark.task.cpus=1 \ --conf spark.rapids.sql.explain=ALL \ --conf spark.locality.wait=0 \ --conf spark.yarn.maxAppAttempts=1 \ --conf spark.sql.shuffle.partitions=24 \ --conf spark.sql.files.maxPartitionBytes=128m \ --conf spark.sql.warehouse.dir=$OUT \ --conf spark.plugins=com.nvidia.spark.SQLPlugin \ --conf spark.rapids.sql.incompatibleOps.enabled=true \ --conf spark.rapids.sql.variableFloatAgg.enabled=true \ --conf spark.rapids.sql.concurrentGpuTasks=2 \ --conf spark.sql.adaptive.enabled=true \ --conf spark.rapids.memory.gpu.pool=ARENA \ --conf spark.rapids.memory.pinnedPool.size=${PINNEDMEMORY}G \ --conf spark.rapids.memory.gpu.allocFraction=0.9 \ --conf spark.rapids.sql.castStringToInteger.enabled=true \ --conf spark.rapids.memory.pinnedPool.size=5g \ --conf spark.task.resource.gpu.amount=0.08 \ --conf spark.executor.resource.gpu.amount=1 \ --conf spark.rapids.shuffle.ucx.enabled=false \ --conf spark.rapids.sql.parquet.debug.dumpPrefix=$PWD/dump/dump \ --conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \ --files ${SPARK_HOME}/examples/src/main/scripts/getGpusResources.sh \ --jars ${SPARK_CUDF_JAR},${SPARK_RAPIDS_PLUGIN_JAR} \ ../main.py \ --sqlFile sql.txt \ --format parquet \ --appName "GPU Q4" \ --tableFile sql4_table_2::$T2_PATH --tableFile sql4_table_3::$T3_PATH --tableFile sql4_table_4::$T4_PATH