revans2
commented
2 weeks ago @LIN-Yu-Ting Generally we treat GPU OutOfMemory errors as bugs that need to be fixed. There are a few cases where an algorithm cannot be split up into smaller pieces and we cannot fix the issue, but most of the time running out of memory is something that we can work around with spilling to either host memory or disk.
If you want to file three separate bugs, one for each of the stack traces, I am happy to try and fix them. If you don't want to just let me know and I'll file something myself.
In the short term you have a few choices to try and mitigate the problem. The goal would be to try and reduce the memory pressure on the GPU. There are several different config settings you can try to help reduce the memory pressure on the GPU.
You could try and set spark.sql.files.maxPartitionBytes smaller. This should reduce the amount of compressed data each task gets, and would reduce the total number of rows. This does not always work because it cannot split up a single parquet row group.
You could also try to reduce the number of concurrent tasks on the GPU by setting spark.rapids.sql.concurrentGpuTasks to 1. By default we run with 2 tasks on the GPU to try and keep the GPU as busy as possible, but it does increase the amount of data on the GPU.
You could also try and set the spark.rapids.sql.batchSizeBytes config to be smaller. By default we target each task to process about 1 GiB of data, and use up to 3 more GiB of scratch space. But this is a soft limit and we might use more if the algorithm needs it. If we set the target smaller, then it is likely that the overall memory pressure will be less.
Please note that all of these are also likely to reduce the performance of your query. Also I am just guessing based on the stack traces. The first and third stack traces look to be running out of memory when trying to read data in from a file.
The second stack trace looks like you are doing a round robin partitioning as a part of your job and the sort that is implicit in that operation ran out of memory. That one is harder because I don't know what task it is associated with, or what. Sort tends to be fairly well behaved with spilling, but if you can help us with a repro case that would let us try and figure out exactly what is happening.
LIN-Yu-Ting
mattahrens
We are using Spark Rapids + Spark Thrift Server to serve SQL request on Spark 3.3.0 and Rapids 23.10 and we have a Delta Table partitioned by a column, runName.
We executed a SQL query
SELECT DISTINCT runName FROM tableon the mentioned table with which we got the following error messages saying that GPU is out of memory.However, this error does not pop out if we executed
SELECT DISTINCT other columns FROM table.In addition, this error happens if we are using Azure NCas64_T4_v3 server with 64 CPU cores and 4 x T4 GPUs. However, error does not happens if we use 2 x NCas8_T4_v3 cluster with each 8 CPU cores and 1 x T4 GPUs.
It seems quite intuitively that it is caused by large size of partition files (around 100MB each). However, this error does not show if we distinct other columns, which makes our assumption not that reasonable. Moreover, as we are distincting a partitionedBy column, normally, Spark does not need to load parquets to figure out which are the values of runName. Instead, it only need to read directory name to get the value. This should not make GPU out of memory.
Any ideas ?