jlowe
commented
2 years ago Another potential solution here, which may be far preferable, is to have cudf throw different kinds of exceptions based on the mode of failure, or at least change the message text of the error being thrown based on the situation.
At the C++ level, RMM throws a specific type of exception when the pool runs out of memory which hopefully can be distinguished from the exception libcudf throws for a CUDA error resulting from out of memory, the latter being an error that necessarily is a driver-level error implying it must be an issue with insufficient reserved memory. We could then throw different Java exceptions or change the message text based on whether it was an RMM out of memory or a CUDA out of memory.
Is your feature request related to a problem? Please describe. When executing a regular expression kernel, there can be a significant amount of thread stack space needed to launch the kernel. Unfortunately the default plugin setup allocates almost all of the GPU memory for the RMM pool, and without using the CUDA async allocator to back the RMM pool, the driver has only the remaining memory outside of the RMM pool to use for launching kernels. If there is insufficient memory outside of the pool to satisfy the kernel launch, and OOM error will be thrown. This error can be very confusing to users since they may think they need to increase the size of the RMM pool rather than decrease it to solve this problem.
Describe the solution you'd like Ideally switching to the async allocator would be best, but if we cannot do that in a timely manner then it would be nice to have the regex code catch OOM errors when launching the kernels and wrap the OOM error with a new error using a message indicating that potentially the error is caused by a kernel launch failure due to insufficient reserved memory rather than insufficient RMM pool memory. (Logging the free memory available in the RMM in the message would be a nice bonus to help the user determine which is more likely.)