search

rapidsai / dask-cuda

Utilities for Dask and CUDA interactions
https://docs.rapids.ai/api/dask-cuda/stable/
Apache License 2.0
292 stars 93 forks source link

Add arguments to enable cuDF spilling and set statistics #1362

Closed pentschev closed 4 months ago

pentschev commented 4 months ago

Add arguments to enable cuDF spilling and set statistics in dask cuda worker/LocalCUDACluster. This is implemented as a Dask plugin, and does not require users anymore to rely on client.run to do that.

Closes #1280

pentschev commented 4 months ago

Warning added in https://github.com/rapidsai/dask-cuda/pull/1362/commits/d7ffa0a4b5592b1a2fa4b05876da00e45ff7d778 . @madsbk do you think we should just raise a warning or raise a warning and disable JIT-Unspill regardless of the user enabling both together, i.e., --enable-cudf-spill --enable-jit-unspill would emit a warning saying JIT-Unspill has been disabled and only cuDF spill will be active?

Side thought: We have a lot of options at this point. Do you think it makes sense to deprecate anything?

Probably @madsbk also has a better sense of that as well, do you have thoughts on this?

madsbk commented 4 months ago

I think we should raise a warning only.

Side thought: We have a lot of options at this point. Do you think it makes sense to deprecate anything?

I think we should wait a bit with deprecating until we get a good solution from the NO-OOM work.

pentschev commented 4 months ago

Thanks @rjzamora and @madsbk for reviewing!

pentschev commented 4 months ago

/merge