rapidsai / dask-cuda

Utilities for Dask and CUDA interactions
https://docs.rapids.ai/api/dask-cuda/stable/
Apache License 2.0
292 stars 93 forks source link

Enabling cuDF spilling should also enable it on the client. #1391

Closed ayushdg closed 1 month ago

ayushdg commented 1 month ago

Currently specifying --enable-cudf-spill or enable_cudf_spill=True enables cuDF spilling on the workers, but not the client which leads to errors like cannot create <class 'cudf.core.buffer.spillable_buffer.SpillableBufferOwner'> without a global spill manager

Since spilling is an option that needs to be constantly set on both the client and workers, enabling this when starting up the workers should also set the same option on the client to avoid both environments being out of sync.

cc: @VibhuJawa

ayushdg commented 1 month ago

Actually looks like #1369 covered this for 24.10. It's only 24.8 that doesn't have this behavior.