Currently specifying --enable-cudf-spill or enable_cudf_spill=True enables cuDF spilling on the workers, but not the client which leads to errors like cannot create <class 'cudf.core.buffer.spillable_buffer.SpillableBufferOwner'> without a global spill manager
Since spilling is an option that needs to be constantly set on both the client and workers, enabling this when starting up the workers should also set the same option on the client to avoid both environments being out of sync.
Currently specifying
--enable-cudf-spill
orenable_cudf_spill=True
enables cuDF spilling on the workers, but not the client which leads to errors likecannot create <class 'cudf.core.buffer.spillable_buffer.SpillableBufferOwner'> without a global spill manager
Since spilling is an option that needs to be constantly set on both the client and workers, enabling this when starting up the workers should also set the same option on the client to avoid both environments being out of sync.
cc: @VibhuJawa