mrocklin
commented
4 years ago In either case, deploying Dask+RAPIDS typically means using dask-cuda-workers instead of the dask-worker. dask-cuda-workers are opinionated in configuring GPUs for dask (1 worker/1 thread per GPU).
I've raised an issue internally with a design for supporting CUDAWorker (and other things along the way) I think that it will take about a day, but it might not be the first thing we start working on on Monday.
I'm curious if we can, short term, ask for tasks that have a single CPU core and if that would sufficiently mimic dask-cuda-worker for short-term use.
I'm not sure what is the best for users and coiled here. Some up front knowledge may be needed to be exposed to the user about the GPUs, specifically the CUDA VERSION on the machine: 10.1/10.2/11.
Yeah, I'm not sure what is best here yet. Let's think on this.
The RAPIDS nightly docker images is quite nice as it combines RAPIDS/DASK/XGBoost/etc and necessary dependencies. However, the conda env is store /opt/conda/rapids and I don't think coiled exposes a custom path/conda option for docker images at the moments
I don't suppose that the python command on the path points to the right executable, does it?
quasiben
jacobtomlinson
necaris
selshowk
jrbourbeau
When deploying to GPUs I'd like the option of using the RAPIDS stack. This can be deployed either with a set of conda packages (with optional nightlies):
or official rapids images on dockerhub. In either case, deploying Dask+RAPIDS typically means using dask-cuda-workers instead of the dask-worker.
dask-cuda-workersare opinionated in configuring GPUs for dask (1 worker/1 thread per GPU).I'm not sure what is the best for users and coiled here. Some up front knowledge may be needed to be exposed to the user about the GPUs, specifically the
CUDA VERSIONon the machine:10.1/10.2/11. We'd also want the option of using a different worker (maybe this is generalized to any executable ? though that's probably a big can of worms).The RAPIDS nightly docker images is quite nice as it combines RAPIDS/DASK/XGBoost/etc and necessary dependencies. However, the conda env is store
/opt/conda/rapidsand I don't think coiled exposes a custom path/conda option for docker images at the momentsThe next thing after RAPIDS is working nicely is getting BlazingSQL which can also be installed with conda/docker:
From BSQL install docs.