Closed skirui-source closed 7 months ago
Update:
For both single and multi-node clusters, I am able to successfully pip install (and import) Rapids and other libraries via init script with the 14.2 LTS ML Runtime (GPU, Scala 2.12, Spark 3.5.0).
However, with the same ML Runtime, i experience issues launching a multi-node Dask cluster with dask-databricks
, i.e
dask databricks run
: the cluster successfully launches and connects to dask scheduler and workers but fails to submit tasks to the client : RuntimeError: CuPy failed to load libnvrtc.so.12: OSError: libnvrtc.so.12: cannot open shared object file: No such file or directory
dask databricks run --cuda
completely fails to launch the cluster. The init script logs show : OSError: libnvidia-ml.so.1: cannot open shared object file: No such file or directory
and pynvml.nvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found
@jacobtomlinson , did you want the init scripts stored in public S3 bucket (i filed ops ticket) or should we leave as is?
Also, Ready for Review
Let's open a separate ticket for putting a copy in S3 to simplify things (would you mind opening that?). Then we can get this merged.
i have already filed issue with ops team to request a copy in S3 bucket. I think we can go ahead and merge this unless you have any additional feedback?
Fixes: https://github.com/rapidsai/deployment/issues/299
Right now when you spin up a node you get CUDA 12.2 with the NVIDIA Drivers on Databricks. However, the Databricks container we are instructing to users has CUDA Toolkit 11.8 so you need to install
cudf-cu11
.After testing, I was able to successfully pip install RAPIDS via init script using the ML Runtimes. This PR will update the docs, to avoid using the
tensorflow/pytorch
docker runtime containers altogether.Successfully worked for the following Databricks ML Runtimes, (installed Driver Version: 535.54.03 && CUDA Version: 12.2)
Failed: