When starting the GPU docker image, we will get the following error:
Error: Internal error message: Spark error: Driver down cause: java.io.IOException: Cannot run program "virtualenv" (in directory "/local_disk0/.ephemeral_nfs/cluster_libraries"): error=2, No such file or directory
This PR is fixing the image issue. In this PR,
Unified the virtualenv setup in ubuntu/gpu/cuda-11.8/venv/Dockerfile to be the same as ubuntu/python/Dockerfile
Add new user for cluster library installation ubuntu/gpu/cuda-11.8/base/Dockerfile. This is following the same way in ubuntu/minimal/Dockerfile
Add python-lsp
Manually upload the docker to docker hub and create the cluster:
When starting the GPU docker image, we will get the following error:
This PR is fixing the image issue. In this PR,
ubuntu/gpu/cuda-11.8/venv/Dockerfileto be the same asubuntu/python/Dockerfileubuntu/gpu/cuda-11.8/base/Dockerfile. This is following the same way inubuntu/minimal/DockerfileManually upload the docker to docker hub and create the cluster: