Open jaanphare opened 5 months ago
more context in https://groups.google.com/g/google-dl-platform/c/cYvzQuM5JY0 and https://ray-distributed.slack.com/archives/CN2RGCHRR/p1711487521436029 ... @aslonnie is there anything we can do to make sure the next batch of ray oss images with cuda has the dependency correctly configured?
@anyscalesam , the error seems to be permission denied on docker daemon service access. does not seem to be related to cuda
this is the oss cluster launcher, should route to core team. @jjyao
What happened + What you expected to happen
While the example GPU + docker script works (https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/gcp/example-gpu-docker.yaml), a slight modification does not.
The specific modifications to the script:
project_id
image: "rayproject/ray:latest-py310-cu121"
(from the docker hub: https://hub.docker.com/r/rayproject/ray/tags)This should work I think (it works fine on AWS).
Error message:
Versions / Dependencies
Python 3.10 and CUDA 12.1.
Reproduction script
YAML cluster configuration modified from https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/gcp/example-gpu-docker.yaml -
Issue Severity
High: It blocks me from completing my task.