anyscale/aviary docker image has cuda 11.8 which is incompatibility with GKE nvidia daemonset

ray-project / ray-llm

RayLLM - LLMs on Ray

https://aviary.anyscale.com

Apache License 2.0

1.22k stars 89 forks source link

anyscale/aviary docker image has cuda 11.8 which is incompatibility with GKE nvidia daemonset #50

Open NikitaSemenovAiforia opened 1 year ago

NikitaSemenovAiforia commented 1 year ago

If I apply kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded-latest.yaml then GKE will install the latest Nvidia drivers which is 525.125.06 in my case. But 525 drivers don't support 11.8 CUDA: docs

Ideally I would like to specify cuda version in the image tag.