mbana
commented
1 month ago This fixed the startup issue for me:
$ cat /etc/nvidia-container-runtime/config.toml
# We inject all NVIDIA GPUs using the nvidia-container-runtime.
# This requires `accept-nvidia-visible-devices-as-volume-mounts = true` be set
# in `/etc/nvidia-container-runtime/config.toml`
accept-nvidia-visible-devices-as-volume-mounts = true
...
$ sudo systemctl restart docker && sudo systemctl restart containerdHowever when I attempt to run a GPU workload I get the following error:
$ ./demo/clusters/kind/create-cluster.sh
$ ./demo/clusters/kind/install-plugin.sh
$ ```yaml
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
restartPolicy: Never
containers:
- name: cuda-container
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
EOF
$ kubectl logs gpu-pod
Failed to allocate device vector A (error code CUDA driver version is insufficient for CUDA runtime version)!
[Vector addition of 50000 elements]
1. Quick Debug Information
2. Issue or feature description
The
./demo/clusters/kind/create-cluster.shseems to fail:3. Information to attach (optional if deemed irrelevant)
Common error checking:
nvidia-smi -aon your host/etc/docker/daemon.json)sudo journalctl -r -u kubelet)Additional information that might help better understand your environment and reproduce the bug:
docker versionuname -admesgdpkg -l '*nvidia*'orrpm -qa '*nvidia*'I am not sure what the script is trying to do but when I exec into the worker, it is reporting the correct GPU information: