Closed ShangWeiKuo closed 1 year ago
@ShangWeiKuo looks like the node already have NVIDIA drivers installed. Please disable the driver container from the GPU operator by passing --set driver.enabled=false
. This is automatically handled from the v1.11.x onwards. Suggest to use latest version of the GPU operator to handle this case and disable driver container automatically.
The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.
1. Quick Debug Checklist
i2c_core
andipmi_msghandler
loaded on the nodes?kubectl describe clusterpolicies --all-namespaces
)1. Issue or feature description
When conduct bash setup.sh install, the process is stopped in the stage "[Waiting for the Cluster to become available]".
The logs for the pod that is in status "Init:CrashLoopBackOff" are the messages below:
2. Steps to reproduce the issue
First, download resource using NGC CLI (ngc registry resource download-version "nvidia/tao/tao-getting-started:4.0.0") and edit the hosts file.
After configuring the hosts information, run bash setup.sh install.
3. Information to attach (optional if deemed irrelevant)
[x] kubernetes pods status:
kubectl get pods --all-namespaces
[ ] kubernetes daemonset status:
kubectl get ds --all-namespaces
[ ] If a pod/ds is in an error state or pending state
kubectl describe pod -n NAMESPACE POD_NAME
[x] If a pod/ds is in an error state or pending state
kubectl logs -n NAMESPACE POD_NAME
[ ] Output of running a container on the GPU machine:
docker run -it alpine echo foo
[ ] Docker configuration file:
cat /etc/docker/daemon.json
[ ] Docker runtime configuration:
docker info | grep runtime
[ ] NVIDIA shared directory:
ls -la /run/nvidia
[ ] NVIDIA packages directory:
ls -la /usr/local/nvidia/toolkit
[ ] NVIDIA driver directory:
ls -la /run/nvidia/driver
[ ] kubelet logs
journalctl -u kubelet > kubelet.logs