1. Quick Debug Information

OS/Version(e.g. RHEL8.6, Ubuntu22.04): Ubuntu 22.04.4 LTS
Kernel Version: 5.15.0-101-generic
Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): containerd://1.7.15-k3s1
K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): Rancher, v1.29.5+k3s1
GPU Operator Version: v24.3.0

2. Issue or feature description

The nvidia-operator-validator-.* pod does not start correctly and enters a Init:CrashLoopBackOff state with the nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown error message. This issue persists until the problematic pods (nvidia-operator-validator-.*, gpu-feature-discovery-.*, nvidia-dcgm-exporter-.*, nvidia-device-plugin-daemonset-.*) are deleted and recreated.

3. Steps to reproduce the issue

Deploy the Helm chart with the following values:


driver:
# By default, the Operator deploys NVIDIA drivers as a container on the system.
# Set this value to false when using the Operator on systems with pre-installed drivers.
enabled: true

# Version of the NVIDIA datacenter driver supported by the Operator.
version: 550.90.07

upgradePolicy:
# Global switch for automatic upgrade feature.
# If set to false all other options are ignored
autoUpgrade: true
# How many nodes can be upgraded in parallel.
# 0 means no limit, all nodes will be upgraded in parallel.
maxParallelUpgrades: 1

migManager:

The MIG manager watches for changes to the MIG geometry and applies reconfiguration as needed.

By default, the MIG manager only runs on nodes with GPUs that support MIG (for e.g. A100).

enabled: false

Controls the strategy to be used with MIG on supported NVIDIA GPUs.

Options are either mixed or single.

strategy: single

toolkit:

By default, the Operator deploys the NVIDIA Container Toolkit (nvidia-docker2 stack) as a

container on the system. Set this value to false when using the Operator on systems with

pre-installed NVIDIA runtimes.

enabled: true

Version of the NVIDIA Container Toolkit supported by the Operator.

version: v1.16.1-ubuntu20.04

Environment variables for configuring the NVIDIA Container Toolkit.

NOTE: https://www.virtualthoughts.co.uk/2022/11/21/installing-using-the-nvidia-gpu-operator-in-k3s-with-rancher

env:

name: CONTAINERD_CONFIG value: /var/lib/rancher/k3s/agent/etc/containerd/config.toml

name: CONTAINERD_SOCKET value: /run/k3s/containerd/containerd.sock

2. See that the `nvidia-operator-validator-.*` is crashing:
```shell
$ kubectl get pods -n gpu-operator -o wide --field-selector spec.nodeName=mu
NAME                                               READY   STATUS                  RESTARTS      AGE     IP            NODE   NOMINATED NODE   READINESS GATES
gpu-feature-discovery-cqrwd                        0/1     Init:0/1                0             4m16s   10.42.28.14   mu     <none>           <none>
gpu-operator-7448b89c59-mrffg                      1/1     Running                 0             5m19s   10.42.28.5    mu     <none>           <none>
gpu-operator-node-feature-discovery-worker-cbbwt   1/1     Running                 0             7m11s   10.42.28.4    mu     <none>           <none>
nvidia-container-toolkit-daemonset-mtb2g           1/1     Running                 0             4m16s   10.42.28.16   mu     <none>           <none>
nvidia-dcgm-exporter-62lmv                         0/1     Init:0/1                0             4m16s   10.42.28.15   mu     <none>           <none>
nvidia-device-plugin-daemonset-4p2dj               0/1     Init:0/1                0             4m16s   10.42.28.17   mu     <none>           <none>
nvidia-driver-daemonset-vx5hj                      1/1     Running                 0             4m56s   10.42.28.8    mu     <none>           <none>
nvidia-operator-validator-bbc8r                    0/1     Init:CrashLoopBackOff   5 (39s ago)   4m16s   10.42.28.13   mu     <none>           <none>

$ kubectl describe pod -n gpu-operator nvidia-operator-validator-bbc8r 
Name:                 nvidia-operator-validator-bbc8r
Namespace:            gpu-operator
Priority:             2000001000
Priority Class Name:  system-node-critical
Runtime Class Name:   nvidia
Service Account:      nvidia-operator-validator
Node:                 mu/64.247.196.26
Start Time:           Mon, 05 Aug 2024 15:48:13 +0400
Labels:               app=nvidia-operator-validator
                  app.kubernetes.io/managed-by=gpu-operator
                  app.kubernetes.io/part-of=gpu-operator
                  controller-revision-hash=78d6f8dfb5
                  helm.sh/chart=gpu-operator-v24.3.0
                  pod-template-generation=2
Annotations:          <none>
Status:               Pending
IP:                   10.42.28.13
IPs:
IP:           10.42.28.13
Controlled By:  DaemonSet/nvidia-operator-validator
Init Containers:
driver-validation:
Container ID:  containerd://f294b0623945435cc94cb84f3e4473e6469818849b40ccf23c51c5b2309973d4
Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
Image ID:      nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:2edc1d4ed555830e70010c82558936198f5faa86fc29ecf5698219145102cfcc
Port:          <none>
Host Port:     <none>
Command:
  sh
  -c
Args:
  nvidia-validator
State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Mon, 05 Aug 2024 15:48:14 +0400
  Finished:     Mon, 05 Aug 2024 15:49:54 +0400
Ready:          True
Restart Count:  0
Environment:
  WITH_WAIT:  true
  COMPONENT:  driver
Mounts:
  /host from host-root (ro)
  /host-dev-char from host-dev-char (rw)
  /run/nvidia/driver from driver-install-path (rw)
  /run/nvidia/validations from run-nvidia-validations (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x2ts8 (ro)
toolkit-validation:
Container ID:  containerd://a71fc759f7a80e34f675d47965501dd9f4a9061da07eb9ae1d2ac53c15010c5f
Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
Image ID:      nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:2edc1d4ed555830e70010c82558936198f5faa86fc29ecf5698219145102cfcc
Port:          <none>
Host Port:     <none>
Command:
  sh
  -c
Args:
  nvidia-validator
State:       Waiting
  Reason:    CrashLoopBackOff
Last State:  Terminated
  Reason:    StartError
  Message:   failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
  Exit Code:    128
  Started:      Thu, 01 Jan 1970 04:00:00 +0400
  Finished:     Mon, 05 Aug 2024 15:51:50 +0400
Ready:          False
Restart Count:  5
Environment:
  NVIDIA_VISIBLE_DEVICES:  all
  WITH_WAIT:               false
  COMPONENT:               toolkit
Mounts:
  /run/nvidia/validations from run-nvidia-validations (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x2ts8 (ro)
cuda-validation:
Container ID:
Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
Image ID:
Port:          <none>
Host Port:     <none>
Command:
  sh
  -c
Args:
  nvidia-validator
State:          Waiting
  Reason:       PodInitializing
Ready:          False
Restart Count:  0
Environment:
  WITH_WAIT:                    false
  COMPONENT:                    cuda
  NODE_NAME:                     (v1:spec.nodeName)
  OPERATOR_NAMESPACE:           gpu-operator (v1:metadata.namespace)
  VALIDATOR_IMAGE:              nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
  VALIDATOR_IMAGE_PULL_POLICY:  IfNotPresent
  VALIDATOR_RUNTIME_CLASS:      nvidia
Mounts:
  /run/nvidia/validations from run-nvidia-validations (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x2ts8 (ro)
plugin-validation:
Container ID:
Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
Image ID:
Port:          <none>
Host Port:     <none>
Command:
  sh
  -c
Args:
  nvidia-validator
State:          Waiting
  Reason:       PodInitializing
Ready:          False
Restart Count:  0
Environment:
  COMPONENT:                    plugin
  WITH_WAIT:                    false
  WITH_WORKLOAD:                false
  MIG_STRATEGY:                 single
  NODE_NAME:                     (v1:spec.nodeName)
  OPERATOR_NAMESPACE:           gpu-operator (v1:metadata.namespace)
  VALIDATOR_IMAGE:              nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
  VALIDATOR_IMAGE_PULL_POLICY:  IfNotPresent
  VALIDATOR_RUNTIME_CLASS:      nvidia
Mounts:
  /run/nvidia/validations from run-nvidia-validations (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x2ts8 (ro)
Containers:
nvidia-operator-validator:
Container ID:
Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0
Image ID:
Port:          <none>
Host Port:     <none>
Command:
  sh
  -c
Args:
  echo all validations are successful; sleep infinity
State:          Waiting
  Reason:       PodInitializing
Ready:          False
Restart Count:  0
Environment:    <none>
Mounts:
  /run/nvidia/validations from run-nvidia-validations (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x2ts8 (ro)
Conditions:
Type                        Status
PodReadyToStartContainers   True
Initialized                 False
Ready                       False
ContainersReady             False
PodScheduled                True
Volumes:
run-nvidia-validations:
Type:          HostPath (bare host directory volume)
Path:          /run/nvidia/validations
HostPathType:  DirectoryOrCreate
driver-install-path:
Type:          HostPath (bare host directory volume)
Path:          /run/nvidia/driver
HostPathType:
host-root:
Type:          HostPath (bare host directory volume)
Path:          /
HostPathType:
host-dev-char:
Type:          HostPath (bare host directory volume)
Path:          /dev/char
HostPathType:
kube-api-access-x2ts8:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  3607
ConfigMapName:           kube-root-ca.crt
ConfigMapOptional:       <nil>
DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              nvidia.com/gpu.deploy.operator-validator=true
Tolerations:                 env:NoExecute op=Exists
                         gpu:NoExecute op=Exists
                         node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                         node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                         node.kubernetes.io/not-ready:NoExecute op=Exists
                         node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                         node.kubernetes.io/unreachable:NoExecute op=Exists
                         node.kubernetes.io/unschedulable:NoSchedule op=Exists
                         nvidia.com/gpu:NoSchedule op=Exists
                         service:NoExecute op=Exists
Events:
Type     Reason     Age                    From               Message
----     ------     ----                   ----               -------
Normal   Scheduled  4m39s                  default-scheduler  Successfully assigned gpu-operator/nvidia-operator-validator-bbc8r to mu
Normal   Pulled     4m39s                  kubelet            Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0" already present on machine
Normal   Created    4m39s                  kubelet            Created container driver-validation
Normal   Started    4m39s                  kubelet            Started container driver-validation
Normal   Pulled     2m49s (x2 over 2m51s)  kubelet            Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0" already present on machine
Normal   Created    2m49s (x2 over 2m51s)  kubelet            Created container toolkit-validation
Warning  Failed     2m49s (x2 over 2m50s)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
Warning  BackOff  2m48s                kubelet  Back-off restarting failed container toolkit-validation in pod nvidia-operator-validator-bbc8r_gpu-operator(af7b6068-fa4f-44c6-9237-6f75a9dc78a3)
Normal   Pulled   63s (x4 over 2m28s)  kubelet  Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.3.0" already present on machine
Normal   Created  63s (x4 over 2m28s)  kubelet  Created container toolkit-validation
Warning  Failed   63s (x4 over 2m28s)  kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
Warning  BackOff  2s (x12 over 2m27s)  kubelet  Back-off restarting failed container toolkit-validation in pod nvidia-operator-validator-bbc8r_gpu-operator(af7b6068-fa4f-44c6-9237-6f75a9dc78a3)

Reschedule the problematic pods and see that's all ok now:

$ kubectl delete pod -n gpu-operator gpu-feature-discovery-cqrwd nvidia-dcgm-exporter-62lmv nvidia-device-plugin-daemonset-4p2dj nvidia-operator-validator-ftvxw
pod "gpu-feature-discovery-cqrwd" deleted
pod "nvidia-dcgm-exporter-62lmv" deleted
pod "nvidia-device-plugin-daemonset-4p2dj" deleted
pod "nvidia-operator-validator-ftvxw" deleted

$ kubectl get pods -n gpu-operator -o wide --field-selector spec.nodeName=mu
NAME                                               READY   STATUS      RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
gpu-feature-discovery-qkzwb                        1/1     Running     0          13m   10.42.28.20   mu     <none>           <none>
gpu-operator-7448b89c59-mrffg                      1/1     Running     0          25m   10.42.28.5    mu     <none>           <none>
gpu-operator-node-feature-discovery-worker-cbbwt   1/1     Running     0          27m   10.42.28.4    mu     <none>           <none>
nvidia-container-toolkit-daemonset-mtb2g           1/1     Running     0          24m   10.42.28.16   mu     <none>           <none>
nvidia-cuda-validator-f99jp                        0/1     Completed   0          13m   10.42.28.24   mu     <none>           <none>
nvidia-dcgm-exporter-hdzhr                         1/1     Running     0          13m   10.42.28.21   mu     <none>           <none>
nvidia-device-plugin-daemonset-d2r8z               1/1     Running     0          13m   10.42.28.22   mu     <none>           <none>
nvidia-driver-daemonset-vx5hj                      1/1     Running     0          25m   10.42.28.8    mu     <none>           <none>
nvidia-operator-validator-p6sd7                    1/1     Running     0          13m   10.42.28.23   mu     <none>           <none>

NVIDIA / gpu-operator

nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown #898