cdesiniotis
commented
2 years ago Hi @JunyaTaniai is the Network Operator installed and MOFED driver deployed? The GPU driver container waits for MOFED driver to be ready first before proceeding. And setting driver.rdma.useHostMofed=false indicates to the GPU Operator that MOFED driver is being installed via container (and not on the host) -- in this case we wait for the status file /run/mellanox/drivers/.driver-ready to be created before proceeding.
From your logs it appears this file is not being created:
$ sudo cat /var/log/pods/gpu-operator_nvidia-driver-daemonset-k8tvc_d6d1d91a-5e0e-4701-a01d-9a0585c5232e/mofed-validation/0.log
2022-11-22T05:38:21.156056921+00:00 stdout F running command bash with args [-c stat /run/mellanox/drivers/.driver-ready]
2022-11-22T05:38:21.160991578+00:00 stderr F stat: cannot statx '/run/mellanox/drivers/.driver-ready': No such file or directory
2022-11-22T05:38:21.161307447+00:00 stdout F command failed, retrying after 5 seconds
JunyaTaniai
Hi all,
1. Quick Debug Checklist
i2c_coreandipmi_msghandlerloaded on the nodes? No[x] Did you apply the CRD (
kubectl describe clusterpolicies --all-namespaces) Yesdescribe clusterpolicies
``` $ k -n gpu-operator describe clusterpolicies.nvidia.com --all-namespaces Name: cluster-policy Namespace: Labels: app.kubernetes.io/component=gpu-operator app.kubernetes.io/managed-by=Helm Annotations: meta.helm.sh/release-name: gpu-operator meta.helm.sh/release-namespace: gpu-operator API Version: nvidia.com/v1 Kind: ClusterPolicy Metadata: Creation Timestamp: 2022-11-22T02:35:24Z Generation: 1 Managed Fields: API Version: nvidia.com/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:meta.helm.sh/release-name: f:meta.helm.sh/release-namespace: f:labels: .: f:app.kubernetes.io/component: f:app.kubernetes.io/managed-by: f:spec: .: f:daemonsets: .: f:priorityClassName: f:tolerations: f:dcgm: .: f:enabled: f:hostPort: f:image: f:imagePullPolicy: f:repository: f:version: f:dcgmExporter: .: f:enabled: f:env: f:image: f:imagePullPolicy: f:repository: f:serviceMonitor: .: f:additionalLabels: f:enabled: f:honorLabels: f:interval: f:version: f:devicePlugin: .: f:config: .: f:default: f:name: f:enabled: f:env: f:image: f:imagePullPolicy: f:repository: f:version: f:driver: .: f:certConfig: .: f:name: f:enabled: f:image: f:imagePullPolicy: f:kernelModuleConfig: .: f:name: f:licensingConfig: .: f:configMapName: f:nlsEnabled: f:manager: .: f:env: f:image: f:imagePullPolicy: f:repository: f:version: f:rdma: .: f:enabled: f:useHostMofed: f:repoConfig: .: f:configMapName: f:repository: f:rollingUpdate: .: f:maxUnavailable: f:version: f:virtualTopology: .: f:config: f:gfd: .: f:enabled: f:env: f:image: f:imagePullPolicy: f:repository: f:version: f:mig: .: f:strategy: f:migManager: .: f:config: .: f:name: f:enabled: f:env: f:gpuClientsConfig: .: f:name: f:image: f:imagePullPolicy: f:repository: f:version: f:nodeStatusExporter: .: f:enabled: f:image: f:imagePullPolicy: f:repository: f:version: f:operator: .: f:defaultRuntime: f:initContainer: .: f:image: f:imagePullPolicy: f:repository: f:version: f:runtimeClass: f:psp: .: f:enabled: f:sandboxDevicePlugin: .: f:enabled: f:image: f:imagePullPolicy: f:repository: f:version: f:sandboxWorkloads: .: f:defaultWorkload: f:enabled: f:toolkit: .: f:enabled: f:image: f:imagePullPolicy: f:installDir: f:repository: f:version: f:validator: .: f:image: f:imagePullPolicy: f:plugin: .: f:env: f:repository: f:version: f:vfioManager: .: f:driverManager: .: f:env: f:image: f:imagePullPolicy: f:repository: f:version: f:enabled: f:image: f:imagePullPolicy: f:repository: f:version: f:vgpuDeviceManager: .: f:config: .: f:default: f:name: f:enabled: f:image: f:imagePullPolicy: f:repository: f:version: f:vgpuManager: .: f:driverManager: .: f:env: f:image: f:imagePullPolicy: f:repository: f:version: f:enabled: f:image: f:imagePullPolicy: Manager: helm Operation: Update Time: 2022-11-22T02:35:24Z API Version: nvidia.com/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:namespace: f:state: Manager: gpu-operator Operation: Update Subresource: status Time: 2022-11-22T02:35:44Z Resource Version: 23128 UID: 5ff18092-9076-4801-802d-c088cf09cc66 Spec: Daemonsets: Priority Class Name: system-node-critical Tolerations: Effect: NoSchedule Key: nvidia.com/gpu Operator: Exists Dcgm: Enabled: false Host Port: 5555 Image: dcgm Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: 3.0.4-1-ubuntu20.04 Dcgm Exporter: Enabled: true Env: Name: DCGM_EXPORTER_LISTEN Value: :9400 Name: DCGM_EXPORTER_KUBERNETES Value: true Name: DCGM_EXPORTER_COLLECTORS Value: /etc/dcgm-exporter/dcp-metrics-included.csv Image: dcgm-exporter Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/k8s Service Monitor: Additional Labels: Enabled: false Honor Labels: false Interval: 15s Version: 3.0.4-3.0.0-ubuntu20.04 Device Plugin: Config: Default: Name: Enabled: true Env: Name: PASS_DEVICE_SPECS Value: true Name: FAIL_ON_INIT_ERROR Value: true Name: DEVICE_LIST_STRATEGY Value: envvar Name: DEVICE_ID_STRATEGY Value: uuid Name: NVIDIA_VISIBLE_DEVICES Value: all Name: NVIDIA_DRIVER_CAPABILITIES Value: all Image: k8s-device-plugin Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia Version: v0.12.3-ubi8 Driver: Cert Config: Name: Enabled: true Image: driver Image Pull Policy: IfNotPresent Kernel Module Config: Name: Licensing Config: Config Map Name: Nls Enabled: false Manager: Env: Name: ENABLE_AUTO_DRAIN Value: true Name: DRAIN_USE_FORCE Value: false Name: DRAIN_POD_SELECTOR_LABEL Value: Name: DRAIN_TIMEOUT_SECONDS Value: 0s Name: DRAIN_DELETE_EMPTYDIR_DATA Value: false Image: k8s-driver-manager Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: v0.4.2 Rdma: Enabled: true Use Host Mofed: false Repo Config: Config Map Name: Repository: nvcr.io/nvidia Rolling Update: Max Unavailable: 1 Version: 520.61.05 Virtual Topology: Config: Gfd: Enabled: true Env: Name: GFD_SLEEP_INTERVAL Value: 60s Name: GFD_FAIL_ON_INIT_ERROR Value: true Image: gpu-feature-discovery Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia Version: v0.6.2-ubi8 Mig: Strategy: single Mig Manager: Config: Name: Enabled: true Env: Name: WITH_REBOOT Value: false Gpu Clients Config: Name: Image: k8s-mig-manager Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: v0.5.0-ubuntu20.04 Node Status Exporter: Enabled: false Image: gpu-operator-validator Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: v22.9.0 Operator: Default Runtime: docker Init Container: Image: cuda Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia Version: 11.7.1-base-ubi8 Runtime Class: nvidia Psp: Enabled: false Sandbox Device Plugin: Enabled: true Image: kubevirt-gpu-device-plugin Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia Version: v1.2.1 Sandbox Workloads: Default Workload: container Enabled: false Toolkit: Enabled: true Image: container-toolkit Image Pull Policy: IfNotPresent Install Dir: /usr/local/nvidia Repository: nvcr.io/nvidia/k8s Version: v1.11.0-ubuntu20.04 Validator: Image: gpu-operator-validator Image Pull Policy: IfNotPresent Plugin: Env: Name: WITH_WORKLOAD Value: true Repository: nvcr.io/nvidia/cloud-native Version: v22.9.0 Vfio Manager: Driver Manager: Env: Name: ENABLE_AUTO_DRAIN Value: false Image: k8s-driver-manager Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: v0.4.2 Enabled: true Image: cuda Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia Version: 11.7.1-base-ubi8 Vgpu Device Manager: Config: Default: default Name: Enabled: true Image: vgpu-device-manager Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: v0.2.0 Vgpu Manager: Driver Manager: Env: Name: ENABLE_AUTO_DRAIN Value: false Image: k8s-driver-manager Image Pull Policy: IfNotPresent Repository: nvcr.io/nvidia/cloud-native Version: v0.4.2 Enabled: false Image: vgpu-manager Image Pull Policy: IfNotPresent Status: Namespace: gpu-operator State: notReady Events:1. Issue or feature description
After installing GPU-Operator with MIG and RDMA enabled, all but the gpu-operator pod and gpu-operator-node-feature-discovery pod stop at the initialization process.
2. Steps to reproduce the issue
3. Information
[x] kubernetes pods status:
kubectl get pods --all-namespacespods status
``` $ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE calico-apiserver calico-apiserver-884778456-ddcsc 1/1 Running 0 168m calico-apiserver calico-apiserver-884778456-gttx9 1/1 Running 0 168m calico-system calico-kube-controllers-6b57db7fd6-hz8j4 1/1 Running 0 169m calico-system calico-node-bflqc 1/1 Running 0 169m calico-system calico-typha-59885bb5d7-hx2rz 1/1 Running 0 169m gpu-operator gpu-feature-discovery-rwkvl 0/1 Init:0/1 0 37m gpu-operator gpu-operator-5dc6b8989b-p6mbc 1/1 Running 0 37m gpu-operator gpu-operator-node-feature-discovery-master-65c9bd48c4-hqf5q 1/1 Running 0 37m gpu-operator gpu-operator-node-feature-discovery-worker-5p2h5 1/1 Running 0 37m gpu-operator nvidia-container-toolkit-daemonset-k8dl6 0/1 Init:0/1 0 37m gpu-operator nvidia-dcgm-exporter-bhjnb 0/1 Init:0/1 0 37m gpu-operator nvidia-device-plugin-daemonset-jzlbm 0/1 Init:0/1 0 37m gpu-operator nvidia-driver-daemonset-k8tvc 0/2 Init:0/2 0 37m gpu-operator nvidia-operator-validator-hlfp7 0/1 Init:0/4 0 37m kube-system coredns-565d847f94-9dq8f 1/1 Running 0 174m kube-system coredns-565d847f94-s9jd5 1/1 Running 0 174m kube-system etcd-mito26.server.org 1/1 Running 2 175m kube-system kube-apiserver-mito26.server.org 1/1 Running 2 175m kube-system kube-controller-manager-mito26.server.org 1/1 Running 0 174m kube-system kube-proxy-wpf76 1/1 Running 0 174m kube-system kube-scheduler-mito26.server.org 1/1 Running 2 174m tigera-operator tigera-operator-6bb5985474-srnn8 1/1 Running 0 169m ```[x] kubernetes daemonset status:
kubectl get ds --all-namespacesget ds
``` $ kubectl get ds --all-namespaces NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE calico-system calico-node 1 1 1 1 1 kubernetes.io/os=linux 170m calico-system csi-node-driver 0 0 0 0 0 kubernetes.io/os=linux 170m gpu-operator gpu-feature-discovery 1 1 0 1 0 nvidia.com/gpu.deploy.gpu-feature-discovery=true 37m gpu-operator gpu-operator-node-feature-discovery-worker 1 1 1 1 1[x] If a pod/ds is in an error state or pending state
kubectl describe pod -n NAMESPACE POD_NAMEdescribe pod gpu-feature-discovery
``` $ k -n gpu-operator describe po gpu-feature-discovery-rwkvl Name: gpu-feature-discovery-rwkvl Namespace: gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-gpu-feature-discovery Node: mito26.server.org/192.168.200.96 Start Time: Tue, 22 Nov 2022 02:35:44 +0000 Labels: app=gpu-feature-discovery app.kubernetes.io/part-of=nvidia-gpu controller-revision-hash=5d85dbd666 pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: e6ccd423694c07c80a917c696727b0a4febdfb9d8e1d12efde6d1999aa1a1641 cni.projectcalico.org/podIP: 10.244.11.124/32 cni.projectcalico.org/podIPs: 10.244.11.124/32 Status: Pending IP: 10.244.11.124 IPs: IP: 10.244.11.124 Controlled By: DaemonSet/gpu-feature-discovery Init Containers: toolkit-validation: Container ID: cri-o://3f4d69d9b0914823a7e79a5378f60979579cc971410328c4e05c1c92611fcd20 Image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.0 Image ID: nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:6fe4200960b2b49d6dac1c91e596f61dacb6b3dcff878c84eb74c5136fedd5b6 Port:describe pod nvidia-container-toolkit-daemonset
``` $ k -n gpu-operator describe po nvidia-container-toolkit-daemonset-k8dl6 Name: nvidia-container-toolkit-daemonset-k8dl6 Namespace: gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-container-toolkit Node: mito26.server.org/192.168.200.96 Start Time: Tue, 22 Nov 2022 02:35:43 +0000 Labels: app=nvidia-container-toolkit-daemonset controller-revision-hash=85d9894fb6 pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: d13b07524f1df468caf094c0fccfaa4f0cbcedb8129f8fdb880389b16b125be5 cni.projectcalico.org/podIP: 10.244.11.120/32 cni.projectcalico.org/podIPs: 10.244.11.120/32 Status: Pending IP: 10.244.11.120 IPs: IP: 10.244.11.120 Controlled By: DaemonSet/nvidia-container-toolkit-daemonset Init Containers: driver-validation: Container ID: cri-o://6b6f1f6b7c1b90e571d5fa57464dcb1f7202a6f9796c743e3f152aeb0bad0072 Image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.0 Image ID: nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:6fe4200960b2b49d6dac1c91e596f61dacb6b3dcff878c84eb74c5136fedd5b6 Port:describe pod nvidia-dcgm-exporter-
``` $ k -n gpu-operator describe po nvidia-dcgm-exporter-bhjnb Name: nvidia-dcgm-exporter-bhjnb Namespace: gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-dcgm-exporter Node: mito26.server.org/192.168.200.96 Start Time: Tue, 22 Nov 2022 02:35:44 +0000 Labels: app=nvidia-dcgm-exporter controller-revision-hash=558d5c6485 pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: c88c059e59af4f85e13e504e4a65a8ef5e0ec0cdd16b512cb0afbc04e58abfcc cni.projectcalico.org/podIP: 10.244.11.125/32 cni.projectcalico.org/podIPs: 10.244.11.125/32 Status: Pending IP: 10.244.11.125 IPs: IP: 10.244.11.125 Controlled By: DaemonSet/nvidia-dcgm-exporter Init Containers: toolkit-validation: Container ID: cri-o://523f91431044841a1706de65676b1efa761a53822b9689578687b52a9c82b7ec Image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.0 Image ID: nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:6fe4200960b2b49d6dac1c91e596f61dacb6b3dcff878c84eb74c5136fedd5b6 Port:describe pod nvidia-device-plugin-daemonset
``` $ k -n gpu-operator describe po nvidia-device-plugin-daemonset-jzlbm Name: nvidia-device-plugin-daemonset-jzlbm Namespace: gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-device-plugin Node: mito26.server.org/192.168.200.96 Start Time: Tue, 22 Nov 2022 02:35:43 +0000 Labels: app=nvidia-device-plugin-daemonset controller-revision-hash=759699b885 pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: 15e9af34d96f8f5a1c27dc25d32ba7f7358ccd2e93a317ea23a98139e3a26c0a cni.projectcalico.org/podIP: 10.244.11.121/32 cni.projectcalico.org/podIPs: 10.244.11.121/32 Status: Pending IP: 10.244.11.121 IPs: IP: 10.244.11.121 Controlled By: DaemonSet/nvidia-device-plugin-daemonset Init Containers: toolkit-validation: Container ID: cri-o://53cff9e385493146546f6b4fb12b10037e7b1a99aa9124f56cb769be30e9cb6f Image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.0 Image ID: nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:6fe4200960b2b49d6dac1c91e596f61dacb6b3dcff878c84eb74c5136fedd5b6 Port:describe pod nvidia-driver-daemonset
``` $ k -n gpu-operator describe po nvidia-driver-daemonset-k8tvc Name: nvidia-driver-daemonset-k8tvc Namespace: gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-driver Node: mito26.server.org/192.168.200.96 Start Time: Tue, 22 Nov 2022 02:35:43 +0000 Labels: app=nvidia-driver-daemonset controller-revision-hash=6775f65988 pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: 3f65dab3a8ed4b1e90955c04c9f9e8b61b9cc15f763d95e5cf06a8aa58974e2b cni.projectcalico.org/podIP: 10.244.11.123/32 cni.projectcalico.org/podIPs: 10.244.11.123/32 Status: Pending IP: 10.244.11.123 IPs: IP: 10.244.11.123 Controlled By: DaemonSet/nvidia-driver-daemonset Init Containers: mofed-validation: Container ID: cri-o://3bcf5bf91d19a174f28467e05cf7d0171b140d2b5261fbf37f9b93b9b065ce67 Image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.0 Image ID: nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:6fe4200960b2b49d6dac1c91e596f61dacb6b3dcff878c84eb74c5136fedd5b6 Port:describe pod nvidia-operator-validator
``` $ k -n gpu-operator describe po nvidia-operator-validator-hlfp7 Name: nvidia-operator-validator-hlfp7 Namespace: gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-operator-validator Node: mito26.server.org/192.168.200.96 Start Time: Tue, 22 Nov 2022 02:35:43 +0000 Labels: app=nvidia-operator-validator app.kubernetes.io/part-of=gpu-operator controller-revision-hash=df5cbdc4f pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: f85abe423ababe00c088804732b87bdaac351df67a81d90f6c169a8ea71c56af cni.projectcalico.org/podIP: 10.244.11.122/32 cni.projectcalico.org/podIPs: 10.244.11.122/32 Status: Pending IP: 10.244.11.122 IPs: IP: 10.244.11.122 Controlled By: DaemonSet/nvidia-operator-validator Init Containers: driver-validation: Container ID: cri-o://78c087e010432e9a3bb21d98a701bb46d14ec2a10348cca52729547f58f7a88f Image: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.0 Image ID: nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:6fe4200960b2b49d6dac1c91e596f61dacb6b3dcff878c84eb74c5136fedd5b6 Port:[x] If a pod/ds is in an error state or pending state
kubectl logs -n NAMESPACE POD_NAMESince kubectl log has no good output, we submit the output of /var/log/pods.gpu-feature-discovery log
``` $ sudo cat /var/log/pods/gpu-operator_gpu-feature-discovery-rwkvl_82dbe642-8f8b-4634-812e-171ee06cd5c1/toolkit-validation/0.log 2022-11-22T04:05:53.914790700+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:05:58.917849636+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:06:03.921302039+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:06:08.924586551+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:06:13.927938990+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:06:18.931333373+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:06:23.934661339+00:00 stdout F waiting for nvidia container stack to be setup ```nvidia-container-toolkit-daemonsets log
``` $ sudo cat /var/log/pods/gpu-operator_nvidia-container-toolkit-daemonset-k8dl6_3c2a649b-4eb5-4b15-b60b-7114f77692cf/driver-validation/0.log 2022-11-22T04:09:04.909735641+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T04:09:04.910083989+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T04:09:09.910755502+00:00 stdout F running command chroot with args [/run/nvidia/driver nvidia-smi] 2022-11-22T04:09:09.913204824+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T04:09:09.913523235+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T04:09:14.914704866+00:00 stdout F running command chroot with args [/run/nvidia/driver nvidia-smi] 2022-11-22T04:09:14.917195040+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T04:09:14.917522610+00:00 stdout F command failed, retrying after 5 seconds ```nvidia-dcgm-exporter log
``` $ sudo cat /var/log/pods/gpu-operator_nvidia-dcgm-exporter-bhjnb_f76a839f-4670-4244-b145-a2bb6d506b86/toolkit-validation/0.log 2022-11-22T04:17:04.572623880+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:17:09.575852179+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:17:14.579083583+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:17:19.582244567+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:17:24.585574474+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:17:29.588779832+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T04:17:34.591940411+00:00 stdout F waiting for nvidia container stack to be setup ```nvidia-device-plugin-daemonset log
``` $ sudo cat /var/log/pods/gpu-operator_nvidia-device-plugin-daemonset-jzlbm_3fb1cb66-758c-4187-92cf-f8c956276ed1/toolkit-validation/0.log 2022-11-22T05:36:37.303169869+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:36:42.306521315+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:36:47.309980509+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:36:52.313370794+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:36:57.316784848+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:37:02.320326747+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:37:07.323688230+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:37:12.327225309+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:37:17.330619005+00:00 stdout F waiting for nvidia container stack to be setup 2022-11-22T05:37:22.334102431+00:00 stdout F waiting for nvidia container stack to be setu ```nvidia-driver-daemonset log
``` $ sudo cat /var/log/pods/gpu-operator_nvidia-driver-daemonset-k8tvc_d6d1d91a-5e0e-4701-a01d-9a0585c5232e/mofed-validation/0.log 2022-11-22T05:38:21.156056921+00:00 stdout F running command bash with args [-c stat /run/mellanox/drivers/.driver-ready] 2022-11-22T05:38:21.160991578+00:00 stderr F stat: cannot statx '/run/mellanox/drivers/.driver-ready': No such file or directory 2022-11-22T05:38:21.161307447+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:38:26.162320590+00:00 stdout F running command bash with args [-c stat /run/mellanox/drivers/.driver-ready] 2022-11-22T05:38:26.167358516+00:00 stderr F stat: cannot statx '/run/mellanox/drivers/.driver-ready': No such file or directory 2022-11-22T05:38:26.167683096+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:38:31.168620993+00:00 stdout F running command bash with args [-c stat /run/mellanox/drivers/.driver-ready] 2022-11-22T05:38:31.173547738+00:00 stderr F stat: cannot statx '/run/mellanox/drivers/.driver-ready': No such file or directory 2022-11-22T05:38:31.173882297+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:38:36.174883940+00:00 stdout F running command bash with args [-c stat /run/mellanox/drivers/.driver-ready] 2022-11-22T05:38:36.179943712+00:00 stderr F stat: cannot statx '/run/mellanox/drivers/.driver-ready': No such file or directory 2022-11-22T05:38:36.180201659+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:38:41.180712282+00:00 stdout F running command bash with args [-c stat /run/mellanox/drivers/.driver-ready] 2022-11-22T05:38:41.185811180+00:00 stderr F stat: cannot statx '/run/mellanox/drivers/.driver-ready': No such file or directory 2022-11-22T05:38:41.186148142+00:00 stdout F command failed, retrying after 5 seconds ```nvidia-operator-validator log
``` $ sudo cat /var/log/pods/gpu-operator_nvidia-operator-validator-hlfp7_eb50eb18-4f2b-402d-9308-8de3322a05ab/driver-validation/0.log 2022-11-22T05:39:40.043664142+00:00 stdout F running command chroot with args [/run/nvidia/driver nvidia-smi] 2022-11-22T05:39:40.046023497+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T05:39:40.046306559+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:39:45.047051403+00:00 stdout F running command chroot with args [/run/nvidia/driver nvidia-smi] 2022-11-22T05:39:45.049490493+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T05:39:45.049808369+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:39:50.050370297+00:00 stdout F running command chroot with args [/run/nvidia/driver nvidia-smi] 2022-11-22T05:39:50.052832256+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T05:39:50.053162408+00:00 stdout F command failed, retrying after 5 seconds 2022-11-22T05:39:55.053958031+00:00 stdout F running command chroot with args [/run/nvidia/driver nvidia-smi] 2022-11-22T05:39:55.056469130+00:00 stderr F chroot: failed to run command 'nvidia-smi': No such file or directory 2022-11-22T05:39:55.056816581+00:00 stdout F command failed, retrying after 5 seconds ```[ ] Output of running a container on the GPU machine:
docker run -it alpine echo foo[ ] Docker configuration file:
cat /etc/docker/daemon.json[ ] Docker runtime configuration:
docker info | grep runtime[x] NVIDIA shared directory:
ls -la /run/nvidia[x] NVIDIA packages directory:
ls -la /usr/local/nvidia/toolkit[x] NVIDIA driver directory:
ls -la /run/nvidia/driver[ ] kubelet logs
journalctl -u kubelet > kubelet.logs