When using a precompiled driver and all gpu nodes are not ready, gpu-operator will loop to deleted and recreated `nvidia-driver-daemonset`

Levi080513 commented 1 month ago

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

1. Quick Debug Information

OS/Version(e.g. RHEL8.6, Ubuntu22.04): RHEL8.9
Kernel Version: 4.18.0-513.24.1.el8.9
Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd 1.6.31
K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): 1.25.16
GPU Operator Version: 23.6.2

2. Issue or feature description

Briefly explain the issue in terms of expected behavior and current behavior.

When using a precompiled driver and all gpu nodes are not ready, gpu-operator will loop to eleted and recreated nvidia-driver-daemonset.

3. Steps to reproduce the issue

Detailed steps to reproduce the issue.

Create a k8s cluster with one GPU nodes.
Install gpu-operator and configure driver.usePrecompiled = true.
Login to the GPU node and trigger the node not ready through systemctl stop kubelet
nvidia-driver-daemonset will be deleted and recreated, and this process will continue until the GPU node is Ready.

When the node is not ready, the node taints like this：

Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule

But the nvidia-driver-daemonset pod tolerations is like this:

  tolerations:
  - effect: NoSchedule
    key: nvidia.com/gpu
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists

Node taint node.kubernetes.io/unreachable:NoSchedule is not tolerated, so nvidia-driver-daemonset .status.desiredNumberScheduled is 0.

Following the logic of cleanupStalePrecompiledDaemonsets, nvidia-driver-daemonset will be deleted and then created again because the cluster still has GPU nodes.

https://github.com/NVIDIA/gpu-operator/blob/a9e6a947216518e5940c21523c2400a2f8f4def5/controllers/object_controls.go#L3689-L3728

This does not appear to be normal behavior.

The temporary solution is to add the following configuration when installing gpu-operator:

daemonsets:
  tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoSchedule

4. Information to attach (optional if deemed irrelevant)

[ ] kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE
[ ] kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE
[ ] If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME
[ ] If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers
[ ] Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi
[ ] containerd logs journalctl -u containerd > containerd.log

Collecting full debug bundle (optional):

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/hack/must-gather.sh 
chmod +x must-gather.sh
./must-gather.sh

NOTE: please refer to the must-gather script for debug data collected.

This bundle can be submitted to us via email: operator_feedback@nvidia.com

cdesiniotis commented 1 month ago

@Levi080513 thanks for the detailed issue! I think our logic which detects stale DaemonSets and cleans them up can be improved to avoid the behavior you are experiencing.

Levi080513 commented 4 weeks ago

Is there any latest progress on this issue?

cdesiniotis commented 4 weeks ago

@Levi080513 This change was merged into master and should fix the issue you reported: https://gitlab.com/nvidia/kubernetes/gpu-operator/-/merge_requests/1085. It will be included in the next release. If you are willing to try out these changes before then and confirm it resolves your issue that would be helpful as well.

Levi080513 commented 4 weeks ago

I cherry-pick this MR to version 23.6.2 and it works well. Thx!

charanteja333 commented 1 week ago

Hi @cdesiniotis

We are facing a similart issue but not a precompiled library. Whenever nfd restarts , driver daemon is restarting as well. When driver restarts, it is stuck leading to either restarting the gpu pods or draining the node. We are using chart 24.3.0. will fix be cherry picked to 24.3.0 as well ?

cdesiniotis commented 1 week ago

@charanteja333 what you described is different than what is being reported in this issue. Can you create a new issue with more details on the behavior you are observing?

NVIDIA / gpu-operator