Unable to cordon nodes

[guyst16]

1. Quick Debug Checklist

• [ ] Are you running on an Ubuntu 18.04 node?
• [ ] Are you running Kubernetes v1.13+?
• [ ] Are you running Docker (>= 18.06) or CRIO (>= 1.13+)?
• [ ] Do you have i2c_core and ipmi_msghandler loaded on the nodes?
• [x] Did you apply the CRD (kubectl describe clusterpolicies --all-namespaces)
• [x] Run on OpenShift v4.10.16

• [x] GPU operator v22.9.1

  1. Issue or feature description

  I tried to cordon a GPU worker node but it immediately initiated un-cordon. I followed this closed issue and relabeled the node to nvidia.com/gpu.deploy.driver: 'false', the driver pod got terminated but the node still got back to un-cordon every time.

  Cluster policy:

  apiVersion: nvidia.com/v1
  kind: ClusterPolicy
  metadata:
  creationTimestamp: "2023-05-14T13:18:50Z"
  generation: 3
  name: gpu-cluster-policy
  resourceVersion: "113161953"
  uid: d5b6ac03-bc99-40dd-a41a-e899d3874596
  spec:
  daemonsets:
  rollingUpdate:
    maxUnavailable: "1"
  updateStrategy: RollingUpdate
  dcgm:
  enabled: true
  dcgmExporter:
  config:
    name: ""
  enabled: true
  serviceMonitor:
    enabled: true
  devicePlugin:
  config:
    default: ""
    name: ""
  enabled: true
  driver:
  certConfig:
    name: ""
  enabled: true
  kernelModuleConfig:
    name: ""
  licensingConfig:
    configMapName: ""
    nlsEnabled: false
  repoConfig:
    configMapName: ""
  upgradePolicy:
    autoUpgrade: true
    drain:
      deleteEmptyDir: false
      enable: false
      force: false
      timeoutSeconds: 300
    maxParallelUpgrades: 1
    podDeletion:
      deleteEmptyDir: false
      force: false
      timeoutSeconds: 300
    waitForCompletion:
      timeoutSeconds: 0
  virtualTopology:
    config: ""
  gfd:
  enabled: true
  mig:
  strategy: single
  migManager:
  enabled: true
  nodeStatusExporter:
  enabled: true
  operator:
  defaultRuntime: crio
  initContainer: {}
  runtimeClass: nvidia
  use_ocp_driver_toolkit: true
  sandboxDevicePlugin:
  enabled: true
  sandboxWorkloads:
  defaultWorkload: container
  enabled: false
  toolkit:
  enabled: true
  installDir: /usr/local/nvidia
  validator:
  plugin:
    env:
    - name: WITH_WORKLOAD
      value: "true"
  vfioManager:
  enabled: true
  vgpuDeviceManager:
  config:
    default: default
  enabled: true
  vgpuManager:
  enabled: false
  status:
  namespace: nvidia-gpu-operator
  state: ready

  Node info:

  Name:               <node-name>
  Roles:              worker
  Labels:             beta.kubernetes.io/arch=amd64
                  beta.kubernetes.io/instance-type=Standard_NC12s_v3
                  beta.kubernetes.io/os=linux
                  cpu-feature.node.kubevirt.io/3dnow=true
                  cpu-feature.node.kubevirt.io/3dnowext=true
                  cpu-feature.node.kubevirt.io/abm=true
                  cpu-feature.node.kubevirt.io/acpi=true
                  cpu-feature.node.kubevirt.io/adx=true
                  cpu-feature.node.kubevirt.io/aes=true
                  cpu-feature.node.kubevirt.io/arat=true
                  cpu-feature.node.kubevirt.io/bmi1=true
                  cpu-feature.node.kubevirt.io/bmi2=true
                  cpu-feature.node.kubevirt.io/clflushopt=true
                  cpu-feature.node.kubevirt.io/clwb=true
                  cpu-feature.node.kubevirt.io/cr8legacy=true
                  cpu-feature.node.kubevirt.io/erms=true
                  cpu-feature.node.kubevirt.io/fsgsbase=true
                  cpu-feature.node.kubevirt.io/hypervisor=true
                  cpu-feature.node.kubevirt.io/la57=true
                  cpu-feature.node.kubevirt.io/mmxext=true
                  cpu-feature.node.kubevirt.io/monitor=true
                  cpu-feature.node.kubevirt.io/movbe=true
                  cpu-feature.node.kubevirt.io/mpx=true
                  cpu-feature.node.kubevirt.io/npt=true
                  cpu-feature.node.kubevirt.io/pclmuldq=true
                  cpu-feature.node.kubevirt.io/pcommit=true
                  cpu-feature.node.kubevirt.io/pdpe1gb=true
                  cpu-feature.node.kubevirt.io/pku=true
                  cpu-feature.node.kubevirt.io/popcnt=true
                  cpu-feature.node.kubevirt.io/rdrand=true
                  cpu-feature.node.kubevirt.io/rdtscp=true
                  cpu-feature.node.kubevirt.io/smap=true
                  cpu-feature.node.kubevirt.io/smep=true
                  cpu-feature.node.kubevirt.io/ss=true
                  cpu-feature.node.kubevirt.io/sse4.2=true
                  cpu-feature.node.kubevirt.io/sse4a=true
                  cpu-feature.node.kubevirt.io/svm=true
                  cpu-feature.node.kubevirt.io/xgetbv1=true
                  cpu-feature.node.kubevirt.io/xsave=true
                  cpu-feature.node.kubevirt.io/xsaveopt=true
                  cpu-model-migration.node.kubevirt.io/Nehalem=true
                  cpu-model-migration.node.kubevirt.io/Opteron_G1=true
                  cpu-model-migration.node.kubevirt.io/Opteron_G2=true
                  cpu-model-migration.node.kubevirt.io/Opteron_G3=true
                  cpu-model-migration.node.kubevirt.io/Penryn=true
                  cpu-model-migration.node.kubevirt.io/Westmere=true
                  cpu-model.node.kubevirt.io/Nehalem=true
                  cpu-model.node.kubevirt.io/Opteron_G1=true
                  cpu-model.node.kubevirt.io/Opteron_G2=true
                  cpu-model.node.kubevirt.io/Penryn=true
                  cpu-model.node.kubevirt.io/Westmere=true
                  cpu-vendor.node.kubevirt.io/AMD=true
                  cpumanager=false
                  failure-domain.beta.kubernetes.io/region=westus
                  failure-domain.beta.kubernetes.io/zone=1
                  feature.node.kubernetes.io/cpu-cpuid.ADX=true
                  feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                  feature.node.kubernetes.io/cpu-cpuid.AVX=true
                  feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                  feature.node.kubernetes.io/cpu-cpuid.FMA3=true
                  feature.node.kubernetes.io/cpu-cpuid.HLE=true
                  feature.node.kubernetes.io/cpu-cpuid.HYPERVISOR=true
                  feature.node.kubernetes.io/cpu-cpuid.RTM=true
                  feature.node.kubernetes.io/cpu-hardware_multithreading=false
                  feature.node.kubernetes.io/kernel-config.NO_HZ=true
                  feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true
                  feature.node.kubernetes.io/kernel-selinux.enabled=true
                  feature.node.kubernetes.io/kernel-version.full=4.18.0-305.45.1.el8_4.x86_64
                  feature.node.kubernetes.io/kernel-version.major=4
                  feature.node.kubernetes.io/kernel-version.minor=18
                  feature.node.kubernetes.io/kernel-version.revision=0
                  feature.node.kubernetes.io/pci-10de.present=true
                  feature.node.kubernetes.io/pci-1414.present=true
                  feature.node.kubernetes.io/system-os_release.ID=rhcos
                  feature.node.kubernetes.io/system-os_release.OPENSHIFT_VERSION=4.10
                  feature.node.kubernetes.io/system-os_release.OSTREE_VERSION=410.84.202205191234-0
                  feature.node.kubernetes.io/system-os_release.RHEL_VERSION=8.4
                  feature.node.kubernetes.io/system-os_release.VERSION_ID=4.10
                  feature.node.kubernetes.io/system-os_release.VERSION_ID.major=4
                  feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=10
                  host-model-cpu.node.kubevirt.io/Opteron_G3=true
                  host-model-required-features.node.kubevirt.io/3dnow=true
                  host-model-required-features.node.kubevirt.io/3dnowext=true
                  host-model-required-features.node.kubevirt.io/acpi=true
                  host-model-required-features.node.kubevirt.io/adx=true
                  host-model-required-features.node.kubevirt.io/aes=true
                  host-model-required-features.node.kubevirt.io/arat=true
                  host-model-required-features.node.kubevirt.io/bmi1=true
                  host-model-required-features.node.kubevirt.io/bmi2=true
                  host-model-required-features.node.kubevirt.io/clflushopt=true
                  host-model-required-features.node.kubevirt.io/clwb=true
                  host-model-required-features.node.kubevirt.io/cr8legacy=true
                  host-model-required-features.node.kubevirt.io/erms=true
                  host-model-required-features.node.kubevirt.io/fsgsbase=true
                  host-model-required-features.node.kubevirt.io/hypervisor=true
                  host-model-required-features.node.kubevirt.io/la57=true
                  host-model-required-features.node.kubevirt.io/mmxext=true
                  host-model-required-features.node.kubevirt.io/monitor=true
                  host-model-required-features.node.kubevirt.io/movbe=true
                  host-model-required-features.node.kubevirt.io/mpx=true
                  host-model-required-features.node.kubevirt.io/npt=true
                  host-model-required-features.node.kubevirt.io/pclmuldq=true
                  host-model-required-features.node.kubevirt.io/pcommit=true
                  host-model-required-features.node.kubevirt.io/pdpe1gb=true
                  host-model-required-features.node.kubevirt.io/pku=true
                  host-model-required-features.node.kubevirt.io/rdrand=true
                  host-model-required-features.node.kubevirt.io/smap=true
                  host-model-required-features.node.kubevirt.io/smep=true
                  host-model-required-features.node.kubevirt.io/ss=true
                  host-model-required-features.node.kubevirt.io/sse4.2=true
                  host-model-required-features.node.kubevirt.io/xgetbv1=true
                  host-model-required-features.node.kubevirt.io/xsave=true
                  host-model-required-features.node.kubevirt.io/xsaveopt=true
                  kubernetes.io/arch=amd64
                  kubernetes.io/hostname=<node-name>
                  kubernetes.io/os=linux
                  kubevirt.io/schedulable=true
                  node-role.kubernetes.io/worker=
                  node.kubernetes.io/instance-type=Standard_NC12s_v3
                  node.openshift.io/os_id=rhcos
                  nvidia.com/cuda.driver.major=525
                  nvidia.com/cuda.driver.minor=60
                  nvidia.com/cuda.driver.rev=13
                  nvidia.com/cuda.runtime.major=12
                  nvidia.com/cuda.runtime.minor=0
                  nvidia.com/gfd.timestamp=1685350119
                  nvidia.com/gpu-driver-upgrade-state=upgrade-done
                  nvidia.com/gpu.compute.major=7
                  nvidia.com/gpu.compute.minor=0
                  nvidia.com/gpu.count=2
                  nvidia.com/gpu.deploy.container-toolkit=true
                  nvidia.com/gpu.deploy.dcgm=true
                  nvidia.com/gpu.deploy.dcgm-exporter=true
                  nvidia.com/gpu.deploy.device-plugin=true
                  nvidia.com/gpu.deploy.driver=false
                  nvidia.com/gpu.deploy.gpu-feature-discovery=true
                  nvidia.com/gpu.deploy.node-status-exporter=true
                  nvidia.com/gpu.deploy.nvsm=
                  nvidia.com/gpu.deploy.operator-validator=true
                  nvidia.com/gpu.family=volta
                  nvidia.com/gpu.machine=Virtual-Machine
                  nvidia.com/gpu.memory=16384
                  nvidia.com/gpu.present=true
                  nvidia.com/gpu.product=Tesla-V100-PCIE-16GB
                  nvidia.com/gpu.replicas=1
                  nvidia.com/mig.capable=false
                  nvidia.com/mig.strategy=single
                  topology.disk.csi.azure.com/zone=
                  topology.kubernetes.io/region=westus
                  topology.kubernetes.io/zone=1
  Annotations:        cloud.network.openshift.io/egress-ipconfig:
                    [{"interface":"<node-name>-nic","ifaddr":{"ipv4":"10.0.128.0/17"},"capacity":{"ip":255}}]
                  csi.volume.kubernetes.io/nodeid: {"disk.csi.azure.com":"<node-name>"}
                  kubevirt.io/heartbeat: 2023-05-29T09:27:14Z
                  machine.openshift.io/machine: openshift-machine-api/<node-name>
                  machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
                  machineconfiguration.openshift.io/currentConfig: rendered-worker-f2f7335cce531a3aea2207786af06534
                  machineconfiguration.openshift.io/desiredConfig: rendered-worker-f2f7335cce531a3aea2207786af06534
                  machineconfiguration.openshift.io/reason: 
                  machineconfiguration.openshift.io/state: Done
                  nfd.node.kubernetes.io/extended-resources: 
                  nfd.node.kubernetes.io/feature-labels:
                    cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.FMA3,cpu-cpuid.HLE,cpu-cpuid.HYPERVISOR,cpu-cpuid.RTM,cpu-hardware_mu...
                  nfd.node.kubernetes.io/worker.version: undefined
                  nvidia.com/gpu-driver-upgrade-enabled: true
                  volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp:  Sun, 28 May 2023 16:24:25 +0300
  Taints:             <none>
  Unschedulable:      false
  Lease:
  HolderIdentity:  <HolderIdentity>
  AcquireTime:     <unset>
  RenewTime:       Mon, 29 May 2023 12:28:09 +0300
  Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 29 May 2023 12:23:42 +0300   Sun, 28 May 2023 16:24:25 +0300   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 29 May 2023 12:23:42 +0300   Sun, 28 May 2023 16:24:25 +0300   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 29 May 2023 12:23:42 +0300   Sun, 28 May 2023 16:24:25 +0300   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 29 May 2023 12:23:42 +0300   Sun, 28 May 2023 16:24:55 +0300   KubeletReady                 kubelet is posting ready status
  Addresses:
  Hostname:    <Hostname>
  InternalIP:  10.0.128.8
  Capacity:
  attachable-volumes-azure-disk:  24
  cpu:                            12
  devices.kubevirt.io/kvm:        1k
  devices.kubevirt.io/sev:        1k
  devices.kubevirt.io/tun:        1k
  devices.kubevirt.io/vhost-net:  1k
  ephemeral-storage:              133682156Ki
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         231103484Ki
  nvidia.com/gpu:                 2
  pods:                           250
  Allocatable:
  attachable-volumes-azure-disk:  24
  cpu:                            11500m
  devices.kubevirt.io/kvm:        0
  devices.kubevirt.io/sev:        0
  devices.kubevirt.io/tun:        1k
  devices.kubevirt.io/vhost-net:  1k
  ephemeral-storage:              123201474766
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         229952508Ki
  nvidia.com/gpu:                 2
  pods:                           250
  System Info:
  Machine ID:                             <Machine ID>
  System UUID:                            <System UUID>
  Boot ID:                                812180ed-6f6b-4980-9a05-893c0aa7ac42
  Kernel Version:                         4.18.0-305.45.1.el8_4.x86_64
  OS Image:                               Red Hat Enterprise Linux CoreOS 410.84.202205191234-0 (Ootpa)
  Operating System:                       linux
  Architecture:                           amd64
  Container Runtime Version:              cri-o://1.23.2-11.rhaos4.10.git41c0779.el8
  Kubelet Version:                        v1.23.5+3afdacb
  Kube-Proxy Version:                     v1.23.5+3afdacb
  ProviderID:                               <Provider-ID>
  Non-terminated Pods:                      (25 in total)
  Namespace                               Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                               ----                                        ------------  ----------  ---------------  -------------  ---
  nvidia-gpu-operator                     gpu-feature-discovery-vw84l                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         39m
  nvidia-gpu-operator                     nvidia-container-toolkit-daemonset-h96d9    0 (0%)        0 (0%)      0 (0%)           0 (0%)         43m
  nvidia-gpu-operator                     nvidia-dcgm-exporter-wlpdv                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         43m
  nvidia-gpu-operator                     nvidia-dcgm-mhstg                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         43m
  nvidia-gpu-operator                     nvidia-device-plugin-daemonset-j2l76        0 (0%)        0 (0%)      0 (0%)           0 (0%)         43m
  nvidia-gpu-operator                     nvidia-node-status-exporter-jgdfx           0 (0%)        0 (0%)      0 (0%)           0 (0%)         39m
  nvidia-gpu-operator                     nvidia-operator-validator-t5dj6             0 (0%)        0 (0%)      0 (0%)           0 (0%)         43m
  openshift-cluster-csi-drivers           azure-disk-csi-driver-node-bcsnf            30m (0%)      0 (0%)      150Mi (0%)       0 (0%)         20h
  openshift-cluster-node-tuning-operator  tuned-8g279                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         20h
  openshift-cnv                           bridge-marker-b9ww9                         10m (0%)      0 (0%)      15Mi (0%)        0 (0%)         20h
  openshift-cnv                           kube-cni-linux-bridge-plugin-7mghq          10m (0%)      0 (0%)      15Mi (0%)        0 (0%)         20h
  openshift-cnv                           nmstate-handler-w4smd                       100m (0%)     0 (0%)      100Mi (0%)       0 (0%)         20h
  openshift-cnv                           virt-handler-jtbb7                          10m (0%)      0 (0%)      230Mi (0%)       0 (0%)         20h
  openshift-dns                           dns-default-cxqqr                           60m (0%)      0 (0%)      110Mi (0%)       0 (0%)         20h
  openshift-dns                           node-resolver-5ddkb                         5m (0%)       0 (0%)      21Mi (0%)        0 (0%)         20h
  openshift-image-registry                node-ca-hk8dh                               10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         20h
  openshift-ingress-canary                ingress-canary-qkd5x                        10m (0%)      0 (0%)      20Mi (0%)        0 (0%)         20h
  openshift-machine-config-operator       machine-config-daemon-rfrhp                 40m (0%)      0 (0%)      100Mi (0%)       0 (0%)         20h
  openshift-monitoring                    node-exporter-vlhbj                         9m (0%)       0 (0%)      47Mi (0%)        0 (0%)         20h
  openshift-multus                        multus-additional-cni-plugins-zccwm         10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         20h
  openshift-multus                        multus-t4w44                                10m (0%)      0 (0%)      65Mi (0%)        0 (0%)         20h
  openshift-multus                        network-metrics-daemon-4mwkn                20m (0%)      0 (0%)      120Mi (0%)       0 (0%)         20h
  openshift-network-diagnostics           network-check-target-6vpws                  10m (0%)      0 (0%)      15Mi (0%)        0 (0%)         20h
  openshift-nfd                           nfd-worker-v5dfp                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h
  openshift-sdn                           sdn-7t67j                                   115m (1%)     0 (0%)      240Mi (0%)       0 (0%)         20h
  Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests     Limits
  --------                       --------     ------
  cpu                            469m (4%)    0 (0%)
  memory                         1318Mi (0%)  0 (0%)
  ephemeral-storage              0 (0%)       0 (0%)
  hugepages-1Gi                  0 (0%)       0 (0%)
  hugepages-2Mi                  0 (0%)       0 (0%)
  attachable-volumes-azure-disk  0            0
  devices.kubevirt.io/kvm        0            0
  devices.kubevirt.io/sev        0            0
  devices.kubevirt.io/tun        0            0
  devices.kubevirt.io/vhost-net  0            0
  nvidia.com/gpu                 0            0

  Nvidia pods:

  NAME                                                  READY   STATUS      RESTARTS   AGE
  gpu-feature-discovery-lw45k                           1/1     Running     0          22h
  gpu-feature-discovery-vw84l                           1/1     Running     0          93m
  gpu-operator-99bb8d4d6-dn8sh                          1/1     Running     0          25h
  nvidia-container-toolkit-daemonset-h96d9              1/1     Running     0          97m
  nvidia-container-toolkit-daemonset-jqh89              1/1     Running     0          22h
  nvidia-cuda-validator-bhbw4                           0/1     Completed   0          94m
  nvidia-cuda-validator-dxl25                           0/1     Completed   0          22h
  nvidia-dcgm-exporter-vnvfh                            1/1     Running     0          22h
  nvidia-dcgm-exporter-wlpdv                            1/1     Running     0          97m
  nvidia-dcgm-f8569                                     1/1     Running     0          22h
  nvidia-dcgm-mhstg                                     1/1     Running     0          97m
  nvidia-device-plugin-daemonset-j2l76                  1/1     Running     0          97m
  nvidia-device-plugin-daemonset-pwmnb                  1/1     Running     0          22h
  nvidia-device-plugin-validator-494wg                  0/1     Completed   0          22h
  nvidia-device-plugin-validator-kwkt4                  0/1     Completed   0          94m
  nvidia-driver-daemonset-410.84.202205191234-0-cpc7h   2/2     Running     0          22h
  nvidia-node-status-exporter-dtv29                     1/1     Running     0          22h
  nvidia-node-status-exporter-jgdfx                     1/1     Running     0          93m
  nvidia-operator-validator-sfznw                       1/1     Running     0          22h
  nvidia-operator-validator-t5dj6                       1/1     Running     0          97m

  2. Steps to reproduce the issue

  1. Relabel node:

     $ oc label node <gpu-node-name> nvidia.com/gpu.deploy.driver='false' --overwrite

     Mark the node as unscheduable:

     $ oc adm cordon <gpu-node-name>

[shivamerla]

@guyst16 can you attach gpu-operator pod logs to confirm if gpu-operator is triggering un-cordon of the node. Also, can you try with driver.upgradePolicy.autoUpgrade=false in ClusterPolicy and verify if same behavior is happening?