search

NVIDIA / gpu-operator

NVIDIA GPU Operator creates/configures/manages GPUs atop Kubernetes
Apache License 2.0
1.77k stars 286 forks source link

NVIDIA GPU Operator 24.23.0 Failed on OCP 4.14.23 Cluster #723

Open habjouqa opened 4 months ago

habjouqa commented 4 months ago

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

1. Quick Debug Information

2. Issue or feature description

Briefly explain the issue in terms of expected behavior and current behavior.

Installed OCP 4.14.23 cluster, where one worker node has a GPU but the gpu-cluster-policy has failed and the GPU-related pods are not working.

3. Steps to reproduce the issue

Detailed steps to reproduce the issue.

  1. Install OCP 4.14.23.
  2. Follow the instructions at IBM Docs to install "Node Feature Discovery Operator" and "Nvidia GPU Operator" from Nvidia's website.
  3. Install the Node Feature Discovery Operator: Nvidia Docs
    • Installed successfully.
  4. Install the Nvidia GPU Operator: Nvidia Docs
    • Failed. Followed steps related to GPU (not vGPU) using the web console (not the CLI).
  5. Create the ClusterPolicy instance.
  6. Create the cluster policy using the web console:
    • By the end of this step, the state of the gpu-cluster-policy should be "State: ready," but the state is "State: not ready," causing the GPU-related pods to fail (see attached screenshot "gpu-pods.png").
    • ClusterPolicy logs indicate: "ClusterPolicy is not ready, states not ready: [state-driver state-container-toolkit state-operator-validation state-device-plugin state-dcgm state-dcgm-exporter gpu-feature-discovery]".

I contacted IBM Support, and they referred me to the logs at /var/log/nvidia-installer.log. These logs show failed driver installations. I uninstalled the "Node Feature Discovery Operator" and "Nvidia GPU Operator," then reinstalled them following the same steps and restarted the node. The drivers are now successfully installed. Attached are two nvidia-installer logs showing the state before and after the restart:

4. Information to attach (optional if deemed irrelevant)

nvidia-installer_BeforeRestart.log nvidia-installer_AfterRestart.log

Collecting full debug bundle (optional):

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/master/hack/must-gather.sh 
chmod +x must-gather.sh
./must-gather.sh

NOTE: please refer to the must-gather script for debug data collected.

This bundle can be submitted to us via email: operator_feedback@nvidia.com

### Tasks
habjouqa commented 4 months ago

I've just send the must gather to the following email: operator_feedback@nvidia.com

shivamerla commented 4 months ago

thanks @habjouqa will take a look

AlessandroPomponio commented 3 months ago

Hello @shivamerla, has there been any update on this?

psy-q commented 2 months ago

There's a workaround that doesn't involve reinstalling the operator:

  1. Take a backup of your ClusterPolicy manifest
  2. Delete the ClusterPolicy
  3. Create the same ClusterPolicy again based on your backup

After a few minutes the items in states not ready: should resolve themselves.