Closed parthyadav3105 closed 7 months ago
I am using kubernetes v1.29.0
I0103 07:34:27.951578 1 controller.go:658] "resource controller: Found no pending pod claims" key="schedulingCtx:default/dra-sample"
The above log from nvidia-dra-controller occurs because from kubernetes v1.28+ onwards ResourceClaim
created from ResourceClaimTemplate
are created using metadata.generatedName
(113722, 117351) to prevent name conflicts.
NVIDIA/k8s-dra-driver repository is currently using older v0.27.0-beta.0 version of package k8s.io/dynamic-resource-allocation which does not handles this. Updating driver to use version v0.28.0+ for k8s.io/dynamic-resource-allocation
will fix the issue.
Hence closing.
@klueska, @elezar I can add this fix along with support for batch allocation in the driver if that does not affect any ongoing roadmaps.
I am working with a kubeadm cluster on a baremetal server with 3 Nvidia A40 gpu.
Problem: The gpu.nvidia.com driver is not able to allocate gpu.
My deployment:
Here are Outputs for debugging:
$ kubectl describe pod dra-sample
$ kubectl logs -n gpu-operator nvidia-dra-device-plugin-k8s-dra-driver-controller-c8d57bb4fw6s
$ kubectl logs -n gpu-operator nvidia-dra-device-plugin-k8s-dra-driver-kubelet-plugin-7txkm
We can see the node is not able to pick up gpu resources: $ kubectl describe node nm-shakti-worker6
$ kubectl describe -n gpu-operator nodeallocationstates.nas.gpu.resource.nvidia.com nm-shakti-worker6
How I installed k8s-dra-driver?
To fulfill prerequisites, I simply used gpu-operator as I already had it working in cluster. Just disabled devicePlugin.
For installing NVIDIA/k8s-dra-driver, I followed script(demo/clusters/kind/scripts/build-driver-image.sh) to build image and then ran:
Hence, it gave me: $ kubectl get pods -n gpu-operator
The step-up works fine with default driver from gpu-operator. But it fails to allocate and find gpu with NVIDIA/k8s-dra-driver. My installation is probably wrong. What did I miss?
Thanks!