虚拟机HAMI无法调用GPU资源Virtual machine HAMI cannot call GPU resources

[1219354801]

我在wsl2虚拟机上部署了HAMI，当提交任务的时候，HAMI总是无法调用我的GPU资源。这应该怎么办？ I deployed HAMI on a wsl2 virtual machine. When submitting tasks, HAMI always fails to call my GPU resources. What should I do?

Please provide an in-depth description of the question you have:

What do you think about this question?:

Environment:

• HAMi version:
• Kubernetes version:
• Others:

[1219354801]

user@sfyfb:~$ kubectl get pods NAME READY STATUS RESTARTS AGE gpu-pod 0/1 Pending 0 82s user@sfyfb:~$ kubectl describe pod gpu-pod Name: gpu-pod Namespace: default Priority: 0 Service Account: default Node: Labels: Annotations: Status: Pending IP: IPs: Containers: ubuntu-container: Image: ubuntu:18.04 Port: Host Port: Command: bash -c sleep 86400 Limits: nvidia.com/gpu: 1 nvidia.com/gpumem: 3k Requests: nvidia.com/gpu: 1 nvidia.com/gpumem: 3k Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hndm4 (ro) Conditions: Type Status PodScheduled False Volumes: kube-api-access-hndm4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Warning FailedScheduling 103s default-scheduler 0/1 nodes are available: 1 Insufficient nvidia.com/gpu, 1 Insufficient nvidia.com/gpumem. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. user@sfyfb:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME docker-desktop Ready control-plane 3d17h v1.30.2 192.168.65.3 Docker Desktop 5.15.153.1-microsoft-standard-WSL2 docker://27.1.1 user@sfyfb:~$ kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS docker-desktop Ready control-plane 3d21h v1.30.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu=on,kubernetes.io/arch=amd64,kubernetes.io/hostname=docker-desktop,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers= user@sfyfb:~$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-7db6d8ff4d-544gc 1/1 Running 0 3d21h coredns-7db6d8ff4d-slq8z 1/1 Running 0 3d21h etcd-docker-desktop 1/1 Running 6 3d21h kube-apiserver-docker-desktop 1/1 Running 6 3d21h kube-controller-manager-docker-desktop 1/1 Running 6 3d21h kube-proxy-fz4kk 1/1 Running 0 3d21h kube-scheduler-docker-desktop 1/1 Running 6 3d21h storage-provisioner 1/1 Running 0 3d21h vpnkit-controller 1/1 Running 0 3d21h

[1219354801]

虚拟机运行nvidia-smi是没有问题的

[1219354801]

kubectl get pods，节点是pending。Pod处于Pending状态的原因是没有可用的GPU资源，调度器报告了Insufficient nvidia.com/gpu和Insufficient nvidia.com/gpumem。 🙏这应该怎么办

[archlitchi]

user@sfyfb:~$ kubectl get pods NAME READY STATUS RESTARTS AGE gpu-pod 0/1 Pending 0 82s user@sfyfb:~$ kubectl describe pod gpu-pod Name: gpu-pod Namespace: default Priority: 0 Service Account: default Node: Labels: Annotations: Status: Pending IP: IPs: Containers: ubuntu-container: Image: ubuntu:18.04 Port: Host Port: Command: bash -c sleep 86400 Limits: nvidia.com/gpu: 1 nvidia.com/gpumem: 3k Requests: nvidia.com/gpu: 1 nvidia.com/gpumem: 3k Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hndm4 (ro) Conditions: Type Status PodScheduled False Volumes: kube-api-access-hndm4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Warning FailedScheduling 103s default-scheduler 0/1 nodes are available: 1 Insufficient nvidia.com/gpu, 1 Insufficient nvidia.com/gpumem. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. user@sfyfb:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME docker-desktop Ready control-plane 3d17h v1.30.2 192.168.65.3 Docker Desktop 5.15.153.1-microsoft-standard-WSL2 docker://27.1.1 user@sfyfb:~$ kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS docker-desktop Ready control-plane 3d21h v1.30.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu=on,kubernetes.io/arch=amd64,kubernetes.io/hostname=docker-desktop,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers= user@sfyfb:~$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-7db6d8ff4d-544gc 1/1 Running 0 3d21h coredns-7db6d8ff4d-slq8z 1/1 Running 0 3d21h etcd-docker-desktop 1/1 Running 6 3d21h kube-apiserver-docker-desktop 1/1 Running 6 3d21h kube-controller-manager-docker-desktop 1/1 Running 6 3d21h kube-proxy-fz4kk 1/1 Running 0 3d21h kube-scheduler-docker-desktop 1/1 Running 6 3d21h storage-provisioner 1/1 Running 0 3d21h vpnkit-controller 1/1 Running 0 3d21h

based on the pods in kube-system, it seems you haven't installed hami correctly