GKE: Pods cannot access/detect GPU device + drive on GPU nodes

[cjidboon94]

When trying to setup nvshare on GKE, installation goes fine and scheduling pods (e.g. the test pods from the README or a simple cuda pod that runs nvidia-smi) goes fine. nvshare.com/gpu gets consumed. However, pods error with nvidia-smi is not found or in the case of e.g. from the pytorch small pod: Traceback (most recent call last): File "/pytorch-add-small.py", line 29, in <module> device = torch.cuda.current_device() File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 479, in current_device _lazy_init() File "/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py", line 214, in _lazy_init torch._C._cuda_init() RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

When scheduling the pod by requesting nvidia.com/gpu, the GPU is visible and the drivers + nvidia-smi are available.

Setup: GKE k8s version: 1.25.10-gke.2700 nvidia-gpu-device-plugin: GKE's own GPU device plugin

How to reproduce:

• Install nvidia-driver-installer daemonset to acquire drivers on nodes as per https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers

• Install nvshare daemonsets according to readme

• Add the following env var to nvshare-device-plugin daemonset as GKE's gpu-device-plugin does not expose this env var and nvshare-device-plugin depends on it:

     - name: NVIDIA_VISIBLE_DEVICES
       value: "0"

• Optional: Add the following affinity to nvshare-device-plugin daemonset so that pods only get scheduled on GPU nodes.

  • 
    affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: cloud.google.com/gke-accelerator
            operator: Exists 

• Deploy a pod that requests an nvshare.com/gpu resource:

  apiVersion: v1
  kind: Pod 
  metadata:
  name: test-nvshare
  spec:
  restartPolicy: OnFailure\
  tolerations:
  - key: nvidia.com/gpu
  effect: NoSchedule
  operator: Exists
  containers:
    - name: test-nvshare
      env:
      - name: NVSHARE_DEBUG
        value: "1"
      image: nvidia/cuda:11.0.3-base-ubi7
      command:
      - bash
      - -c
      - |
        /usr/local/nvidia/bin/nvidia-smi -L; sleep 300
      resources:
        limits:
          nvshare.com/gpu: 1      

Expectation when checking logs: Get GPU information GPU 0: NVIDIA L4 (UUID: GPU-7e0c893c-3254-dfa8-db40-73942c3de761) (This is what you see when scheduling with a nvidia.com/gpu request) Actual output: bash: /usr/local/nvidia/bin/nvidia-smi: No such file or directory

[grgalex]

@cjidboon94

Unfortunately, nvshare currently strictly depends on NVIDIA's upstream K8s device plugin [1].

This is because nvshare's implementation is strictly coupled with NVIDIA's container runtime.

When I have some time next week, I will elaborate on this fully.

A short summary is that nvshare-device-plugin sets the NVIDIA_VISIBLE_DEVICES environment var (or its symbolic /dev/null mount alternative) in containers that request nvshare.com/gpu. NVIDIA's container runtime, which is a runc hook that runs the containers on the node reads this environment variable and mounts the necessary files (libaries, device nodes, binaries [such as nvidia-smi]) into the container.

Without NVIDIA's device plugin, containers requesting a nvshare.com/gpu device will not see the device exposed when running and will fail.

TL;DR:

For the time being, nvidia-device-plugin [1] is a strict prerequisite for operatingnvshare on Kubernetes.

[1] https://github.com/NVIDIA/k8s-device-plugin

[cjidboon94]

Thanks for clarifying. Will see if I can change GKE's device plugin easily to Nvidia's upstream and then get the rest to work