Issue with installation of multi-node K8s for Kubeflow on OCI and GCP instances

[mshaikh786]

Hello, I am using the following instances on GCP: ` Deepops tag : 22.01

• bootstrap instance -- Ubuntu 18.04 LTS
• kf-master node -- Ubuntu 18.04 LTS (4 CPUs and 16 GB memory) -- set only of management (not listed in inventory as worker node)
• kf-gpu-01 node -- A GPU instance with Tesla T4 running Ubuntu 18.04 LTS `

The setup script on bootstrap node runs fine and installs ansible and other tools. I have set the deepops_gpu_operator_enabled: true in config/group_vars/k8s-cluster.yml The following runs and installs K8s on both management and worker node without any issue. When I try to verify the installation for presence of GPU, I get the following: $ ./scripts/k8s/verify_gpu.sh Starting './scripts/k8s/verify_gpu.sh'; DeepOps version '' job_name=cluster-gpu-tests total_gpus=0 Creating/Deleting sandbox Namespace updating test yml downloading containers ... job.batch/cluster-gpu-tests condition met executing ... No resources found in cluster-gpu-verify namespace. GPU verification can't be executed, please check GPU node directly

The state of pods from all namespaces is as follows: `

• deepops-nfs-client-provisioner nfs-subdir-external-provisioner-76c77f6b97-w8fxg 1/1 Running 0 79m
• default gpu-operator-5f8b7c4f59-48hpf 1/1 Running 0 80m
• default nvidia-gpu-operator-node-feature-discovery-master-74db7c569dhzz 1/1 Running 0 80m
• default nvidia-gpu-operator-node-feature-discovery-worker-2pxl4 0/1 CrashLoopBackOff 17 80m
• default nvidia-gpu-operator-node-feature-discovery-worker-cnkx2 1/1 Running 0 80m
• kube-system calico-kube-controllers-8575b76f66-q4z7j 1/1 Running 0 82m
• kube-system calico-node-6jws9 1/1 Running 0 82m
• kube-system calico-node-rth77 1/1 Running 0 82m
• kube-system coredns-8474476ff8-x8xf6 1/1 Running 0 81m
• kube-system coredns-8474476ff8-zs98z 1/1 Running 0 81m
• kube-system dns-autoscaler-7df78bfcfb-qdc82 1/1 Running 0 81m
• kube-system kube-apiserver-kf-master 1/1 Running 0 83m
• kube-system kube-controller-manager-kf-master 1/1 Running 1 83m
• kube-system kube-proxy-24qsr 1/1 Running 0 82m
• kube-system kube-proxy-2nbk7 1/1 Running 0 82m
• kube-system kube-scheduler-kf-master 1/1 Running 1 83m
• kube-system kubernetes-dashboard-6c96f5b677-l8jjt 1/1 Running 0 81m
• kube-system kubernetes-metrics-scraper-694c6bdbc9-hmhrx 1/1 Running 0 81m
• kube-system nginx-proxy-kf-gpu-01 1/1 Running 0 82m
• kube-system nodelocaldns-5cs9g 1/1 Running 0 81m
• kube-system nodelocaldns-gxdnb 1/1 Running 0 81m

`

I am seeing the same behaviour on Oracle cloud instances too. any prompt guidance will be highly appreciated.

[ajdecon]

@mshaikh786 :

I'm guessing there's some issue here with GPU Operator not detecting the GPUs. I'm basing this on the lack of GPUs found, and on the pod nvidia-gpu-operator-node-feature-discovery-worker-2pxl4 in CrashLoopBackoff. 😄

Can you please share the following information? This will hopefully give us an idea for why this isn't working, or at least tell us where to dig next.

• Logs from the crashlooping pod, i.e. kubectl logs nvidia-gpu-operator-node-feature-discovery-worker-2pxl4
• Output of kubectl describe for the nodes where you expect to find GPUs
• Any other changes made to the group_vars/k8s-cluster.yml file

[mshaikh786]

Dear @ajdeco, After running the ansible-playbook to install gpu-operator separately, I investigated the pod (nvidia-gpu-operator-node-feature-discovery) in question with CrashLoopBackOff . It appeared to have issue with connecting to node with GPU. Fixing the iptables did the job, and after a short time, the pod was able to connect to the endpoint. The scripts/k8s/verify_gpu.sh is working now.

[ajdecon]

Wonderful, happy to hear it's working! I'll close this issue now, but please feel free to open another if you run into further problems.