Closed harryge00 closed 5 years ago
I am also seeing this issue. At times it detects all the configured VFs and at other times it fails with the same error.
Hi, I think we were able to reproduce the issue. we think it related to the fact that one or more VFs are in namspaces before you start/restart the device plugin Can you check how many VF network interfaces you see on the host? is it match the max num_vfs?
I don't see all 32 (max) VFs. The following is the output of ibdev2netdev on host $ ibdev2netdev mlx5_0 port 1 ==> ens3f0 (Down) mlx5_1 port 1 ==> ens3f1 (Up) mlx5_10 port 1 ==> enp2s5f2 (Down) mlx5_11 port 1 ==> enp2s5f3 (Down) mlx5_12 port 1 ==> enp2s5f4 (Down) mlx5_13 port 1 ==> enp2s5f5 (Down) mlx5_14 port 1 ==> enp2s5f6 (Down) mlx5_15 port 1 ==> enp2s5f7 (Down) mlx5_16 port 1 ==> enp2s6 (Down) mlx5_17 port 1 ==> enp2s6f1 (Down) mlx5_18 port 1 ==> enp2s6f2 (Down) mlx5_19 port 1 ==> enp2s6f3 (Down) mlx5_20 port 1 ==> enp2s6f4 (Down) mlx5_21 port 1 ==> enp2s6f5 (Down) mlx5_22 port 1 ==> enp2s6f6 (Down) mlx5_23 port 1 ==> enp2s6f7 (Down) mlx5_24 port 1 ==> enp2s7 (Down) mlx5_25 port 1 ==> enp2s7f1 (Down) mlx5_26 port 1 ==> enp2s7f2 (Down) mlx5_27 port 1 ==> enp2s7f3 (Down) mlx5_28 port 1 ==> enp2s7f4 (Down) mlx5_29 port 1 ==> enp2s7f5 (Down) mlx5_30 port 1 ==> enp2s7f6 (Down) mlx5_31 port 1 ==> enp2s7f7 (Down) mlx5_32 port 1 ==> enp2s8 (Down) mlx5_33 port 1 ==> enp2s8f1 (Down)
I have 2 nodes connected back to back on ens3f1.
And, following is the log of rdma device plugin kubectl logs -n kube-system rdma-sriov-dp-ds-h9fmb
2018/10/18 17:46:02 Starting K8s RDMA SRIOV Device Plugin version= 0.2 2018/10/18 17:46:02 Starting FS watcher. 2018/10/18 17:46:02 Starting OS watcher. 2018/10/18 17:46:02 Reading /k8s-rdma-sriov-dev-plugin/config.json 2018/10/18 17:46:02 loaded config: {"mode":"sriov","pfNetdevices":["ens3f1"]} 2018/10/18 17:46:02 sriov device mode Configuring SRIOV on ndev= ens3f1 6 max_vfs = 32 cur_vfs = 32 vf = &{26 virtfn26 true false} vf = &{7 virtfn7 false false} Fail to config vfs for ndev = ens3f1 Fail to configure sriov; error = Link not found 2018/10/18 17:46:02 Starting to serve on /var/lib/kubelet/device-plugins/rdma-sriov-dp.sock 2018/10/18 17:46:02 Registered device plugin with Kubelet exposing devices: []
So are you using sriov-cni as well? do you have some containers that are using the VF in that node?
Yes for sriov-cni.
It worked for a few PODs initially which got sriov vf. I wasn't able to launch more, I was getting admission error. So, I cleaned up all the PODs, rebooted and restarted device plugin.
Following is the output of get pods on k8s. IPs in 10.55.xx.xx are given out by sriov-cni.
dkube@dkube-217:~$ kubectl get po -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE etcd-dkube-217 1/1 Running 17 15d 192.168.50.217 dkube-217 kube-apiserver-dkube-217 1/1 Running 18 15d 192.168.50.217 dkube-217 kube-controller-manager-dkube-217 1/1 Running 27 15d 192.168.50.217 dkube-217 kube-dns-86f4d74b45-tnbl8 1/3 CrashLoopBackOff 3988 9d 10.55.206.47 dkube-217 kube-proxy-ttcjv 1/1 Running 12 15d 192.168.50.228 smicro-228 kube-proxy-xk7z2 1/1 Running 17 15d 192.168.50.217 dkube-217 kube-scheduler-dkube-217 1/1 Running 18 15d 192.168.50.217 dkube-217 kube-sriov-cni-ds-installer-ktqp7 1/1 Running 4 1d 192.168.50.228 smicro-228 kube-sriov-cni-ds-installer-lhwqr 1/1 Running 5 1d 192.168.50.217 dkube-217 nvidia-device-plugin-daemonset-7cr2x 1/1 Running 8 10d 10.55.206.25 smicro-228 nvidia-device-plugin-daemonset-x6wb6 1/1 Running 12 10d 10.55.206.52 dkube-217 rdma-sriov-dp-ds-h9fmb 1/1 Running 0 3m 192.168.50.217 dkube-217 rdma-sriov-dp-ds-xjjn7 1/1 Running 0 3m 192.168.50.228 smicro-228
ok so some VFs move to other namespace and you restarted the device plugin. That the same issue we saw as well. I will look in to the code next week and provide a fix for that
Thanks. Could I expect a fix by early next week? We are blocked on RDMA experiments.
After deleting the pod, the new pod works fine.
# ibdev2netdev
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Up)
I don't know how to list VF devices...
[root@gpu28 ~]# find /sys/devices/ -name sriov_totalvfs
/sys/devices/pci0000:5d/0000:5d:02.0/0000:5f:00.0/0000:60:03.0/0000:61:00.0/sriov_totalvfs
/sys/devices/pci0000:5d/0000:5d:02.0/0000:5f:00.0/0000:60:03.0/0000:61:00.1/sriov_totalvfs
/sys/devices/pci0000:5d/0000:5d:02.0/0000:5f:00.0/0000:60:03.0/0000:61:00.2/sriov_totalvfs
/sys/devices/pci0000:5d/0000:5d:02.0/0000:5f:00.0/0000:60:03.0/0000:61:00.3/sriov_totalvfs
[root@gpu28 ~]# cat /sys/devices/pci0000:5d/0000:5d:02.0/0000:5f:00.0/0000:60:03.0/0000:61:00.0/sriov_totalvfs
32
So I think the total VF number is 32 * 4 = 128?
After deleting the pod, the new pod works fine.
I deleted the running PODs, deleted the device plugin daemon set and still not able to recover. Is there a way to recover from this situation? Please suggest.
You can try to reboot the server or reload the mlnx driver
Rebooting didn't help.
Do you see any errors in the dmesg? can you try to decrease the NUM_OF_VFS=16 with mft?
I can try that. However, this requires to enable SR-IOV for the entire cluster. I need more than 16 to bring up my application. It would be an academic excercise for me unless I can enable multus cni. I am waiting for documentation on that as we ran into issues integrating with multus.
Attached are dmesg logs for mlx. mlx_logs.txt
Reduced the VFs to 16. Problem persists.
2018/10/22 22:34:50 sriov device mode Configuring SRIOV on ndev= ens3f1 6 max_vfs = 16 cur_vfs = 16 vf = &{7 virtfn7 true false} vf = &{5 virtfn5 true false} vf = &{14 virtfn14 true false} vf = &{3 virtfn3 true false} vf = &{12 virtfn12 true false} vf = &{1 virtfn1 true false} vf = &{10 virtfn10 true false} vf = &{8 virtfn8 true false} vf = &{6 virtfn6 true false} vf = &{15 virtfn15 true false} vf = &{4 virtfn4 false false} Fail to config vfs for ndev = ens3f1 Fail to configure sriov; error = Link not found
Do you have a fix in the works?
@songole Maybe you can check if the VF are used by other processes?
@harryge00 No. I am running kubernetes on this node and nothing else.
@harryge00 after reinstalling kubernetes cluster - things started working. Deleting the jobs and rebooting the nodes didnt help, had to reinstall the k8s to recover.
@songole we should have fix soon. I am facing some issue with accessing repo on github.
@harryge00 @songole @moshe010 @mak-454 new fix is available at docker hub now. https://hub.docker.com/r/rdma/k8s-rdma-sriov-dev-plugin
Please delete the old container image and pull the new one for device plugin. Sometimes I have seen K8s is not able to pick up the new image and repeatedly fails. And in that case only option I found was to reconfigure, but may be fixed in latest k8s.
@harryge00 or @songole can you close the issue
We don't see the problem anymore with the fix. @moshe010 I don't think I have the permissions to close it.
I have followed the README to create configmap and daemonset, but it seems like the device plugin does not work correctly: