Closed flymark2010 closed 6 years ago
@flymark2010 , when plugin is deleted, those vhca interfaces will exist. However when you restart the plugin, you shouldn't see this error.
It is likely a driver issue which failed to initialize the Vfs. Can you please share your /var/log/messages driver log or /var/log/kern.log based on centos/ubuntu respectively?
I restarted the Pod of the plugin, and catched log from /var/log/kern.log like below:
Jul 23 09:42:20 CQ-YJY-10-128-2-17 kernel: [345967.237844] aufs au_opts_verify:1597:dockerd[2537]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 23 09:42:20 CQ-YJY-10-128-2-17 kernel: [345967.283161] aufs au_opts_verify:1597:dockerd[11730]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 23 09:42:20 CQ-YJY-10-128-2-17 kernel: [345967.333785] aufs au_opts_verify:1597:dockerd[2442]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 23 09:42:20 CQ-YJY-10-128-2-17 kernel: [345967.507524] aufs au_opts_verify:1597:dockerd[11730]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 23 09:42:20 CQ-YJY-10-128-2-17 kernel: [345967.544930] aufs au_opts_verify:1597:dockerd[11730]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 23 09:42:20 CQ-YJY-10-128-2-17 kernel: [345967.573501] aufs au_opts_verify:1597:dockerd[11730]: dirperm1 breaks the protection by the permission bits on the lower branch
And the Pod log is the same with previous.
The OS and kernel are respectively:
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
# uname -r
4.4.0-87-generic
I think there's something related with the kernel config intel_iommu
. I setted intel_iommu=off
, then reboot and restart the plugin. Then the plugin runs normally, and miraculous the GPU P2P communication is OK, too(with intel_iommu=on
, the GPU P2P communication will hang). The OFED driver tutorial mentioned that we should set intel_iommu=on
. So I'm wondered if the setting intel_iommu=off
will affect the RDMA SRIOV plugin use?
@flymark2010 if you are using VMs than you likely need intel_iommu=on. But otherwise for using VFs in containers, you don't need it on because there is single kernel allocating and programming hw addresses in the NIC/HCA. GPU P2P might have some issues with iommu, I don't recall the details anymore as PCIe addressing gets complicated with iommu being on. So intel_iomu=off is fine for plugin, it won't affect.
Ok, thanks!
Hi, I have started a new node and create the sriov device plugin, everythin goes fine. Then I delete the plugin and then to create it again, but failed.
Here is the Pod log on that node:
but the network interface
ens5f0
is actually exist. Here is theifconfig
result:Beside
ens5f0
andens5f1
, other network interface are created by the plugin, and even when I deleted the plugin, they are still exists.