Attach port failed on dpdk mode?

[bettswang]

Hi, everyone

I just finished compiling ovs with dpdk. But I meet a issue when I want to attach a port into the ovs.

root@ubuntu2004:/usr/src/dpdk-stable-22.11.2/usertools# dpdk-devbind.py --status Network devices using kernel driver 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens32 drv=e1000 unused=vfio-pci *Active* 0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens33 drv=e1000 unused=vfio-pci *Active*

root@ubuntu2004:/usr/src/dpdk-stable-22.11.2/usertools# ovs-vsctl get Open_vSwitch . dpdk_initialized true

root@ubuntu2004:/usr/src/dpdk-stable-22.11.2/usertools# ovs-vsctl get Open_vSwitch . dpdk_version "DPDK 22.11.2"

root@ubuntu2004:/usr/src/dpdk-stable-22.11.2/usertools# ovs-vsctl add-port br0 ens33 -- set Interface ens33 type=dpdk options:dpdk-devargs=0000:02:01.0 ovs-vsctl: Error detected while setting up 'ens33': Error attaching device '0000:02:01.0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch".

2023-07-31T10:37:41.661Z|00067|dpdk|ERR|Invalid port_id=32 2023-07-31T13:59:56.054Z|00068|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:01.0) 2023-07-31T13:59:56.054Z|00069|dpdk|ERR|EAL: Failed to attach device on primary process 2023-07-31T13:59:56.054Z|00070|netdev_dpdk|WARN|Error attaching device '0000:02:01.0' to DPDK 2023-07-31T13:59:56.054Z|00071|netdev|WARN|ens33: could not set configuration (Invalid argument) 2023-07-31T13:59:56.054Z|00072|dpdk|ERR|Invalid port_id=32 2023-07-31T14:05:53.951Z|00073|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:01.0) 2023-07-31T14:05:53.951Z|00074|dpdk|ERR|EAL: Failed to attach device on primary process 2023-07-31T14:05:53.951Z|00075|netdev_dpdk|WARN|Error attaching device '0000:02:01.0' to DPDK 2023-07-31T14:05:53.951Z|00076|netdev|WARN|ens33: could not set configuration (Invalid argument) 2023-07-31T14:05:53.951Z|00077|dpdk|ERR|Invalid port_id=32

[igsilya]

As you can see, all the devices are attached to a kernel driver, you forgot to re-bind them.

[bettswang]

Yes, I change it. But it also failed.

root@ubuntu2004:~# dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' drv=vfio-pci unused=e1000
Network devices using kernel driver
===================================
0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=ens32 drv=e1000 unused=vfio-pci *Active*

2023-08-01T10:31:03.950Z|00100|dpdk|ERR|EAL: 0000:02:01.0 VFIO group is not viable! Not all devices in IOMMU group bound to VFIO or unbound
2023-08-01T10:31:03.950Z|00101|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:01.0)
2023-08-01T10:31:03.950Z|00102|dpdk|ERR|EAL: Failed to attach device on primary process
2023-08-01T10:31:03.950Z|00103|netdev_dpdk|WARN|Error attaching device '0000:02:01.0' to DPDK
2023-08-01T10:31:03.950Z|00104|netdev|WARN|ens33: could not set configuration (Invalid argument)
2023-08-01T10:31:03.950Z|00105|dpdk|ERR|Invalid port_id=32

[igsilya]

Log tells you exactly what is wrong: VFIO group is not viable! Not all devices in IOMMU group bound to VFIO or unbound Re-bind the second port as well. If the VFIO group will still be not viable after that, you'll need to re-configure your IOMMU on a system, or unbind all other PCI devices that are in the same VFIO group.

[bettswang]

yes, I had tried to bind the other nic to dpdk, but it failed too. What do you mean re-configrue my IOMMU, BIOS or Grub?

[bettswang]

[igsilya]

Try following instructions in the documentation: https://docs.openvswitch.org/en/latest/intro/install/dpdk/#setup-dpdk-devices-using-vfio

[bettswang]

Try following instructions in the documentation: https://docs.openvswitch.org/en/latest/intro/install/dpdk/#setup-dpdk-devices-using-vfio

That is what I had done before. I had configed iommu in grub and bios. This machine is a vm. Is it related with the issue?

[igsilya]

Since you're running in a VM, you need to have a virtual IOMMU device in your VM. Or you need to enable unsafe no-IOMMU mode for vfio-pci driver. See DPDK documentation: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

Alternatively you may just switch from vfio to uio instead.

[bettswang]

I have find some solution ：https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html They said：

So, the solutions are these two:
1. Install the device into a different slot
2. Bypass ACS using the ACS overrides patch

But I am a vm, I cloudn't change slot. So I change the vm file .vmx. It works. ethernet1.virtualDev = "e1000e"

root@ubuntu2004:~# ovs-vsctl add-port  br0  ens160 -- set Interface ens160 type=dpdk options:dpdk-devargs=0000:03:00.0
root@ubuntu2004:~# ovs-vsctl show
540a4a4f-44bb-4fc5-bc34-d815b8d985f8
    Bridge br0
        datapath_type: netdev
        Port ens160
            Interface ens160
                type: dpdk
                options: {dpdk-devargs="0000:03:00.0"}
        Port br0
            Interface br0
                type: internal
    ovs_version: "3.2.90"
root@ubuntu2004:~#

Thank you for your kindly help.