EAL: Expecting 'PA' IOVA mode but current mode is 'VA'

[Pesa]

This happens while running ndndpdk-ctrl create-eth-port on an Intel X710.

{"level":"error","ts":1654188163.6431565,"logger":"eal","msg":"rte_dev_probe error","addr":"0000:44:00.3","args":"","error":"22 Invalid argument"}
{"level":"error","ts":1654188163.6432307,"logger":"DPDK","msg":"EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing"}
{"level":"error","ts":1654188163.643283,"logger":"DPDK","msg":"EAL: Driver cannot attach the device (0000:44:00.3)"}
{"level":"error","ts":1654188163.6433094,"logger":"DPDK","msg":"EAL: Failed to attach device on primary process"}

Originally reported by @sankalpatimilsina12.

Subsequent troubleshooting discovered that starting the forwarder with --iova-mode pa (pass it via eal.extraFlags) fixes the issue and allows port creation to succeed. I suggest mentioning this somewhere in the documentation.

dpdk-devbind.py -s

``` Network devices using DPDK-compatible driver ============================================ 0000:44:00.3 'Ethernet Controller X710 for 10 Gigabit SFP+ 104e' drv=igb_uio unused=i40e,vfio-pci Network devices using kernel driver =================================== 0000:44:00.0 'Ethernet Controller X710 for 10GBASE-T 15ff' if=enp68s0f0 drv=i40e unused=igb_uio,vfio-pci *Active* 0000:44:00.1 'Ethernet Controller X710 for 10GBASE-T 15ff' if=enp68s0f1 drv=i40e unused=igb_uio,vfio-pci 0000:44:00.2 'Ethernet Controller X710 for 10 Gigabit SFP+ 104e' if=enp68s0f2 drv=i40e unused=igb_uio,vfio-pci 0000:c1:00.0 'MT2892 Family [ConnectX-6 Dx] 101d' if=enp193s0f0 drv=mlx5_core unused=igb_uio,vfio-pci *Active* 0000:c1:00.1 'MT2892 Family [ConnectX-6 Dx] 101d' if=enp193s0f1 drv=mlx5_core unused=igb_uio,vfio-pci ```

[yoursunny]

I knew this is going to happen sooner or later. Some driver wants VA and some wants PA.

Let me retest whether setting PA mode would cause undesirable side effects. If it doesn't cause other problems, I'd create a separate field (e.g., .eal.iovaMode) and not ask operator to use .eal.extraFlags.

[Pesa]

Let me retest whether setting PA mode would cause undesirable side effects.

From my (very limited) understanding, there was never a choice for the igb_uio driver, since it only supports PA mode. Which means that everyone that ever used an Intel NIC in the past was implicitly using PA mode.

It seems that the modern/preferred alternative is vfio-pci, which supports both PA and VA modes. Is there anything that prevents us from using vfio-pci with ndn-dpdk?

[yoursunny]

I checked my devbox and it defaults to PA, so that PA would not cause any problem. I've added the .eal.iovaMode parameter that can be specified as needed, but the default is still auto-detect. In docs/hardware.md Intel section, it's noted that igb_uio requires PA.

[Pesa]

It seems that the modern/preferred alternative is vfio-pci, which supports both PA and VA modes. Is there anything that prevents us from using vfio-pci with ndn-dpdk?

What about this? DPDK recommends using vfio-pci. Quoting from https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

It is recommended that vfio-pci be used as the kernel module for DPDK-bound ports in all cases. If an IOMMU is unavailable, the vfio-pci can be used in no-iommu mode.

[yoursunny]

Nah, I never got vfio-pci working on Intel cards, but that's more than a year ago. I'll try again soon.

[yoursunny]

I've tested vfio-pci with Intel X710. It works for both physical function and virtual function. Thus, docs/hardware.md is updated to recommend vfio-pci instead of igb_uio.