Dual port SRIOV VF issue with NIC count

[bssrikanth]

Hi, I having been working on enabling SRIOV tests which are currently part of qemu tests on IBM PowerKVM. On IBM PowerKVM 3.1 [next GA] we support SRIOV on Mellanox Connectx cards. I have completed most of the patch work now, but not sure on one of the issue which I am facing requesting your suggestion.

Currently in qemu/libvirt tests in virttest/avocado-vt we assume that 'nics' param has 1-1 mapping with ports. That is if I have nic = "nic1" in my cfg file, it is assumed to have single port [which gets probed into guest as 1 iface interface].

But in case of Mellanox Connectx cards we have 2 Port Virtual Functions[VFs], I am having below in my configuration file: .. nics = "nic1" pci_assignable_nic1 = vf device_driver = vfio-pci .. so 1 VFIO is passthroughed to guest in my case, which has 2 ports [2 ifaces] and this cause issue in current code where virttest assumes 1 nic = 1 port.

File "/var/lib/libvirt/images/srikanth/fvt-pkvm-test/powerkvm-virt-test/virttest/utils_net.py", line 2941, in update_mac_ip_address
  vm.virtnet.set_mac_address(vlan, mac)
File "/var/lib/libvirt/images/srikanth/fvt-pkvm-test/powerkvm-virt-test/virttest/utils_net.py", line 2626, in set_mac_address
  nic = self[nic_index_or_name]
File "/var/lib/libvirt/images/srikanth/fvt-pkvm-test/powerkvm-virt-test/virttest/utils_net.py", line 2093, in __getitem__
  return super(VMNet, self).__getitem__(index_or_name)

IndexError: list index out of range

I am thinking for a solution. Welcome if any of you have suggestions.

[clebergnu]

@bssrikanth It looks like you're still running out of the deprecated virt-test repo, instead of from avocado-vt sources. Can you confirm that?

[lmr]

@bssrikanth already sent updated PRs to avocado-vt, so I'm assuming he has current code. Our good friends @ Beijing do VF testing, so I'm guessing there's some configuration he's missing. I don't know who is the best contact person right now, but maybe @suqinhuang can direct you to a contact.

[suqinhuang]

@luckyh @PyLearner test sr-iov before, did you meet it ?

[bssrikanth]

@suqinhuang @PyLearner @luckyh I have tested existing sriov tests [which assume igb driver] and it works with no issues, on x86, since it has 1 port per VF. But the issue arise where we try to run with mlx4_core driver which has 2 ports per VF, I am porting it to PowerKVM. It seems by default virttest/avocado-vt assumes 1 port per nic. This leads to issues pointed in initial description.

[bssrikanth]

@lmr yes. @clebergnu I have started using avocado-VT

[luckyh]

@bssrikanth Sorry, I'm not familiar with vfio/sriov related code, but I think we should handle vf things by respecting its pci address, not the iface name. So maybe it needs lots of changes..

[bssrikanth]

@luckyh we might also want to look at ifaces on guest as well to ensure that the VF are getting probed properly and getting IP addresses.

@lmr, @suqinhuang , @luckyh I have currently worked around the issue here by adding below:

1. introduced 'iface_per_pci' param : this would be a param which user can use when he has multi port nic [can be pf/vf].
2. as we initially set mac considering 'nics' count I have constrained by adding a check to avoid the 'IndexError: list index out of range' error

in virttest/utils_net.py:
         nics = params.get("nics")
-        nic_minimum = len(re.split(r"\s+", nics.strip()))
+       iface_per_pci = params.get("iface_per_pci",1)
+        nic_minimum = (len(re.split(r"\s+", nics.strip()))) * int(iface_per_pci)
+       logging.info("nic_minimum: %d",nic_minimum)
+       logging.info("len of macs_ips: %d",len(macs_ips))
         if len(macs_ips) == nic_minimum:
             break
         num += 1
         time.sleep(5)
     if len(macs_ips) < nic_minimum:
         logging.error("Not all nics get ip address")
-
+    niccount = 1
+    expected_count = len(re.split(r"\s+", nics.strip()))
     for (_ip, mac) in macs_ips:
         vlan = macs_ips.index((_ip, mac))
         # _ip, mac are in different sequence in Fedora and RHEL guest.
@@ -2932,6 +2948,9 @@ def update_mac_ip_address(vm, params, timeout=None):
             mac = mac.replace("-", ".")
         vm.address_cache[mac.lower()] = _ip
         vm.virtnet.set_mac_address(vlan, mac)
+       niccount = niccount + 1
+       if niccount > expected_count:
+           break;

Please let me know your views.

[bssrikanth]

please ignore the logging comments in above code, it's still getting tested. But I wanted your suggestions.

[bssrikanth]

Any thoughts on this one? the testing is done with above mentioned workaround.