How to boot guest with multiple pci bridge

[bssrikanth]

I am trying to boot guest via QEMU which will be having 63 vnics [I am testing boundary conditions for SRIOV on Power].

Below is the issue which I am facing when I tried to passthrough >=28 VF/pci devices:

2016-11-03 04:17:23,642 utils_misc L0170 ERROR| Fail to create qemu command: 2016-11-03 04:17:23,643 utils_misc L0177 ERROR| Original Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 2534, in create self.devices = self.make_create_command() File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 1681, in make_create_command pci_bus=pci_bus) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 802, in add_pcidevice devices.insert(dev) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_devices/qcontainer.py", line 495, in insert % (details, device, devices)) DeviceError: Failed to insert device: device aid = None aobject = None parent_bus = ({'aobject': 'pci.0'},) child_bus = [] params: driver = vfio-pci host = 0005:01:03.7 <======= this is the 28th VF PCI device id = id_0005.01.03.7

Because: ParentBus({'aobject': 'pci.0'}): No free matching bus

Sometime back I had below code added to qemu_vm.py:

1556 if iov >=28: 1557 pci_bus = {'aobject': "pci." + str(iov/28)} 1558 add_pcidevice(devices, pci_id, params=nic_params, 1559 device_driver=device_driver, 1560 pci_bus=pci_bus)

Which was working fine for some time. But now this logic as well errors out with below error:

File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 2536, in create self.devices = self.make_create_command() File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 1683, in make_create_command pci_bus=pci_bus) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 802, in add_pcidevice devices.insert(dev) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_devices/qcontainer.py", line 495, in insert % (details, device, devices)) DeviceError: Failed to insert device: device aid = None aobject = None parent_bus = ({'aobject': 'pci.1'},) child_bus = [] params: driver = vfio-pci host = 0005:01:03.7 <======= this is the 28th VF PCI device id = id_0005.01.03.7

Because: ParentBus({'aobject': 'pci.1'}): No matching bus

I want to know how to add more pci parent buses depending on the number of pci devices which I am passing.

Appreciate any help here.

[bssrikanth]

I also tried passing below param in my config file:

    pci_bus += "pci.0 pci.1"

This results in below error:

2016-11-03 04:24:18,884 utils_misc L0170 ERROR| Fail to create qemu command: 2016-11-03 04:24:18,885 utils_misc L0177 ERROR| Original Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 2534, in create self.devices = self.make_create_command() File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 1502, in make_create_command devices.insert(dev) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_devices/qcontainer.py", line 495, in insert % (details, device, devices)) DeviceError: Failed to insert device: device aid = None aobject = usb1 parent_bus = ({'aobject': 'pci.0 pci.1'},) child_bus = [<virttest.qemu_devices.qbuses.QUSBBus object at 0x3fffa9af11d0>] params: driver = pci-ohci id = usb1

Because: ParentBus({'aobject': 'pci.0 pci.1'}): No matching bus

[xutian]

Maybe enable multi-function will helpful to attache so many device in a pci root bus.

[bssrikanth]

@xutian I reran tests by passing 'multifunction='on'' in my config file. Test seems to be failing with same error:

2016-11-04 00:33:45,527 utils_misc L0170 ERROR| Fail to create qemu command: 2016-11-04 00:33:45,528 utils_misc L0177 ERROR| Original Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 2534, in create self.devices = self.make_create_command() File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 1681, in make_create_command pci_bus=pci_bus) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_vm.py", line 802, in add_pcidevice devices.insert(dev) File "/usr/local/lib/python2.7/dist-packages/avocado_plugins_vt-39.0-py2.7.egg/virttest/qemu_devices/qcontainer.py", line 495, in insert % (details, device, devices)) DeviceError: Failed to insert device: device aid = None aobject = None parent_bus = ({'aobject': 'pci.0'},) child_bus = [] params: driver = vfio-pci host = 0005:01:03.7 id = id_0005.01.03.7 <========= 28th VF PCI device

Because: ParentBus({'aobject': 'pci.0'}): No free matching bus

[bssrikanth]

I see only usb device getting multifunction -> on

---------------< 03-00 >--------------- device aid = usb1 aobject = usb1 parent_bus = ({'aobject': 'pci.0'},) child_bus = [<virttest.qemu_devices.qbuses.QUSBBus object at 0x3fffab08c790>] params: driver = pci-ohci id = usb1 multifunction = on bus = pci.0 addr = 03

---------------< 11-00 >--------------- device aid = id_0005.01.02.0 aobject = None parent_bus = ({'aobject': 'pci.0'},) child_bus = [] params: driver = vfio-pci host = 0005:01:02.0 id = id_0005.01.02.0 bus = pci.0 addr = 11 .. .. ..

[bssrikanth]

Tried passing ' extra_params = " -device pci-bridge"' But no luck..

[bssrikanth]

Okies.. I kind of got it working by adding pci_controller params into my config file + qemu_vm code changes to switch to new parent pci bus when the pci device count reaches max allowed on the original pci bus.. need to make changes generic and test with more scenarios.

[xutian]

Hi @bssrikanth Is your problem solved? if yes, let close this issue. else I think you need add configurations like below in your test-case.cfg

pci_bus=  pci.0
pci_controllers += “ pci_br1”
type_pri_br1 = pci-bridge

for q35 machine we have a default pci-bridge support, see commit ff519d5f7c929f02043b938fe57364591e701e85

thanks, Xu

[xutian]

@bssrikanth we have meet same issue when boot guest with more than 18 pci-bridge, now I have send a patch to fix it, can you have a try, does it fixed your issue also. https://github.com/avocado-framework/avocado-vt/pull/898