Closed shroud404 closed 1 year ago
Nice work, I'll work on getting that cleaned up.
The BARs have to be configured after creating the IP core instance. In the Corundum testbenches, it is usually done like so:
self.dev.functions[0].configure_bar(0, 2**len(dut.core_pcie_inst.axil_ctrl_araddr), ext=True, prefetch=True)
if hasattr(dut.core_pcie_inst, 'pcie_app_ctrl'):
self.dev.functions[0].configure_bar(2, 2**len(dut.core_pcie_inst.axil_app_ctrl_araddr), ext=True, prefetch=True)
This will configure one or two BARs on function 0. If you want to configure BARs on all of the functions, then perhaps that should be wrapped in a loop or similar.
ohh, I forgot this. Thank u
Please try out the git master version of cocotbext-pcie and let me know if that fixes the enumeration issues.
Ok, let’s try.
shroud404@linux:~$ pip show cocotbext-pcie
Name: cocotbext-pcie
Version: 0.2.12
Summary: PCI express simulation framework for cocotb
Home-page: https://github.com/alexforencich/cocotbext-pcie
Author: Alex Forencich
Author-email: alex@alexforencich.com
License: MIT
Location: /home/shroud404/.local/lib/python3.9/site-packages
Requires: cocotb, cocotbext-axi
Required-by:
And corundum is the latest.
commit c708bc45cd0d5b44274d238dd9fadd16335f337d (HEAD -> master, origin/master, origin/HEAD)
Author: Alex Forencich <alex@alexforencich.com>
Date: Tue Dec 6 17:23:28 2022 -0800
fpga/mqnic/fb2CG: Update testbench
I will test with VCU118 100G tb. I changed the test_fpga_core.py, to make it has an UltraScalePlusPcieDevice with two PF. The pf1 is the same as PF0, with the same msix table set. Set the configuration of PF1 BAR to be the same as that of PF0.
I add some log info when make_functions
like this:
usp_model.py line 754
self.make_function()
self.log.info("EP Device make function[0] done")
self.log.info("EP Device function[0] first is multifunction_device:%s", self.functions[0].multifunction_device)
Use script
to capture the typescript when running the test.
Let's see: first, set EP
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] Xilinx UltraScale+ PCIe hard IP core configuration:
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] PCIe speed: gen 3
......
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] Enable RC 4 TLP straddling: True
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] PF count: 2
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] Current append_function pcie_id is 00:00.0
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] EP Device make function[0] done
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] EP Device function[0] first is multifunction_device:False
0.00ns INFO ..raScalePlusPcieFunction[00:00.0] Assigned PCIe ID 00:00.1
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] Current append_function pcie_id is 00:00.1
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] EP evice make function[1] done
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] EP Device function[0] second is multifunction_device:True
0.00ns INFO ..cie.UltraScalePlusPcieDevice[00] EP Device function[1] second is multifunction_device:True
next Enumerating
208.00ns INFO cocotb.pcie.RootComplex Enumerating bus 1 device 0
.......
219.27ns INFO ..raScalePlusPcieFunction[00:00.0] Assigned PCIe ID 01:00.0
219.27ns INFO ..raScalePlusPcieFunction[00:00.1] Assigned PCIe ID 01:00.1
230.79ns INFO cocotb.pcie.RootComplex Found device at 01:00.0
........
299.17ns INFO cocotb.pcie.RootComplex Found function at 01:00.0
...
435.92ns INFO cocotb.pcie.RootComplex Enumerate function 01:00.0 BAR0
...
527.09ns INFO cocotb.pcie.RootComplex pci 01:00.0: Mem BAR0 (64-bit) raw: 0xffffffffff00000c, mask: 0x0000000000ffffff, size: 16777216
.......
1347.60ns INFO cocotb.pcie.RootComplex Found function at 01:00.1
...
1484.35ns INFO cocotb.pcie.RootComplex Enumerate function 01:00.1 BAR0
...
1575.52ns INFO cocotb.pcie.RootComplex pci 01:00.1: Mem BAR0 (64-bit) raw: 0xffffffffff00000c, mask: 0x0000000000ffffff, size: 16777216
.......
3161.56ns INFO cocotb.pcie.RootComplex Configure function 01:00.0 BAR0
3207.15ns INFO cocotb.pcie.RootComplex Configure function 01:00.1 BAR0
......
3252.73ns INFO cocotb.pcie.RootComplex Enumeration complete
3252.73ns INFO cocotb.pcie.RootComplex Device tree:
[00-01]---01.0-[01]-+-00.0
\-00.1
OK, it is work!
Excellent. I'll cut a release with these fixes relatively soon, let me know if you find any other bugs.
I found some problems when I wanted to use it to simulate PCIE models with multi-PF. There is a problem when scanning for devices and functions in pci.py. First, in line 145, it should be
dev.pcie_id = PcieId(self.bus_num, d, f)
instead ofdev.pcie_id = PcieId(self.bus_num, d, 0)
. Second, in line 152, the dev.header_type has changed inawit dev.setup
fromval
toval & 0x7f
. So the judgment condition of if should beif dev.multifunction == 0:
. Also, there is a mistake in device.py line 93:f.multifunction_devlce = True
should bef.multifunction_device = True
. Last, I want to know how to set/change an EP's BAR count/BAR size. After I fixed the above issue, I set up anUltraScalePlusPcieDevice
to have two PFs. Like this:RC successfully enumerates two PFs, Device tree is: [00-01]---01.0-[01]-+-00.0/-00.1 Scanned to the BAR of PF0: pci 01:00.0: Mem BAR0 (64-bit) raw: 0xffffffffff00000c, mask: 0x0000000000ffffff, size: 16777216. But the BAR of PF1 is completely empty, I printed the register read: Config type 0 read, reg 0x004 print current dev 01:00.1 BAR 0 register is 00 Why PF0 with a right BAR set, and PF1 with no bar set?