alexforencich / cocotbext-pcie

PCI express simulation framework for Cocotb
MIT License
139 stars 47 forks source link

Some issues with multi-PF support #14

Closed shroud404 closed 1 year ago

shroud404 commented 1 year ago

I found some problems when I wanted to use it to simulate PCIE models with multi-PF. There is a problem when scanning for devices and functions in pci.py. First, in line 145, it should be dev.pcie_id = PcieId(self.bus_num, d, f) instead of dev.pcie_id = PcieId(self.bus_num, d, 0). Second, in line 152, the dev.header_type has changed in awit dev.setup from val to val & 0x7f. So the judgment condition of if should be if dev.multifunction == 0:. Also, there is a mistake in device.py line 93: f.multifunction_devlce = True should be f.multifunction_device = True. Last, I want to know how to set/change an EP's BAR count/BAR size. After I fixed the above issue, I set up an UltraScalePlusPcieDevice to have two PFs. Like this:

        self.dev = UltraScalePlusPcieDevice(
            # configuration options
            pcie_generation=3,
            pcie_link_width=16,
            user_clk_frequency=250e6,
            alignment="dword",
            pf_count=2,
            .......

RC successfully enumerates two PFs, Device tree is: [00-01]---01.0-[01]-+-00.0/-00.1 Scanned to the BAR of PF0: pci 01:00.0: Mem BAR0 (64-bit) raw: 0xffffffffff00000c, mask: 0x0000000000ffffff, size: 16777216. But the BAR of PF1 is completely empty, I printed the register read: Config type 0 read, reg 0x004 print current dev 01:00.1 BAR 0 register is 00 Why PF0 with a right BAR set, and PF1 with no bar set?

alexforencich commented 1 year ago

Nice work, I'll work on getting that cleaned up.

The BARs have to be configured after creating the IP core instance. In the Corundum testbenches, it is usually done like so:

    self.dev.functions[0].configure_bar(0, 2**len(dut.core_pcie_inst.axil_ctrl_araddr), ext=True, prefetch=True)
    if hasattr(dut.core_pcie_inst, 'pcie_app_ctrl'):
        self.dev.functions[0].configure_bar(2, 2**len(dut.core_pcie_inst.axil_app_ctrl_araddr), ext=True, prefetch=True)

This will configure one or two BARs on function 0. If you want to configure BARs on all of the functions, then perhaps that should be wrapped in a loop or similar.

shroud404 commented 1 year ago

ohh, I forgot this. Thank u

alexforencich commented 1 year ago

Please try out the git master version of cocotbext-pcie and let me know if that fixes the enumeration issues.

shroud404 commented 1 year ago

Ok, let’s try.

shroud404@linux:~$ pip show cocotbext-pcie
Name: cocotbext-pcie
Version: 0.2.12
Summary: PCI express simulation framework for cocotb
Home-page: https://github.com/alexforencich/cocotbext-pcie
Author: Alex Forencich
Author-email: alex@alexforencich.com
License: MIT
Location: /home/shroud404/.local/lib/python3.9/site-packages
Requires: cocotb, cocotbext-axi
Required-by:

And corundum is the latest.

commit c708bc45cd0d5b44274d238dd9fadd16335f337d (HEAD -> master, origin/master, origin/HEAD)
Author: Alex Forencich <alex@alexforencich.com>
Date:   Tue Dec 6 17:23:28 2022 -0800

    fpga/mqnic/fb2CG: Update testbench

I will test with VCU118 100G tb. I changed the test_fpga_core.py, to make it has an UltraScalePlusPcieDevice with two PF. The pf1 is the same as PF0, with the same msix table set. Set the configuration of PF1 BAR to be the same as that of PF0.

I add some log info when make_functions like this:

        usp_model.py line 754
        self.make_function()
        self.log.info("EP Device make function[0] done")
        self.log.info("EP Device function[0] first is multifunction_device:%s", self.functions[0].multifunction_device)

Use script to capture the typescript when running the test.

Let's see: first, set EP

     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] Xilinx UltraScale+ PCIe hard IP core configuration:
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00]   PCIe speed: gen 3
     ......
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00]   Enable RC 4 TLP straddling: True
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00]   PF count: 2
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] Current append_function pcie_id is 00:00.0
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] EP Device make function[0] done
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] EP Device function[0] first is multifunction_device:False
     0.00ns INFO     ..raScalePlusPcieFunction[00:00.0] Assigned PCIe ID 00:00.1
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] Current append_function pcie_id is 00:00.1
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] EP evice make function[1] done
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] EP Device function[0] second is multifunction_device:True
     0.00ns INFO     ..cie.UltraScalePlusPcieDevice[00] EP Device function[1] second is multifunction_device:True

next Enumerating

   208.00ns INFO     cocotb.pcie.RootComplex            Enumerating bus 1 device 0
   .......
   219.27ns INFO     ..raScalePlusPcieFunction[00:00.0] Assigned PCIe ID 01:00.0
   219.27ns INFO     ..raScalePlusPcieFunction[00:00.1] Assigned PCIe ID 01:00.1
   230.79ns INFO     cocotb.pcie.RootComplex            Found device at 01:00.0
   ........
   299.17ns INFO     cocotb.pcie.RootComplex            Found function at 01:00.0
   ...
   435.92ns INFO     cocotb.pcie.RootComplex            Enumerate function 01:00.0 BAR0
   ...
   527.09ns INFO     cocotb.pcie.RootComplex            pci 01:00.0: Mem BAR0 (64-bit) raw: 0xffffffffff00000c, mask: 0x0000000000ffffff, size: 16777216
   .......
  1347.60ns INFO     cocotb.pcie.RootComplex            Found function at 01:00.1
  ...
  1484.35ns INFO     cocotb.pcie.RootComplex            Enumerate function 01:00.1 BAR0
  ...
  1575.52ns INFO     cocotb.pcie.RootComplex            pci 01:00.1: Mem BAR0 (64-bit) raw: 0xffffffffff00000c, mask: 0x0000000000ffffff, size: 16777216
  .......
  3161.56ns INFO     cocotb.pcie.RootComplex            Configure function 01:00.0 BAR0
  3207.15ns INFO     cocotb.pcie.RootComplex            Configure function 01:00.1 BAR0
  ......
  3252.73ns INFO     cocotb.pcie.RootComplex            Enumeration complete
  3252.73ns INFO     cocotb.pcie.RootComplex            Device tree: 
                                                        [00-01]---01.0-[01]-+-00.0
                                                                            \-00.1

OK, it is work!

alexforencich commented 1 year ago

Excellent. I'll cut a release with these fixes relatively soon, let me know if you find any other bugs.