search

alexforencich / cocotbext-pcie

PCI express simulation framework for Cocotb
MIT License
136 stars 44 forks source link

[Question] Ways to use the BFM functionality #7

Closed thrakkor closed 3 years ago

thrakkor commented 3 years ago

We all know the Xilinx PCIe Root Port BFM sucks beyond basic PIO Reads/Writes to an EP, and the Altera/Intel RP BFM is a bit better. I have also experimented with the DrivExpress approach (Python), which I liked for the most part (PIO reads not returning the read data was a huge hole for me). But it is discontinued anyway.

So I stumbled across this repo and cocotb. It seems very promising to me.

I am currently simulating a very complex design in a KU115 device (using Questa64). I would really like a more flexible root port BFM so I can better emulate the software side of the real system my FPGA is part of. I use a custom EP with the Xilinx HIP (streaming desc intf). I need to be able to read and write to the EP BAR space (single DW MemWr/Rd transfers aka PIO) from the RP as well as initiate block read and writes from the EP to/from RP memory. Detecting MSI events is a must (I allocate 32 MSI numbers) (even if it just monitoring a mem location in the RP like Altera does). Being able to setup Max Payload size and Max Read Req size to match the real HW system is also a must (Xilinx BFM can't do this). I am currently using the PIPE interface.

I see some possible approaches:

  1. completely replace the Xilinx RP BFM with something else (using PIPE interface)
  2. completely replace the Xilinx RP BFM with something else (using XCVR interface)
  3. replace the Xilinx user app com, cfg, rx and tx modules with something from this repo that connects to the CQ, RQ, CC and RC interfaces, while leaving the rest of the RP as is.

I realize 1 and 2 above could be addressed with various commercial solutions. Not sure that is currently a possibility for me (recent questa mixed lang license wasn't cheap).

Which leads to my actual questions. Which of the above does cocotbext-pcie address? I have some ideas to this answer based upon looking around the web and in your repos, but would like some direction. If I bring the RP CQ/RQ/CC/RC interfaces up to the top level (would be a wrapper around my testbench essentially), could I interface to those buses with the cocotbext-pcie BFM/tools?

I apologize if this is not the right place for this question. Please let me know if I need to post or ask this elsewhere.

thanks!

alexforencich commented 3 years ago

The idea behind the PCIe hard IP core interface modules in this repo is that you connect them to your design in place of the hard IP core instance itself. So you would have your user logic that has CQ/CC/RQ/RC streaming interfaces (HDL) -> Xilinx US+ PCIe HIP model in this repo (Python) -> root complex model in this repo (Python) -> your testbench/driver model code (Python). Take a look at how it's used in https://github.com/corundum/corundum as that's the most comprehensive example at the moment.

MMIO, DMA, P2P DMA, and MSI interrupts are all currently supported, and you can absolutely set the max read request and max payload size to whatever you want. And MSI interrupts are properly caught and you can handle them either as events or callbacks.

Eventually I'm planning on supporting PIPE so you can use this library to talk to the actual hard IP core model, but it will be some time before that's possible. I have to implement quite a bit more of the PCIe protocol stack before that will be possible.

I also just realized that I forgot to turn on discussions on this repo; that's now active.