USP PCIe x8 dma_test hanging

[smunaut]

When trying to use a x8 link width, the dma_test hangs. Everything seems to be fine with a x4 core.

So for the x4 core, I instanciate it with :

            self.submodules.pcie_phy = USPPCIEPHY(platform, platform.request("pcie_x4"),
                speed = "gen3",
                data_width = 128,
                bar0_size  = 0x20000)

And for x8 core, I do it like this :

            self.submodules.pcie_phy = USPPCIEPHY(platform, platform.request("pcie_x8"),
                speed = "gen3",
                data_width = 256,
                bar0_size  = 0x20000)

The litepcie module loads fine and detects the card :

[  134.322424] litepcie: loading out-of-tree module taints kernel.
[  134.322526] litepcie: module verification failed: signature and/or required key missing - tainting kernel
[  134.323251] litepcie 0000:01:00.0: \x1b[1m[Probing device]\x1b[0m
[  134.323274] litepcie 0000:01:00.0: enabling device (0000 -> 0002)
[  134.323495] litepcie 0000:01:00.0: Version LiteX SoC on Xilinx ADRV2CRR-FMC 2022-03-02 09:26:12
[  134.323570] litepcie 0000:01:00.0: 1 MSI IRQs allocated.
[  134.323603] litepcie 0000:01:00.0: Creating /dev/litepcie0

And the info and scratch_test also work, but not dma_test :

root@asuka /home/tnt # ./driver/user/litepcie_util info
FPGA identification: LiteX SoC on Xilinx ADRV2CRR-FMC 2022-03-02 09:26:12
root@asuka /home/tnt # ./driver/user/litepcie_util scratch_test
Write 0x12345678 to scratch register:
Read: 0x12345678
Write 0xdeadbeef to scratch register:
Read: 0xdeadbeef
root@asuka /home/tnt # ./driver/user/litepcie_util dma_test

[hangs forever...]

[smunaut]

I also just tried a x8 core with data_width=128, this builds but the resulting bitstream crashes the whole host when trying to load the litepcie.ko module.

[enjoy-digital]

@smunaut: I just tested the BCU-1525 with a similar configuration and got it working. If would be interesting to see if the PCIe hardblocks are similar or different on these devices and if a specific adaptation could be required for the Zynq Ultrascale + PCIe hardblock.

[smunaut]

Mmm, AFAICT the ZU11EG and the VU9P both have the PCIE4 block and so should be identical :/

[enjoy-digital]

Just in case, I was testing with a 200MHz sys_clk_freq (it should not make a difference, but could be worth a try).

[smunaut]

PEBCAK

While changing the GTH Quad and regenerating the .xci appropriately, Vivado changed the "Request Completion Straddle" option to "True" which the adaptation layer to standard TLP can't deal with.