search

Xilinx / dma_ip_drivers

Xilinx QDMA IP Drivers
https://xilinx.github.io/dma_ip_drivers/
526 stars 398 forks source link

dma-perf can only get half of the bandwidth of a x8 PCIe Gen 5 card. Any suggestions? #273

Open hjiang2018 opened 3 months ago

hjiang2018 commented 3 months ago

Running dma-perf with c2h/h2c/bi directions on a x8 PCIe Gen 5 card, but we can only get to half (~14GB/s) of limitation of the card's bandwidth (~32GB/s in theory). Below is a table of config file contents and test results:

image

We also noticed that with the following register writing sequence, the bandwidth of dma-perf may reach ~26GB/s, with offset_ch1=0x60000000000, but we don't understand the register settings:

    # register      noc 0 and 1     plaxi 0 and 1
    [ '0x2c8',      '0x0',          '0x0'           ],
    [ '0x308',      '0x0',          '0x0'           ],
    [ '0x824',      '0xffffffff',   '0xffffffff'    ],
    [ '0x828',      '0xffffffff',   '0xffffffff'    ],
    [ '0x82C',      '0xffffffff',   '0xffffffff'    ],
    [ '0x830',      '0xffffffff',   '0xffffffff'    ],
    [ '0x834',      '0xffffffff',   '0xffffffff'    ],
    [ '0x838',      '0xffffffff',   '0xffffffff'    ],
    [ '0x83C',      '0xffffffff',   '0xffffffff'    ],
    [ '0x840',      '0xffffffff',   '0xffffffff'    ],
    [ '0x804',      '0x0',          '0x0'           ],
    [ '0x808',      '0x0',          '0x0'           ],
    [ '0x80C',      '0x00000000',   '0x00000000'    ],
    [ '0x810',      '0x0',          '0x9'           ],
    [ '0x814',      '0x0',          '0x0'           ],
    [ '0x818',      '0x00000000',   '0x00900000'    ],
    [ '0x81C',      '0x0',          '0x0'           ],
    [ '0x820',      '0x0',          '0x0'           ],
    [ '0x844',      '0x34',         '0x34'          ],
    [ '0x2c8',      '0x0',          '0x0'           ],
    [ '0x308',      '0x0',          '0x0'           ],
    [ '0x804',      '0x0',          '0x0'           ],
    [ '0x808',      '0x0',          '0x0'           ],
    [ '0x80C',      '0x40000000',   '0x40000000'    ],
    [ '0x810',      '0x0',          '0x9'           ],
    [ '0x814',      '0x0',          '0x0'           ],
    [ '0x818',      '0x00040000',   '0x00940000'    ],
    [ '0x81C',      '0x0',          '0x0'           ],
    [ '0x820',      '0x0',          '0x0'           ],
    [ '0x844',      '0xb4',         '0xb4'          ]]

name=mm_01 pci_bus=01 pci_device=00 mode=mm dir=c2h pf_range=0:0 q_range=0:31 wb_acc=5 tmw_idx=9 cntr_idx=7 trig_mode=cntr_tmr rngidx=0xF ram_width=15 #31bits - 2^31 = 2GB runtime=30 #sec num_threads=2 bidir_en=1 num_pkt=1 pkt_sz=16777216 mm_chnl=2 offset=0x00000000000 offset_ch1=0x60000000000 dump_en=0 marker_en=0