search

Xilinx / XRT

Run Time for AIE and FPGA based platforms
https://xilinx.github.io/XRT
Other
559 stars 476 forks source link

DMA & mem-bw validation failed with Alveo U280 #6787

Open xsun2001 opened 2 years ago

xsun2001 commented 2 years ago

Our brand new alveo u280 card failed to pass the validation. Detailed information & outputs are listed below. I have noticed that #6104 #6105 are similar to my issue. However I cannot find a solution for it. I will try to disable IOMMU and report the situation later.

Validate Device : [0000:81:00.1] Platform : xilinx_u280_xdma_201920_3 SC Version : 4.3.15 Platform ID : 0x5e278820

Test 1 [0000:81:00.1] : dma Description : Run dma test Details : Buffer size - '16 MB' Error(s) : DMA failed: Input/output error Test Status : [FAILED]

Validation failed

- mem-bw failure:
```bash
$ sudo /opt/xilinx/xrt/bin/xbutil validate -d 0000:81:00.1 -r mem-bw --verbose
Verbose: Enabling Verbosity
Starting validation for 1 devices

Validate Device           : [0000:81:00.1]
    Platform              : xilinx_u280_xdma_201920_3
    SC Version            : 4.3.15
    Platform ID           : 0x5e278820
-------------------------------------------------------------------------------
Test 1 [0000:81:00.1]     : mem-bw 
    Description           : Run 'bandwidth kernel' and check the throughput
    Xclbin                : /opt/xilinx/xsa/xilinx_u280_xdma_201920_3/test
    Testcase              : /opt/xilinx/xrt/test/23_bandwidth.py
    Error(s)              : Host buffer alignment 4096 bytes
                            Compiled kernel =
                            /opt/xilinx/xsa/xilinx_u280_xdma_201920_3/test/bandwidth.xclbin
                            unable to sync BO: Input/output error
                            FAILED TEST

    Test Status           : [FAILED]
-------------------------------------------------------------------------------
Validation failed

XRT Version : 2.13.0 Branch : Hash : Hash Date : 2022-06-20 13:51:55 XOCL : 2.13.0, XCLMGMT : 2.13.0,

Devices present BDF : Shell Platform UUID Device ID Device Ready*
[0000:81:00.1] : xilinx_u280_xdma_201920_3 0x5e278820 user(inst=128) Yes

keryell commented 2 years ago

I have noticed on my U50 that I cannot pass the test with more than 4KiB. Can you try with adding --run dma --param dma:block-size:4096?

AchingSoul000 commented 2 years ago

I have noticed on my U50 that I cannot pass the test with more than 4KiB. Can you try with adding --run dma --param dma:block-size:4096?

Description : Run dma test Details : Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2977.6 MB/s Host <- PCIe <- FPGA read bandwidth = 2344.4 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2963.6 MB/s Host <- PCIe <- FPGA read bandwidth = 2752.6 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2974.9 MB/s Host <- PCIe <- FPGA read bandwidth = 2459.7 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2989.0 MB/s Host <- PCIe <- FPGA read bandwidth = 3114.1 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2963.1 MB/s Host <- PCIe <- FPGA read bandwidth = 3227.1 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2979.5 MB/s Host <- PCIe <- FPGA read bandwidth = 3013.7 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2963.8 MB/s Host <- PCIe <- FPGA read bandwidth = 3220.1 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2964.3 MB/s Host <- PCIe <- FPGA read bandwidth = 3036.2 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2975.0 MB/s Host <- PCIe <- FPGA read bandwidth = 3098.9 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2976.5 MB/s Host <- PCIe <- FPGA read bandwidth = 2959.2 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2968.3 MB/s Host <- PCIe <- FPGA read bandwidth = 3080.4 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2956.0 MB/s Host <- PCIe <- FPGA read bandwidth = 2833.8 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2958.7 MB/s Host <- PCIe <- FPGA read bandwidth = 3214.0 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2962.7 MB/s Host <- PCIe <- FPGA read bandwidth = 2872.4 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2975.0 MB/s Host <- PCIe <- FPGA read bandwidth = 2017.2 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2996.8 MB/s Host <- PCIe <- FPGA read bandwidth = 3164.7 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2936.2 MB/s Host <- PCIe <- FPGA read bandwidth = 2769.3 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2961.3 MB/s Host <- PCIe <- FPGA read bandwidth = 2496.0 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2968.3 MB/s Host <- PCIe <- FPGA read bandwidth = 2445.5 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2704.1 MB/s Host <- PCIe <- FPGA read bandwidth = 2168.6 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2890.4 MB/s Host <- PCIe <- FPGA read bandwidth = 2372.5 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2932.8 MB/s Host <- PCIe <- FPGA read bandwidth = 2555.7 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2961.8 MB/s Host <- PCIe <- FPGA read bandwidth = 3072.6 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2899.6 MB/s Host <- PCIe <- FPGA read bandwidth = 2440.3 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2963.2 MB/s Host <- PCIe <- FPGA read bandwidth = 3221.8 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2901.9 MB/s Host <- PCIe <- FPGA read bandwidth = 2317.8 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2959.2 MB/s Host <- PCIe <- FPGA read bandwidth = 2417.3 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2964.9 MB/s Host <- PCIe <- FPGA read bandwidth = 3211.8 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2950.7 MB/s Host <- PCIe <- FPGA read bandwidth = 3103.7 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2979.0 MB/s Host <- PCIe <- FPGA read bandwidth = 2164.4 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2961.5 MB/s Host <- PCIe <- FPGA read bandwidth = 2386.5 MB/s Buffer size - '40960 Byte' Host -> PCIe -> FPGA write bandwidth = 2826.0 MB/s Host <- PCIe <- FPGA read bandwidth = 2901.6 MB/s Test Status : [PASSED] Validation completed

It seems that the larger than 4096, the transfer speed will be faster.

xsun2001 commented 2 years ago

Thanks guys. I can confirm that IOMMU caused this issue. In my opinion, this incompatibility should be noted in getting started guide or other obvious place.

keryell commented 2 years ago

In my opinion, this incompatibility should be fixed. :-)