zhguanw-amd
commented
11 months ago Hi Xun Nie,
Thanks for your interests and the efforts on porting RecoNIC to VCU128!
From your description, it seems that the client can successfully send RDMA read request packet to the server and the server also can successfully generate RDMA read response packet to the client. The issue happens when the client copies data from the device memory to host memory for verification via the QDMA AXI-MM channel.
Could you reprogram the FPGA, run your read test again and dump RDMA registers? It would be helpful if you can post your dumped RDMA register here after you run read operation. Before you run the read test, could you also share your dmesg information related to onic-driver as well?
I guess the reason is the DDR4 capacity of VCU128 DDR4 is too small, only 4GB, even if its chip architecture is almost the same with AU280 , some operations make the memory boundary overflow, I would like to ask if RecoNIC has some limitations on the capacity of DDR？
I don't think this is related to DDR capacity. RecoNIC defines the device memory size at https://github.com/Xilinx/RecoNIC/blob/main/lib/reconic.h#L47, but this can be configurable, as long as we can instantiate more DDRs in the hardware. Probably you can check whether addresses of the AXI read transaction from host to the device memory after receiving RDMA read response packet at the client side via ILA are correct or not.
Thanks, Guanwen
niexun725
Dear Zhguanw, I am trying to port the project to two Virtex UltraScale + HBM VCU128 FPGA, the system version is Ubuntu 20.04 and linux kernel version 5.4.0-125-generic, and the vivado version is also 2021.2. Everything looks fine until I run the RDMA Read test，Host and Client successfully established a connection, but the Client shows Error : received data mismatched.
I tried to use ila to monitor the AXI4 interface of dev ddr4, and found that the host successfully wrote the data to the DDR4 to be read by the client, and finally the client also read the corresponding amount of data from its own DDR4.However, the client did not actually read the DDR4 of the Host successfully, because I did not see the read operation on the Host DDR4. By the way, the client shows "Successfully send an RDMA read operation"
I guess the reason is the DDR4 capacity of VCU128 DDR4 is too small, only 4GB, even if its chip architecture is almost the same with AU280 , some operations make the memory boundary overflow, I would like to ask if RecoNIC has some limitations on the capacity of DDR？I am learning this project, hoping to get your guidance. Thanks, Xun Nie