host/streaming_host_bandwidth non-blocking fails and hangs

Hi,

As I am trying the host-kernel streaming examples, I found that make run TARGET=hw DEVICE=xilinx_u200_qdma_201920_1 in host/streaming_host_bandwidth fails and hangs with the following output:

./streaming_host_bandwidth ./build_dir.hw.xilinx_u200_qdma_201920_1/krnl_stream_adder1.xclbin
Vector Increment of elements 0x10000000 by 1 
Found Platform
Platform Name: Xilinx
INFO: Reading ./build_dir.hw.xilinx_u200_qdma_201920_1/krnl_stream_adder1.xclbin
Loading: './build_dir.hw.xilinx_u200_qdma_201920_1/krnl_stream_adder1.xclbin'
Trying to program device[0]: xilinx_u280_xdma_201920_3
XRT build version: 2.8.726
Build hash: 7c93966ead2dec777b92bdc379893f22b5bd561e
Build date: 2020-11-11 20:29:19
Git branch: 2020.2
PID: 1234
UID: 1000
[Mon Dec 21 06:59:04 2020 GMT]
HOST: foo
EXE: /path/to/Vitis_Accel_Examples/host/streaming_host_bandwidth/streaming_host_bandwidth
[XRT] ERROR: Xclbin does not match shell on card.
[XRT] ERROR: Shell VBNV is 'xilinx_u280_xdma_201920_3'
[XRT] ERROR: Xclbin VBNV is 'xilinx_u200_qdma_201920_1'
[XRT] ERROR: Use 'xbmgmt flash' to update shell.
[XRT] ERROR: See dmesg log for details. err=-95
[XRT] ERROR: Failed to load xclbin.
Failed to program device[0] with xclbin file!
Trying to program device[1]: xilinx_u250_xdma_201830_2
[XRT] ERROR: Xclbin does not match shell on card.
[XRT] ERROR: Shell VBNV is 'xilinx_u250_xdma_201830_2'
[XRT] ERROR: Xclbin VBNV is 'xilinx_u200_qdma_201920_1'
[XRT] ERROR: Use 'xbmgmt flash' to update shell.
[XRT] ERROR: See dmesg log for details. err=-95
[XRT] ERROR: Failed to load xclbin.
Failed to program device[1] with xclbin file!
Trying to program device[2]: xilinx_u50_gen3x16_xdma_201920_3
[XRT] ERROR: Xclbin does not match shell on card.
[XRT] ERROR: Shell VBNV is 'xilinx_u50_gen3x16_xdma_201920_3'
[XRT] ERROR: Xclbin VBNV is 'xilinx_u200_qdma_201920_1'
[XRT] ERROR: Use 'xbmgmt flash' to update shell.
[XRT] ERROR: See dmesg log for details. err=-95
[XRT] ERROR: Failed to load xclbin.
Failed to program device[2] with xclbin file!
Trying to program device[3]: xilinx_u200_qdma_201920_1
Device[3]: program successful!
############################################################
                     Blocking Stream                        
############################################################
[ Case: 1 ] -> Throughput = 6.7537 GB/s
TEST PASSED
############################################################
                  Non-Blocking Stream                       
############################################################
[XRT] ERROR: xclWriteQueue: NONBLOCK but aio NOT enabled.

[XRT] ERROR: xclReadQueue: NONBLOCK but aio NOT enabled.

[XRT] ERROR: xclPollCompletion: async io is not enabled
[XRT] ERROR: sync io is not enabled
src/host.cpp:249 Error calling xcl::Stream::pollStreams(device.get(), poll_req, 2, 2, &num_compl, 50000, &ret), error code is: -30
[XRT] WARNING: Profiling may contain incomplete information. Please ensure all OpenCL objects are released by your host code (e.g., clReleaseProgram()).

Output from xbutil scan:

INFO: Found total 4 card(s), 4 are usable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
System Configuration
OS name:    Linux
Release:    4.15.0-128-generic
Version:    #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020
Machine:    x86_64
Model:      PowerEdge R740
CPU cores:  32
Memory:     95127 MB
Glibc:      2.27
Distribution:   Ubuntu 18.04.4 LTS
Now:        Mon Dec 21 06:50:32 2020 GMT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
XRT Information
Version:    2.8.726
Git Hash:   7c93966ead2dec777b92bdc379893f22b5bd561e
Git Branch: 2020.2
Build Date: 2020-11-11 20:29:19
XOCL:       2.8.726,7c93966ead2dec777b92bdc379893f22b5bd561e
XCLMGMT:    2.8.726,7c93966ead2dec777b92bdc379893f22b5bd561e
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 [0] 0000:d8:00.1 xilinx_u280_xdma_201920_3(ID=0x5e278820) user(inst=131)
 [1] 0000:af:00.1 xilinx_u250_xdma_201830_2(ID=0x5d14fbe6) user(inst=130)
 [2] 0000:5e:00.1 xilinx_u50_gen3x16_xdma_201920_3 user(inst=129)
 [3] 0000:3b:00.1 xilinx_u200_qdma_201920_1(ID=0x5dccb0ca) user(inst=128)

Did I miss any system configuration? Any suggestions would be helpful. Thank you!

Hi @Blaok , The same design is working in our internal testing:

./streaming_host_bandwidth ./build_dir.hw.xilinx_u200_qdma_201920_1/krnl_stream_adder1.xclbin Vector Increment of elements 0x10000000 by 1 Found Platform Platform Name: Xilinx INFO: Reading ./build_dir.hw.xilinx_u200_qdma_201920_1/krnl_stream_adder1.xclbin Loading: './build_dir.hw.xilinx_u200_qdma_201920_1/krnl_stream_adder1.xclbin' Trying to program device[0]: xilinx_u200_qdma_201920_1 Device[0]: program successful! ############################################################ Blocking Stream
############################################################ [ Case: 1 ] -> Throughput = 4.21837 GB/s TEST PASSED ############################################################ Non-Blocking Stream
############################################################ [ Case: 2 ] -> Throughput = 4.69532 GB/s TEST PASSED

As per below XRT Error message in your case, it looks like some machine setup is missing aio: [XRT] ERROR: xclWriteQueue: NONBLOCK but aio NOT enabled. [XRT] ERROR: xclReadQueue: NONBLOCK but aio NOT enabled. [XRT] ERROR: xclPollCompletion: async io is not enabled

Please post your query to Xilinx Vitis Forum to look into setup related issues: https://forums.xilinx.com/t5/Vitis-Acceleration-SDAccel-SDSoC/bd-p/tools_v

Xilinx / Vitis_Accel_Examples

host/streaming_host_bandwidth non-blocking fails and hangs #32