vmayoral
commented
3 years ago It'd be awesome if I could get your thoughts on this @rwarmstr, @randyh62 or someone else.
Open vmayoral opened 3 years ago
vmayoral
commented
3 years ago It'd be awesome if I could get your thoughts on this @rwarmstr, @randyh62 or someone else.
vmayoral
commented
3 years ago All right, after a few good hints (thanks @randyh62) and insisting on power-booting hardware platforms I conclude the following for non-DFX platforms:
v++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xov++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbinv++ -p -t hw --config ../../src/zcu102.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.shv++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xov++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbinvadd.xclbin into running imagev++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xov++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbinvadd.xclbin into running imagev++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xov++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbinvadd.xclbin into running imageSummarizing:
| Approach | Description | Result |
|---|---|---|
| 1 | recreating whole sd image | works |
| 2 | scp-ing new kernel, rebooting, launch | doesn't |
| 3 | scp-ing new kernel, launching, rebooting, launch | doesn't |
| 4 | scp-ing new kernel, power-off, booting, launch | doesn't |
Why's only option 1. working? Can someone explain what are the processes behind reprogramming the FPGA in non-DFX platforms? Do I need to recreate the whole raw image (for the sd card) with v++ -p every single time? where's this being reflected files-wise (i.e. what's changing in the file system for the FPGA to be reprogrammed?).
vmayoral
commented
3 years ago All right, further testing things out (and through good feedback provided by Randy, Wes, Terry and others) I fount out that for non-DFX base platforms what matters isn't really the *.xclbin file since this seems to be ignored completely. Instead, the corresponding FPGA is programmed by the bitstream contained into the BOOT.BIN file.
This then leads to the following fifth approach:
v++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xov++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbinv++ -p -t hw --config ../../src/zcu102.cfg ./vadd.xclbin --package.out_dir package --package.no_imageBOOT.BIN into boot partitionSummarizing:
| Approach | Description | Result |
|---|---|---|
| 1 | recreating whole sd image | works |
| 2 | scp-ing new kernel, rebooting, launch | doesn't |
| 3 | scp-ing new kernel, launching, rebooting, launch | doesn't |
| 4 | scp-ing new kernel, power-off, booting, launch | doesn't |
| 5 | scp-ing BOOT.bin, reboot, launch | works |
This addresses questions above 1 and 3, but 2 remains:
Premduth
commented
3 years ago @imrickysu can you look at this?
dj-park
commented
2 years ago Hi @vmayoral, I know this is an year ago, but what was your conclusion on the 2nd point?(did you get it work?) I think in the recent version of user guide(https://docs.xilinx.com/r/en-US/ug1393-vitis-application-acceleration/Configuring-the-PL-Kernel-in-DFX-Platforms-and-Non-DFX-Platform), your question regarding non-DFX platform is addressed, but isn't reloading vadd_new.xclbin in DFX platfrom supposed to work? Isn't that the point of DFX platform?
dj-park
commented
2 years ago In fact, I am experiencing the exact same issue. For instance, app_0.exe vadd_0.xclbin => test passed reboot app_1.exe vadd_1.xclbin => test passed
But if I don't reboot, for example, app_0.exe vadd_0.xclbin => test passed app_1.exe vadd_1.xclbin => hangs
mruizglz
commented
2 years ago I have exactly the same behaviour than @dj-park. I am using a ZCU102 (DFX) and 2021.1 tools. Is there any solution to this problem?
imrickysu
commented
2 years ago Hi @dj-park , @mruizglz , thanks for your feedback. I'll try to reproduce my end and follow up.
xerpi
commented
1 year ago If I'm not mistaken, when you target the real hardware (-t hw), a boot.bin file that contains the bitstream is created, which then you can copy to the boot partition on the SD to replace the bitstream loaded at boot.
Which would be equivalent when targeting the hardware emulator (-t hw_emu)?
shantirao
commented
9 months ago Still the case in 2023.2. Is there a way to put the PL in the xclbin file instead of the boot.bin file? Or maybe the FPGA has to boot before petalinux does, so there's no way to hot-swap the PL?
Context
This ticket's part of a debugging effort and connected to past issues including #69 #70 #71 and #72.
While developing kernels it's common to test (also in
hwtarget) incremental variants of it. So far, unfortunately, I haven't been able to do so. Only the first kernel produced (the first time the sd card is introduced is the hardware) remains. I haven't been able to successfully change this. The current issues has been validated in the following hardware and base platform combinations:A simple way to reproduce this is to build and test variants (silly ones, simple modifications of the kernel) of https://github.com/Xilinx/Vitis-Tutorials/tree/master/Getting_Started/Vitis. An exemplary flow to reproduce this issue is as follows:
app.exeobtaining results coherent to thevadd.xclbinkernel (whatever they are)vadd.cpp(source code) introduce a small (or big, up to you) changev++compilervadd.xclbin.newfor DFX platforms (or just build the whole image complete again withv++ -pfor non-DFX platforms)app.exe vadd.xclbin.newThe expected observation is that result remains fixed to 3. and not related to the changes introduced in 4., 5. and 6.. A past issue in XRT identified a similar behavior https://github.com/Xilinx/XRT/issues/2747.
Results so far
NOTE: I'm using Vitis suite 2020.2 on top of Ubuntu 20.04. NOTE 2: Ignore paths, they're specific to my development machines.
xilinx_zcu102_base_202020_1
Regardless of what I do the result's always the same which corresponds with the first kernel I flashed which modified https://github.com/Xilinx/Vitis-Tutorials/blob/master/Getting_Started/Vitis/example/src/vadd.cpp#L15 to: ```math out = in1 + 2 * in2 ``` ```bash cd /home/xilinx/Vitis-Tutorials/Getting_Started/Vitis/example/zcu102/hw; rm -r package/ _x/ vadd.x* app.exe a.xclbin v++* # configure env. source /tools/Xilinx/Vitis/2020.2/settings64.sh export PLATFORM_REPO_PATHS=/home/xilinx/xilinx_zcu102_base_202020_1 export ROOTFS=/home/xilinx/rootfs source /home/xilinx/rootfs/sdk/environment-setup-aarch64-xilinx-linux # build ${CXX} -Wall -g -std=c++11 ../../src/host.cpp -o app.exe -I/usr/include/xrt -lOpenCL -lpthread -lrt -lstdc++ v++ -c -t hw --config ../../src/zcu102.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xo v++ -l -t hw --config ../../src/zcu102.cfg ./vadd.xo -o vadd.xclbin v++ -p -t hw --config ../../src/zcu102.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.sh # test acceleration cd /media/sd-mmcblk0p1 export XILINX_XRT=/usr ./app.exe vadd.xclbin root@zynqmp-common-2020_2:/media/sd-mmcblk0p1# ./app.exe vadd.xclbin [ 265.853107] [drm] Pid 1221 opened device [ 265.857065] [drm] Pid 1221 closed device INFO: Found Xilinx Platform [ 265.863503] [drm] Pid 1221 opened device [ 265.869667] [drm] Pid 1221 closed device [ 265.873637] [drm] Pid 1221 opened device INFO: Loading 'vadd.xclbin' [ 265.962660] [drm] zocl_xclbin_read_axlf The XCLBIN already loaded [ 265.962676] [drm] zocl_xclbin_read_axlf c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 ret: 0 [ 265.971245] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1 [ 265.978917] [drm] Reconfiguration not supported [ 265.990677] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 unlocked, ref=0 [ 265.991808] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1 [ 266.005109] [drm] User buffer is not physical contiguous Error: Result mismatch i = 0 CPU result = 2349 Device result = 4281 TEST FAILED ... (continues) ```xilinx_zcu102_base_dfx_202020_1
```bash cd /home/xilinx/Vitis-Tutorials/Getting_Started/Vitis/example/zcu102/hw; rm -r package/ _x/ vadd.x* app.exe a.xclbin v++* # configure env. source /tools/Xilinx/Vitis/2020.2/settings64.sh export PLATFORM_REPO_PATHS=/home/xilinx/xilinx_zcu102_base_dfx_202020_1 export ROOTFS=/home/xilinx/rootfs source /home/xilinx/rootfs/sdk/environment-setup-aarch64-xilinx-linux # build ${CXX} -Wall -g -std=c++11 ../../src/host.cpp -o app.exe -I/usr/include/xrt -lOpenCL -lpthread -lrt -lstdc++ v++ -c -t hw --config ../../src/zcu102_dfx.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xo v++ -l -t hw --config ../../src/zcu102_dfx.cfg ./vadd.xo -o vadd.xclbin v++ -p -t hw --config ../../src/zcu102_dfx.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.sh # test acceleration cd /media/sd-mmcblk0p1 export XILINX_XRT=/usr ./app.exe vadd.xclbin root@zynqmp-common-2020_2:/media/sd-mmcblk0p1# ./app.exe vadd.xclbin [ 265.853107] [drm] Pid 1221 opened device [ 265.857065] [drm] Pid 1221 closed device INFO: Found Xilinx Platform [ 265.863503] [drm] Pid 1221 opened device [ 265.869667] [drm] Pid 1221 closed device [ 265.873637] [drm] Pid 1221 opened device INFO: Loading 'vadd.xclbin' [ 265.962660] [drm] zocl_xclbin_read_axlf The XCLBIN already loaded [ 265.962676] [drm] zocl_xclbin_read_axlf c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 ret: 0 [ 265.971245] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1 [ 265.978917] [drm] Reconfiguration not supported [ 265.990677] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 unlocked, ref=0 [ 265.991808] [drm] bitstream c82ababc-a138-4cb2-bfd8-595f8a4cd9b3 locked, ref=1 [ 266.005109] [drm] User buffer is not physical contiguous Error: Result mismatch i = 0 CPU result = 2349 Device result = 4281 TEST FAILED (gets stucked in here) ```xilinx_zcu104_base_202020_1
```bash cd /home/xilinx/Vitis-Tutorials/Getting_Started/Vitis/example/zcu104/hw # configure env. source /tools/Xilinx/Vitis/2020.2/settings64.sh export PLATFORM_REPO_PATHS=/home/xilinx/xilinx_zcu104_base_202020_1 # export ROOTFS=/home/xilinx/rootfs export ROOTFS=/home/xilinx/petalinux-xilinx-zcu104-2020.2/images/linux source /home/xilinx/rootfs/sdk/environment-setup-aarch64-xilinx-linux # build ${CXX} -Wall -g -std=c++11 ../../src/host.cpp -o app.exe -I/usr/include/xrt -lOpenCL -lpthread -lrt -lstdc++ v++ -c -t hw --config ../../src/zcu104.cfg -k vadd -I../../src ../../src/vadd.cpp -o vadd.xo v++ -l -t hw --config ../../src/zcu104.cfg ./vadd.xo -o vadd.xclbin v++ -p -t hw --config ../../src/zcu104.cfg ./vadd.xclbin --package.out_dir package --package.rootfs ${ROOTFS}/rootfs.ext4 --package.sd_file ${ROOTFS}/Image --package.sd_file xrt.ini --package.sd_file app.exe --package.sd_file vadd.xclbin --package.sd_file run_app.sh # run ... TEST PASSED ``` I haven't been able to get tests to fail after this. Regardless of the modifications in the kernels.Additional tests performed
Reinstalled Vitis suite and PetaLinux completely in a newly formatted machine with Ubuntu 20.04. Same result.
Temporary fix
A user proposed a temporary fix flashing manually the FPGA which I have not been able to reproduce https://github.com/Xilinx/XRT/issues/2747#issuecomment-583664143. Here're my results when attempting this:
xilinx_zcu104_base_202020_1
```bash # in the workstation xclbinutil --dump-section BITSTREAM:RAW:vadd.bit -i vadd.xclbin # then scp-it to embedded # in the embedded platform cd /media/sd-mmcblk0p1 export XILINX_XRT=/usr fpgautil -b ./vadd.bit root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# ./app.exe [ 158.257308] [drm] Pid 1028 opened device [ 158.261267] [drm] Pid 1028 closed device INFO: Found Xilinx Platform [ 158.267677] [drm] Pid 1028 opened device [ 158.273834] [drm] Pid 1028 closed device [ 158.277794] [drm] Pid 1028 opened device INFO: Loading 'vadd.xclbin' [ 158.430057] [drm] zocl_xclbin_read_axlf The XCLBIN already loaded [ 158.430073] [drm] zocl_xclbin_read_axlf 030235fc-c6ab-4195-83b3-00e2e0860b11 ret: 0 [ 158.438503] [drm] bitstream 030235fc-c6ab-4195-83b3-00e2e0860b11 locked, ref=1 [ 158.446209] [drm] Reconfiguration not supported [ 158.457973] [drm] bitstream 030235fc-c6ab-4195-83b3-00e2e0860b11 unlocked, ref=0 (gets stucked in here) ``` Further debugging this issue led to a kernel panic: ```bash root@xilinx-zcu104-2020_2:~# cd /media/sd-mmcblk0p1 root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# export XILINX_XRT=/usr root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# fpgautil -b ./vadd.bit [ 42.264337] fpga_manager fpga0: writing vadd.bit to Xilinx ZynqMP FPGA Manager Time taken to load BIN is 9302.000000 Milli Seconds BIN FILE loaded through FPGA manager successfully root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# ls BOOT.BIN boot.scr profile_summary.csv timeline_trace.csv xrt.ini Image init.sh run_app.sh vadd.bit app.exe platform_desc.txt system.dtb vadd.xclbin root@xilinx-zcu104-2020_2:/media/sd-mmcblk0p1# gdb ./app.exe GNU gdb (GDB) 8.3.1 Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or laterxilinx_zcu102_base_dfx_202020_1
```bash # in the workstation xclbinutil --dump-section BITSTREAM:RAW:vadd.bit -i vadd.xclbin # then scp-it to embedded # in the embedded platform cd /media/sd-mmcblk0p1 export XILINX_XRT=/usr fpgautil -b ./vadd.bit root@zynqmp-common-2020_2:/media/sd-mmcblk0p1# fpgautil -b ./vadd.bit [ 430.136806] fpga_manager fpga0: writing vadd.bit to Xilinx ZynqMP FPGA Manager [ 445.466900] mmc0: Timeout waiting for hardware interrupt. [ 445.472292] mmc0: sdhci: ============ SDHCI REGISTER DUMP =========== [ 445.478714] mmc0: sdhci: Sys addr: 0x00000000 | Version: 0x00001002 [ 445.485137] mmc0: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000 [ 445.491560] mmc0: sdhci: Argument: 0x0042c800 | Trn mode: 0x00000027 [ 445.497983] mmc0: sdhci: Present: 0x01ff0000 | Host ctl: 0x0000001f [ 445.504406] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080 [ 445.510830] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000207 [ 445.517253] mmc0: sdhci: Timeout: 0x00000009 | Int stat: 0x00000002 [ 445.523676] mmc0: sdhci: Int enab: 0x03ff008b | Sig enab: 0x03ff008b [ 445.530099] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000001 [ 445.536522] mmc0: sdhci: Caps: 0x35ecc881 | Caps_1: 0x00002007 [ 445.542945] mmc0: sdhci: Cmd: 0x0000193a | Max curr: 0x00000000 [ 445.549369] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x003b377f [ 445.555792] mmc0: sdhci: Resp[2]: 0x325b5900 | Resp[3]: 0x00000c00 [ 445.562214] mmc0: sdhci: Host ctl2: 0x00000001 [ 445.566643] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x000000087926d2c0 [ 445.573759] mmc0: sdhci: ============================================ [ 456.306892] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 456.312799] rcu: 0-...0: (1 GPs behind) idle=926/0/0x1 softirq=4120/4121 fqs=2596 [ 456.320444] rcu: 1-...0: (1 GPs behind) idle=e82/1/0x4000000000000000 softirq=2437/2438 fqs=2596 [ 456.329391] (detected by 3, t=5257 jiffies, g=8457, q=24) [ 456.334858] Task dump for CPU 0: [ 456.338071] swapper/0 R running task 0 0 0 0x0000000a [ 456.345109] Call trace: [ 456.347548] __switch_to+0x1c4/0x288 [ 456.351108] 0xffff8000110d98a8 [ 456.354232] Task dump for CPU 1: [ 456.357443] sh R running task 0 1257 1253 0x00000022 [ 456.364482] Call trace: [ 456.366915] __switch_to+0x1c4/0x288 [ 456.370472] 0xfffffe0021cd95c0 (board hangs in here) ```Questions
I was surprised that this (by "this" I mean the development flows explained above in Context) couldn't work in non-DFX platforms. Is this expected to work in non-DFX platforms? If not, what's the proposal for reprogramming the FPGA with XRT flows?addressed in https://github.com/Xilinx/Vitis-Tutorials/issues/74#issuecomment-838149806Since none of the above worked and the proposed fix in a past issue only led to more issues, how can I manually reprogram FPGAs for using them with XRT?addressed in https://github.com/Xilinx/Vitis-Tutorials/issues/74#issuecomment-838149806I'm currently blocked with this (and have been blocked by it for more than a week already) so It'd be great if someone could help with it or suggest and alternative route. Anything for
ZCU102orZCU104would do. Thanks!