search

Xilinx / Vitis-AI

Vitis AI is Xilinx’s development stack for AI inference on Xilinx hardware platforms, including both edge devices and Alveo cards.
https://www.xilinx.com/ai
Apache License 2.0
1.49k stars 633 forks source link

KV260 and Vitis-AI 3.5 compatibility #1290

Open iraadit opened 1 year ago

iraadit commented 1 year ago

Is it possible to run Vitis-AI 3.5 (and so, yolov7 and yolov8) on the KV260 target?

In board_setup folder, there is only v70 and vek280. in 3.0, there was also a mpsoc folder (that was to be used for KV260).

Will KV260 be made compatible later on? Or can I for now compile it by myself to be able to use Vitis-AI 3.5 on KV260? If so, how? Or use the installation from 3.0? If so, will I be able to use yolov7 or yolov8?

Or is KV260 considered obsolete and will not be maintained anymore?

Thank you

iraadit commented 1 year ago

I now saw on this documentation page:

Vitis AI 3.5 supports the VEK280 Evaluation Board and the V70 Development Card for AI Inference.

For the following evaluation boards, refer to Vitis AI 3.0:

AMD Zynq™ UltraScale+™ MPSoC ZCU102 Evaluation Board
AMD Zynq™ UltraScale+™ MPSoC ZCU104 Evaluation Board
AMD Versal™ VCK190 Evaluation Board
AMD Kria™ KV260 Vision AI Starter Kit
AMD Versal™ VCK5000 Development Card

Does it mean I should also only use code (and models) of 3.0 on KV260? Or just that I have to install following the instructions of 3.0, but can run 3.5 models on it?

ahnhy1324 commented 1 year ago

I think you can check this document for that question.

https://xilinx.github.io/Vitis-AI/3.5/html/docs/workflow.html

they said this :

**Vitis™ AI 3.5 supports Zynq™ Ultrascale+™ and Versal™ AI Core architectures, however the IP for these devices is now considered mature and will not be updated with each release. Users should understand that we will continue to support these targets into the future and Vitis AI will update the pre-built board images and reference designs for these architectures with each major release (ie, 4.0).

For minor releases (x.5), users wishing to leverage these architectures have two choices:

Leverage Vitis™ AI 3.0 for initial evaluation and development.**

Build a custom board Petalinux image for their target leveraging the Vitis AI 3.5 runtime and libraries.

I use vitis 3.5 for my ultrascale+ board and tools are working like quantizer or some make make models for dpu flow.

but you'd better use 3.0 branch if you want to use pre-built board setup files.

quentonh commented 1 year ago

Yes, we fully anticipate that the Vitis AI 3.5 release is compatible with ZU+ targets, however, with the 3.5 release we did not do extensive testing, benchmarking nor do we provide pre-built images for those targets. We will update ZU+ targets with the 4.0 release. Keep in mind that there were no updates to the DPU IP with 3.5, so it should be very feasible to update the Petalinux image with the latest VART and VAIL versions.

iraadit commented 1 year ago

Thank you for your answers. I'll try to update the Petalinux image.

quentonh commented 1 year ago

@iraadit Please note that the packagegroup-petalinux-vitisai has not yet been released. However, the package group is simply a matter of convenience, and the availability of that package does not stop users from building their own image. If you look closely at that package group, you will quickly see that the package group is simply invoking BitBake to build the various components of Vitis AI. In fact, the Vitis AI repository provides the required BitBake recipes for the same components (albeit discretely) that are included in the package group. For instance, here is the BitBake recipe for the vitis-ai-library:

Vitis-AI/src/vai_petalinux_recipes/recipes-vitis-ai/vitis-ai-library/vitis-ai-library_3.5.bb at master · Xilinx/Vitis-AI (github.com)

The instructions to integrate these packages are quite simple and are provided in the Github.IO documentation: DPU IP Details and System Integration — Vitis™ AI 3.5 documentation (xilinx.github.io)

Note! There is an error in the above documentation that we just noticed this week and I will work on fixing it next week. Specifically, you can see that Step 5 tells the user to run petalinux-build, but the command that is listed is:

cp Vitis-AI/src/petalinux_recipes/recipes-vai-kernel /project-spec/meta-user/

Somehow this documentation became muddled in the more recent release.

This older link contains some information that is not found in the latest documentation. I have some work to do to fix the latest docs and fix this... https://github.com/Xilinx/Vitis-AI/tree/2.5/setup/petalinux

You will need to use PetaLinux 2023.1 to build an image for Vitis AI 3.5 as there have been updates to packages such as OpenCV in the base PetaLinux image that will not be pulled in if you try to use an older version of PetaLinux with these recipes.

iraadit commented 1 year ago

Hi @quentonh,

Sorry for the delayed answer, I worked on other subject and had long holidays.

Did you have the opportunity to update the documentation?

I think I successfully built PetaLinux 2023.1 for Vitis AI 3.5, here are my notes:

Set Up PetaLinux Working Environment

See https://docs.xilinx.com/r/2021.1-English/ug1144-petalinux-tools-reference-guide/Steps-to-Set-Up-PetaLinux-Working-Environment See https://xilinx.github.io/Vitis-AI/3.5/html/docs/workflow-system-integration.html#rebuilding-the-linux-image-with-petalinux

$ source petalinux-v2023.1-05012318/settings.sh
$ echo $PETALINUX

Download KV260 BSP

See https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-design-tools.html

$ petalinux-create -t project -s xilinx-kv260-starterkit-v2023.1-05080224.bsp
$ cp -r $PATH_TO_VITIS_AI/Vitis-AI/src/vai_petalinux_recipes/recipes-vitis-ai/ xilinx-kv260-starterkit-2023.1/project-spec/meta-user/
$ rm -rf xilinx-kv260-starterkit-2023.1/project-spec/meta-user/recipes-vitis-ai/vart/vart_3.5_vivado.bb

Edit /project-spec/meta-user/conf/user-rootfsconfig file, appending the following lines:

CONFIG_vitis-ai-library
CONFIG_vitis-ai-library-dev
CONFIG_vitis-ai-library-dbg
$ cd xilinx-kv260-starterkit-2023.1
$ petalinux-config -c rootfs

Select the following option (while leaving package packagegroup-petalinux-kria selected?):

Select user packages --->
Select [*] vitis-ai-library
petalinux-build

The full compilation log build.log is stored in the build sub-directory of your PetaLinux project. The final image, /images/linux/image.ub, is a FIT image. The kernel image (including RootFS) is Image for Zynq® UltraScale+™ MPSoC and Versal™ platform. (from https://docs.xilinx.com/r/2021.1-English/ug1144-petalinux-tools-reference-guide/Default-Image)

Generate Boot Image

See https://docs.xilinx.com/r/2021.1-English/ug1144-petalinux-tools-reference-guide/Generate-Boot-Image?tocId=3fqGlFncsr7XtKH0ay0cVA

petalinux-package --boot --u-boot --format BIN

See https://xilinx.github.io/kria-apps-docs/kv260/2021.1/build/html/docs/build_petalinux.html

petalinux-package --wic --bootfiles "ramdisk.cpio.gz.u-boot boot.scr Image system.dtb"

The generated image file will be located at images/linux/petalinux-sdimage.wic.

Tip: The generated wic file assumes a fixed partition size. The total file size is several GB but the actual usage on the ext4 partition is much lower. The file size can be significantly lower by compressing the file e.g. using gzip:

gzip images/linux/petalinux-sdimage.wic

This generates a new output file named images/linux/petalinux-sdimage.wic.gz with a file size of several MB.

Flash the image on an SD card using Balena Etcher. This image is functionally equivalent to the prebuilt wic image inside the bsp.

Write the image on the SD card trough network

We don't have access to the SD card through the Virtual Machine, so we will write on it from another host. On the other host (user@NUC-608235): Allow write for all on the SD card

sudo chmod a+w /dev/mmcblk1

On this machine:

cat cross_compilation_sdk/xilinx-kv260-starterkit-2023.1/images/linux/petalinux-sdimage.wic.gz | ssh user@NUC-608235 "gzip -d - | dd of=/dev/mmcblk1 status=progress"

Then, on the SD card, we have to change the fstab file, so that it mounts mmcblk1p1 (the SD card) instead of mmcblk0p1 as /boot

sudo nano /media/user/root/etc/fstab

How to connect from host to Kria

  1. Plug the USB cable from the host to the Kria (and not the power cable, yet)

  2. BEWARE ! Forward USB through Citrix as we are using a Virtual Desktop: open the Citrix tray on top of the screen, select "Peripheriques" and enable "Xilinx ML Carrier Card"

  3. Check that /dev/ttyUSB0 to dev/ttyUSB3 are present (numbers could be different if another USB device was already plugged in)

  4. Then, in different terminals, execute:

  • screen /dev/ttyUSB0 115200
  • screen /dev/ttyUSB1 115200
  • screen /dev/ttyUSB2 115200
  • screen /dev/ttyUSB3 115200

To exit: CTRL + A, D

  1. Plug in the power in the Kria dev board

  2. Check the different screen sessions, you should see a the Kria boot up ouptut in one of them. If nothing is displaying, try to press "Enter" in the different screen sessions

  3. Login: petalinux Password: root

  4. Execute "ifconfig" in the working screen session, to get the IP address of the board. Beware, the IP address could change, you will maybe have to do this procedure again to get the IP address again.

  5. Right now, IP address = 172.22.46.169

  6. ssh petalinux@172.22.46.169

  7. Password = root

Build the SDK

See https://xilinx.github.io/kria-apps-docs/kv260/2021.1/build/html/docs/build_petalinux.html#build-the-sdk

petalinux-build -s

The resulting self-extracting shell script installer file is located at images/linux/sdk.sh.

The SDK installer script can be copied to the application developer’s host machine and installed by simply running the script. Follow the prompts on the screen.

$ images/linux/sdk.sh
PetaLinux SDK installer version 2021.1_SOM
============================================
Enter target directory for SDK (default: /opt/petalinux/2021.1_SOM): ./images/linux/sdk
You are about to install the SDK to "/opt/petalinux/xilinx-k26-starterkit-2021.1/images/linux/sdk". Proceed [Y/n]? Y

Once the SDK is installed, source the file images/linux/sdk/environment-setup-aarch64-xilinx-linux to set up the cross-development environment. Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g. $ . /mnt/ssd2/cross_compilation_sdk/petalinux/2023.1/environment-setup-cortexa72-cortexa53-xilinx-linux

Build a sample

On host, through SDK 

$ . /mnt/ssd2/cross_compilation_sdk/petalinux/2023.1/environment-setup-cortexa72-cortexa53-xilinx-linux
# cd $PATH_TO/Vitis-AI/examples/vai_library/samples/classification
# ./build.sh

Then scp the executables on the KV260 target (using scp_executable_files.sh)

Installing the AI Library Package

See https://docs.xilinx.com/r/en-US/ug1354-xilinx-ai-sdk/Step-3-Installing-the-AI-Library-Package

$ tar -xzvf vitis-ai-runtime-3.5.0.tar.gz
$ scp -r vitis-ai-runtime-3.5.0/2023.1/aarch64/centos root@172.22.46.174:~/
$ ssh root@172.22.46.174
# cd ~/dpu_sw_optimize/zynqmp/
# ./zynqmp_dpu_optimize.sh
# cd ~/centos
# sudo bash setup.sh

Running Vitis AI Library Examples on Edge

See https://docs.xilinx.com/r/en-US/ug1354-xilinx-ai-sdk/Running-Vitis-AI-Library-Examples

$ scp vitis_ai_library_r3.5.0_images.tar.gz root@172.22.46.174:~/
$ scp vitis_ai_library_r3.5.0_video.tar.gz root@172.22.46.174:~/
$ ssh root@172.22.46.174
# tar -xzvf vitis_ai_library_r3.5*_images.tar.gz -C Vitis-AI/examples/vai_library
# tar -xzvf vitis_ai_library_r3.5*_video.tar.gz -C Vitis-AI/examples/vai_library

Test it

# cd ~/Vitis-AI/examples/vai_library/samples/classification
# ./test_jpeg_classification inception_v1_tf sample_classification.jpg

Installing the AI Model Package

See https://docs.xilinx.com/r/en-US/ug1354-xilinx-ai-sdk/Step-2-Installing-the-AI-Model-Package

For each model, there is a yaml file that describes all the details about the model. The yaml file contains download links for the various AMD target boards. Choose your model and the desired platform and download it. Copy the downloaded file to the target using scp with the following command.

$ scp <model>.tar.gz root@IP_OF_BOARD:~/

If the target board is connected to the Internet, you can also use the wget command to download the model directly to the board. Log in to the target board (using ssh or serial port) and install the model package. If the /usr/share/vitis_ai_library/model folder does not exist on the target, create it first.

# sudo mkdir -p /usr/share/vitis_ai_library/models

Install the model on the target board.

# tar -xzvf <model>.tar.gz -C /usr/share/vitis_ai_library/models

By default, the models are located in the /usr/share/vitis_ai_library/models directory on the target.

I think I have a zorking Petalinux with Vitis AI 3.5 on KV260. At least, it is booting. :D

But my problem now is that I cannot test it. Indeed, let's take the classification example: I need to add a model to use (for example, tf_inceptionv1_3.5) on the KV260 target. However, in the `model.yaml' file, I see available models for GPU, vek280 and v70;, but no KV260. Does it mean I have also to create my own compatible model? Can I use models from 3.0, or can I convert the GPU models to KV260? How should I proceed from here?

Thank you for your help

iraadit commented 1 year ago

I cross-compiled some example code and tried to execute it. It failed with the following error message, and I then tried to dig deeper:

$ ./test_jpeg_yolov7 yolov7_pt sample_yolov7.jpg
WARNING: Logging before InitGoogleLogging() is written to STDERR
F20230913 05:12:28.760835  1116 dpu_controller.cpp:52] Check failed: !get_factory_methods().empty() 
Aborted

$ xbutil examine
System Configuration
  OS Name              : Linux
  Release              : 6.1.5-xilinx-v2023.1
  Version              : #1 SMP Fri Apr 21 07:47:58 UTC 2023
  Machine              : aarch64
  CPU Cores            : 4
  Memory               : 3931 MB
  Distribution         : PetaLinux 2023.1+release-S05010539 (langdale)
  GLIBC                : 2.36
  Model                : ZynqMP KV260 revB

XRT
  Version              : 2.15.0
  Branch               : 2023.1
  Hash                 : 64c933573e7e50a8aba939a74209590c2b739e8b
  Hash Date            : 2023-04-17 09:18:13
  ZOCL                 : 2.15.0, 64c933573e7e50a8aba939a74209590c2b739e8b

Devices present
  0 devices found
BDF  :  Shell  Platform UUID  Device ID  Device Ready*  
--------------------------------------------------------

* Devices that are not ready will have reduced functionality when using XRT tools

$ dmesg | grep dpu

$ xdputil query
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20230913 05:39:56.710660  1171 xrt_bin_stream.cpp:49] Please check your /etc/vart.conf
Its format should be :
    firmware: xx
Example:
    firmware: /run/media/mmcblk0p1/dpu.xclbin
F20230913 05:39:56.710755  1171 xrt_bin_stream.cpp:53] [UNILOG][FATAL][VART_OPEN_DEVICE_FAIL][Cannot open device] open(/run/media/mmcblk0p1/dpu.xclbin) failed.
/usr/bin/xdputil: line 20:  1171 Aborted                 /usr/bin/python3 -m xdputil $*
xilinx-kv260-starterkit-20231:~/Vitis-AI/examples/vai_library/samples/yolov7$ cat /etc/vart.conf 
firmware: /run/media/mmcblk0p1/dpu.xclbin

$ ls /boot                
Image  boot.scr  ramdisk.cpio.gz.u-boot  system.dtb

It seems DPU is not installed / configured

First, it looks for it in /run/media/mmcblk0p1/dpu.xclbin. However, my SD card is mmcblk1p1 (not mmcblk0p1), and has been mounted on /boot. (I also had to modify the 0 to 1 in fstab for the system to boot) Second, there is no dpu.xclbin file in /boot.

Where should I find, or how should I create this dpu.xclbin file ?

Also, in https://github.com/Xilinx/Vitis-AI-Copyleft-Model-Zoo/tree/main/yolov7, pre-compiled models are available for VEK280 and V70, but not KV260. How should I produce/convert such model for KV260?

Thank you @quentonh

MRFiruzabadi commented 11 months ago

Hi @iraadit ,

Have you been able to successfully run the YOLOv7 model on the KV260 target? If so, I would greatly appreciate any guidance or instructions you can provide.

Thank you!

iraadit commented 11 months ago

Hi

No, I've never been able to successfully fully install Vitis-AI 3.5 on the KV260, and sadly had to resort to Vitis-AI 3.0 (and old yolo).

My conclusion from this: better support and more easy to run new models on Nvidia Jetson.