Does TIM-VX utilizes NPU of vim3(A311D)?

[janoslim]

Hello, I'm going to use TIM-VX to use the VIM3 development board. I have some questions because it's my first time dealing with applications related to ovxlib and npu. I'm going to use TIM-VX for research purposes. The goal of the study is utilizing NPU effectively. I wonder that,

1. if the NPU of VIM3 can be used through TIM-VX,
2. if it can be used, do all layers defined at TIM-VX can be run on NPU,
3. Is there any method to check to know that I am utilizing NPU when run time,
4. and how much cpu usage of tim-vx is. TIM-VX seems like a very interesting subject. Thank you for reading it!!

[thezha]

1. Yes
2. Yes
3. Not at the moment
4. if the entire graph is offloaded to the NPU, then CPU usage during inference is near 0. The CPU usage comes from the onetime cost of graph compilation.

[janoslim]

Thank you for your reply!!

I do have just a question that, I am willing to use TIM-VX with seperated graphs layerwisely (such as conv-graph or Relu-graph), Would I have a performance degradation when I create Graphs seperately compare to build graph with full dnn model?

Thank you so much!

[thezha]

You will not get optimal performance when you build graphs with a single layer since compiler cannot apply further optimizations such as fusion and scheduling.

[BUG1989]

@janoslim our project Tengine has been integrated the TIM-VX for Khadas VIM3(A311D inside), you can reference it, thanks. https://github.com/OAID/Tengine/blob/tengine-lite/doc/npu_tim-vx_user_manual.md

[janoslim]

@thezha @BUG1989 Thank you so much for your answers. This would be the last question and I'm afraid my questions might bother you. Does TIM-VX available on Android11 on VIM3? I already deploy Android11 from aosp on vim3 and I want to use TIM-VX on android 11, Is it available to do this? Thanks.

[kalcohol]

@BUG1989 ./tm_classification_timvx -m ./models/mobilenet_uint8.tmfile -i ./models/face1.jpg -g 224,224 -s 0.017,0.017,0.017 -w 104.007,116.669,122.679 -r 500 Tengine plugin allocator TIMVX is registered. tengine-lite library version: 1.2-dev TIM-VX prerun. [ 1] Failed to open device: No such file or directory, Try again... [ 2] Failed to open device: No such file or directory, Try again... [ 3] Failed to open device: No such file or directory, Try again... [ 4] Failed to open device: No such file or directory, Try again... [ 5] _OpenDevice(1130): FATAL: Failed to open device, errno=No such file or directory. Can‘t find the npu?

u have to make sure that the galcore.ko is the right version for tim-vx, it seems your galcore.ko is not 6.4.3.p0.286725. cmd and print like below:

khadas@Khadas:~$ sudo dmesg | grep "Galcore"
[sudo] password for khadas: 
[   16.888126] Galcore version 6.4.3.p0.286725

when you download & decompress vim3 prebuild-sdk of A311D, galcore.ko should in folder /prebuild-sdk-a311d/lib. default galcore.ko in vim3 is in

/usr/lib/modules/4.9.241/kernel/drivers/amlogic/npu/galcore.ko

so u can remove this kernel module and insert new one in /prebuild-sdk-a311d/lib/galcore.ko.

cd /usr/lib/modules/4.9.241/kernel/drivers/amlogic/npu
sudo rmmod galcore.ko
cd <your download & decompress path>/prebuild-sdk-a311d/lib
sudo insmod  galcore.ko

or you can simply replace this galcore and reboot, than check the galcore version.

and because of Tengine was already integrated inside VIM3 since Dec. 2020, export LD_LIBRARY_PATHto avoid path conflict is recommended. suppose your Tengine path is /home/khadas/tengine, and build path is /home/khadas/tengine/build, than cmd is below:

export LD_LIBRARY_PATH=/home/khadas/tengine/3rdparty/lib/aarch64:/home/khadas/tengine/build/src:$LD_LIBRARY_PATH

also you can join our qq group(829565581) for more info.

[kalcohol]

after remote assistance, issue solved.

[Ehsan-aghapour]

@thezha @BUG1989 Thank you so much for your answers. This would be the last question and I'm afraid my questions might bother you. Does TIM-VX available on Android11 on VIM3? I already deploy Android11 from aosp on vim3 and I want to use TIM-VX on android 11, Is it available to do this? Thanks.

Hello,

I have the same question, if google android aosp version 11 support NPU in khadas board? If not, is it possible to work on its support? (Is there manual or link to work on it)

Thanks, Ehsan

[thezha]

Android NN support is available, but not via TIM-VX API. We supply a separate software package for SoC vendor to integrate with Android NN, I think Amlogic/Khadas already has this support.

[Ehsan-aghapour]

Android NN support is available, but not via TIM-VX API. We supply a separate software package for SoC vendor to integrate with Android NN, I think Amlogic/Khadas already has this support.

Thanks for your reply. The AOSP that khadas repository published: https://github.com/khadas/android_manifest.git supports npu. However I should use AOSP of google repository: https://android.googlesource.com/platform/manifest

I am wondering that could I use npu with this AOSP? If yes, would you please tell me how?

Best regards, Ehsan

[sunshinemyson]

@Ehsan-aghapour ,

We should be able to work with Android 11 and before. It depends on Khadas/AML add it or not. We don't have the latest AOSP for VIM3 yet.

[sunshinemyson]

Close issue since inactive over months.