Closed pedrol3001 closed 4 months ago
https://opensource.rock-chips.com/wiki_Main_Page on this page you have an email : opensource@rock-chips.com
i've sent question about integration of npu driver in mainline.
@mcarbonneaux They have an open source rknpu driver here, However, they usually won't bother mainlining their drivers. It is too much trouble for them to write "clean code", merge and maintain it etc.
However, Collabora is working on mainlining the rk3588. They document their progress here, the NPU driver and multimedia codecs for video processing are listed, but they haven't worked on it yet. So, it will take some time before the rockchip platform works in mainline linux.
For everyone who wants to use HA and frigate for rockchip:
Currently, there is only one way (Home Assistant Container): Install a distro that has all necessary drivers (I recommend this one, if you have a rk3588/s soc) and install docker. Then use the HA docker image (see here for install instructions and here for instructions to migrate your data from a previous HAOS installation) and install frigate separately using docker. However, this will be very annoying for people who use addons a lot, because you will need to setup all addons separately using docker. See here for differences between HAOS and HA Container.
In the near there will be (hopefully) another way to use HA and frigate together on rockchip boards (Home Assistant Supervised): Once again install a distro that comes with all drivers and follow the instructions to install HA supervised. Then install the frigate-rockchip addon. As one can see here, HAOS and HA Supervisor offer similar features. Most notably, you can install addons using this method. Migration from previous HAOS install should also be easier using the Backup function. People who want to have an experience similar to HAOS should wait until this method works. However, this will probably take some weeks, because the HA teams needs to implement a feature to allow the addon to mount the device-tree to another path. I tried to contact one of the devs about this feature request, but he didn't respond so far.
The best way is of course to use HAOS directly. However, this will take at least 1+ year, because the Collabora team (or other contributors) needs to develop a rkvdec2 and rknpu driver, merge them into the mainline kernel repo and HAOS needs to use this kernel (it usually doesn't use brand new kernels but rather older/ more stable kernels, so it lacks some months behind new features).
My former approach to tackle this issue (mounting the device-tree to /proc/device-tree
) will not work. So, I came up with another solution. Unfortunately, I can't test this solution myself since I don't have physical access to my board in the next weeks and can't install a new os. I hope someone can test my approach.
My idea is to install home assistant supervised on Debian with Rockchip bsp Kernel (this way there should be no driver issues) and run the addon in full access mode (this will maybe grant the container access to /proc/device-tree).
install debian 12 with rockchip bsp kernel version 5.10 and all drivers:
uname -a
in your terminal. The Output should start with Linux debian 5.10.160-rockchip
or similar (it should mention version 5.10
and prefix -rockchip
).ls /dev/dri
. The output should contain renderD128 renderD129
.sudo cat /sys/kernel/debug/rknpu/version
, it should print v0.9.0
or newer.Next we install home assistant supervised
$ sudo su -
# apt install \
apparmor \
cifs-utils \
curl \
dbus \
jq \
libglib2.0-bin \
lsb-release \
network-manager \
nfs-common \
systemd-journal-remote \
systemd-resolved \
udisks2 \
wget -y
# curl -fsSL get.docker.com | sh
# cd ~ \
&& wget https://github.com/home-assistant/os-agent/releases/download/1.6.0/os-agent_1.6.0_linux_aarch64.deb \
&& dpkg -i os-agent_1.6.0_linux_aarch64.deb
# wget -O homeassistant-supervised.deb https://github.com/home-assistant/supervised-installer/releases/latest/download/homeassistant-supervised.deb \
&& apt install ./homeassistant-supervised.deb
http://homeassistant.local:8123/
or http://X.X.X.X:8123
(replace X.X.X.X with your IP). Do the default setup (enter your name, password etc.) but don't do any additional configuration (don't install any add-ons or setup any Integrations etc.). Enable advanced mode in the home assistant user settings.Now we try to get the frigate-rockchip addon running in full access mode:
addons
under /usr/share/hassio
. Clone the github repo https://github.com/MarcA711/frigate-hass-addons and copy the folder frigate_rochchip
within the repo into your addons folder. Following the documentation here.Now we test the plugin. My guess is that hardware video processing will work but hardware object detection (NPU) not.
All these instructions above are untested and I am not very experienced with Home Assistant addons. So you may encounter problems. If so, please contact me or try to solve them yourself. If you get either video processing or object detection working, please share your experience! My guess is that video processing should work but object detection not (see discussion here https://github.com/home-assistant/supervisor/issues/4863). Thank you really much if you tried this out!
@MarcA711
Tested on Rock5b, installed BSP Debian 11, then updated to BSP debian 12.
installled HOAS supervised (sudo nano /etc/kernel/cmdline && sudo u-boot-update) to update kernel arguments.
Add:
systemd.unified_cgroup_hierarchy=0 apparmor=1 security=apparmor
Then reboot Host: no more warning
Home assistant is working correctly.
Tried to git clone & copy the frigate rochchip folder to /usr/share/hassio/addons folder Rebooted, but the addon is not visible in HASS (i think we need to build it locally). I instead added your repo direcctly on HASS addon repo and installed the frigate rockchip addon.
I can get Hardware acceleration working, but not npu.
Hey @nyok92,
Thanks a lot for testing! This is great news, it worked as expected and at least hardware video processing works now.
I don't know if there is a way to do this, but if you can start the addon manually using docker and add this to the command, the NPU should work: --security-opt systempaths=unconfined
Otherwise, try adding: --security-opt systempaths=unconfined --security-opt apparmor=unconfined
However, I am not sure if it is possible to apply these changes.
@MarcA711 From host working with: docker run -it --privileged --rm --security-opt systempaths=unconfined --security-opt apparmor=unconfined --publish=5000:5000 -v /usr/share/hassio/addons/frigate_rochchip/config:/config ghcr.io/blakeblackshear/frigate:0.13.1-rk
but not seen on HASS Addons UI (somehow not linked to HASS supervisor), should be as expected as this has not been launched from HASS supervisor
This is great. This means the Home Assistant devs only need to add an option to the addon config to add the option --security-opt systempaths=unconfined
to the docker run command. I already requested this change and one of the devs said he will look into this. The corresponding issue can be found here: https://github.com/home-assistant/supervisor/issues/4863
@mcarbonneaux They have an open source rknpu driver here, However, they usually won't bother mainlining their drivers. It is too much trouble for them to write "clean code", merge and maintain it etc.
i've seen this (is in my issue in haos i think) but effectively is on old kernel...not in mainline...
Всем привет, попытался всё сделать по инструкции, добавил фрегат рокчип в список аддонов и установил. Но при настройки камеры вылазит ошибка с ffmpeg. ERROR : GarCam: Unable to read frames from ffmpeg process. 2024-02-11 08:55:19.791278847 [2024-02-11 12:55:19] frigate.video ERROR : GarCam: ffmpeg process is not running. exiting capture thread... 2024-02-11 08:55:34.585656852 [2024-02-11 12:55:34] watchdog.GarCam ERROR : Ffmpeg process crashed unexpectedly for GarCam. 2024-02-11 08:55:34.591003512 [2024-02-11 12:55:34] watchdog.GarCam ERROR : The following ffmpeg logs include the last 100 lines prior to exit. 2024-02-11 08:55:34.593808514 [2024-02-11 12:55:34] ffmpeg.GarCam.detect ERROR : [h264 @ 0x558b0f5230] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/
В командной строке orangepi 5 есть проблема с этими пакетами: Listing... Done libavcodec-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libavcodec59/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libavdevice-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libavdevice59/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4]libavfilter-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libavfilter8/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libavformat-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libavformat59/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4]libavutil-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4]libavutil57/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libpostproc-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libpostproc56/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4]libswresample-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libswresample4/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libswscale-dev/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] libswscale6/stable,stable 7:5.1.4-0+deb12u1 arm64 [upgradable from: 7:5.1.3-4] orangepi@orangepi5:~$
Есть идеи как можно исправить?
Hi ! I have board OrangePi 5, installed deb 12, installed HOAS supervised. The whole problem is that the container in the addon is launched in unprivileged mode, if you launch the container in privileged mode through the terminal, then everything works out of the box. Trying to run
"docker run -it --rm --publish=5000:5000 --security-opt systempaths=unconfined --security-opt apparmor=unconfined -v /usr/share/hassio/addons/local/frigate-rockchipOpi5- dev/config:/config -v /sys/firmware/devicetree/base:/sys/firmware/devicetree/base #########"solves the issue with device-tree, but there are other errors with "MPP".
mpp[792]: mpp_info: mpp version: 1844ec5 author: MarcA711 2023-11-12 add -lstdc++ to Libs.private mpp[792]: mpp_rt: can NOT found any allocator mpp[792]: mpp_log: allocator_std_open Warning: std allocator should be used on simulation mode only mpp[792]: mpp_allocator: mpp_allocator_get type 1 failed mpp[792]: mpp_buffer: Assertion p->allocator failed at get_group:936 mpp[792]: mpp_buffer: Assertion p->alloc_api failed at get_group:937 mpp[792]: vcodec_service: open vcodec_service (null) failed mpp[792]: hal_h265d_api: mpp_dev_init failed ret: -1 mpp[792]: mpp_hal: mpp_hal_init hal h265d_rkdec init failed ret -1 mpp[792]: mpp_hal: mpp_hal_init could not found coding type 16777220 mpp[792]: mpp_dec: mpp_dec_init could not init hal mpp[792]: mpp_time: mpp_clock_put invalid clock (nil)
To summarize, the simplest method would be if the Home Assist developers added the ability to run a container in the add-on in privileged mode.
in the course of the conversation
root@hass:~# docker run -it --rm --publish=5000:5000 --security-opt systempaths=unconfined --security-opt apparmor=unconfined -v /uhare/hassio/addons/local/frigate-rockchipOpi5-dev/config:/config -v /sys/firmware/devicetree/base:/sys/firmware/devicetree/base -v /dri:/dev/dri ########## s6-rc: info: service s6rc-fdholder: starting s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service s6rc-fdholder successfully started s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service log-prepare: starting s6-rc: info: service log-prepare successfully started s6-rc: info: service nginx-log: starting s6-rc: info: service go2rtc-log: starting s6-rc: info: service frigate-log: starting s6-rc: info: service frigate-log successfully started s6-rc: info: service nginx-log successfully started s6-rc: info: service go2rtc-log successfully started s6-rc: info: service go2rtc: starting s6-rc: info: service go2rtc successfully started s6-rc: info: service go2rtc-healthcheck: starting s6-rc: info: service frigate: starting s6-rc: info: service frigate successfully started s6-rc: info: service nginx: starting s6-rc: info: service nginx successfully started 2024-02-11 20:41:20.131192270 [INFO] Preparing Frigate... 2024-02-11 20:41:20.132533620 [INFO] Starting NGINX... s6-rc: info: service go2rtc-healthcheck successfully started s6-rc: info: service legacy-services: starting 2024-02-11 20:41:20.136112014 [INFO] Preparing new go2rtc config... s6-rc: info: service legacy-services successfully started 2024-02-11 20:41:20.179971475 [INFO] Starting Frigate... 2024-02-11 20:41:20.770805467 [INFO] Starting go2rtc... 2024-02-11 20:41:20.892877728 20:41:20.892 INF go2rtc version 1.8.4 linux/arm64 2024-02-11 20:41:20.893461926 20:41:20.893 INF [rtsp] listen addr=:8554 2024-02-11 20:41:20.893836419 20:41:20.893 INF [api] listen addr=:1984 2024-02-11 20:41:20.893846044 20:41:20.893 INF [webrtc] listen addr=:8555 2024-02-11 20:41:22.679752320 [2024-02-11 20:41:22] frigate.app INFO : Starting Frigate (0.13.0-fdecad4) 2024-02-11 20:41:22.680058564 [2024-02-11 20:41:22] frigate.app INFO : Creating directory: /media/frigate/reings 2024-02-11 20:41:22.680453182 [2024-02-11 20:41:22] frigate.app INFO : Creating directory: /media/frigate/cl 2024-02-11 20:41:22.680514140 [2024-02-11 20:41:22] frigate.app INFO : Creating directory: /tmp/cache 2024-02-11 20:41:22.680695553 [2024-02-11 20:41:22] frigate.app INFO : Creating directory: /media/frigate/exs 2024-02-11 20:41:22.714361739 [2024-02-11 20:41:22] peewee_migrate.logs INFO : Starting migrations 2024-02-11 20:41:22.722846169 [2024-02-11 20:41:22] peewee_migrate.logs INFO : There is nothing to migrate 2024-02-11 20:41:22.735119573 [2024-02-11 20:41:22] frigate.app INFO : Recording process started: 454 2024-02-11 20:41:22.739487203 [2024-02-11 20:41:22] frigate.app INFO : go2rtc process pid: 89 2024-02-11 20:41:22.807944804 [2024-02-11 20:41:22] detector.rknn INFO : Starting detection process: 464 2024-02-11 20:41:22.812570555 [2024-02-11 20:41:22] frigate.app INFO : Output process started: 466 2024-02-11 20:41:22.839255824 [2024-02-11 20:41:22] frigate.app INFO : Camera processor started for cam_1: 4 2024-02-11 20:41:22.853247406 [2024-02-11 20:41:22] frigate.app INFO : Capture process started for cam_1: 47 2024-02-11 20:41:23.367910310 E RKNN: [20:41:23.367] failed to open rknpu module, need to insmod rknpu dirver! 2024-02-11 20:41:23.368231138 E RKNN: [20:41:23.367] failed to open rknn device! 2024-02-11 20:41:23.469635104 E Catch exception when init runtime! 2024-02-11 20:41:23.474013526 E Traceback (most recent call last): 2024-02-11 20:41:23.474030442 File "/usr/local/lib/python3.9/dist-packages/rknnlite/api/rknn_lite.py", line 148, in init_runtime 2024-02-11 20:41:23.474053483 self.rknn_runtime.build_graph(self.rknn_data, self.load_model_in_npu) 2024-02-11 20:41:23.474067483 File "rknnlite/api/rknn_runtime.py", line 875, in rknnlite.api.rknn_runtime.RKNNRuntime.build_grap 2024-02-11 20:41:23.474141273 Exception: RKNN init failed. error code: RKNN_ERR_FAIL 2024-02-11 20:41:23.474147398 2024-02-11 20:41:23.481709887 [2024-02-11 20:41:23] frigate.detectors.plugins.rknn E : Error initializing rknn runtime. Do yun docker in privileged mode? mpp[477]: mpp_info: mpp version: 1844ec5 author: MarcA711 2023-11-12 add -lstdc++ to Libs.private mpp[477]: mpp_rt: can NOT found any allocator mpp[477]: mpp_log: allocator_std_open Warning: std allocator should be used on simulation mode only mpp[477]: mpp_allocator: mpp_allocator_get type 1 failed mpp[477]: mpp_buffer: Assertion p->allocator failed at get_group:936 mpp[477]: mpp_buffer: Assertion p->alloc_api failed at get_group:937 mpp[477]: vcodec_service: open vcodec_service (null) failed mpp[477]: hal_h265d_api: mpp_dev_init failed ret: -1 mpp[477]: mpp_hal: mpp_hal_init hal h265d_rkdec init failed ret -1 mpp[477]: mpp_hal: mpp_hal_init could not found coding type 16777220 mpp[477]: mpp_dec: mpp_dec_init could not init hal mpp[477]: mpp_time: mpp_clock_put invalid clock (nil) mpp[477]: mpp_time: mpp_clock_put invalid clock (nil) mpp[477]: mpp_time: mpp_clock_put invalid clock (nil) mpp[477]: mpp_time: mpp_clock_put invalid clock (nil)
Hey @YuryMcv,
I guess the Home Assistant devs left out the option to run add-ons in privileged mode on purpose for security reasons. And I don't think they will add it.
I guess there are some capabilities and security options that one can add in order to allow containers to access the hardware devices. I will do some research and see if I am able to run it on my system without privileged mode.
Hey @Ra3eR63,
I have problems understanding your setup and issue. Are you using HA supervised?
Please check if your Kernel has the required drivers using:
uname -a
in your terminal. The Output should start with Linux debian 5.10.160-rockchip or similar (it should mention version 5.10 and prefix -rockchip).ls /dev/dri
. The output should contain renderD128 renderD129.sudo cat /sys/kernel/debug/rknpu/version
, it should print v0.9.0 or newer.
Хоумассистент установлен как в вашей рекомендации, в докере.
Hey @MarcA711, I'm also exploring the possibility of running it in normal mode, to understand what settings are needed for this. At the moment, you can only access the device tree with " --security-opt systempaths=unconfined --security-opt apparmor=unconfined" and mount "/sys/firmware/devicetree/base:/sys/firmware/devicetree/base", Otherwise it doesn't work. Next, I look at what other configuration is needed for a successful launch. As for launching the add-on in privileged mode, you need to ask the Home Assist developers what they think about this.
I guess you don't need this mount: "/sys/firmware/devicetree/base:/sys/firmware/devicetree/base"
The option: --security-opt systempaths=unconfined should mount it automatically
Привет @Ra3eR63 , Проверки что написал MarcA711 надо сделать из контейнера, вставьте в терминале :~# docker ps , найдите CONTAINER ID фригата , затем в терминале выполните: docker exec -it ***** /bin/bash , вместо звездочек CONTAINER ID, вы попадете в контейнер фригата и там выполните : ls /dev/dri , и cat /sys/kernel/debug/rknpu/version
@YuryMcv Забавно, но я не успеваю это сделать) Айди контейнера меняется с какой-то супер скоростью))
@Ra3eR63 Это значит контейнер останавливается. посмотрите логи фрегата, рекомендую пока использовать фрегат с CPU
@MarcA711 yes, the option is really enough: --security-opt systempaths=unconfined --security-opt apparmor=unconfined, I previously tried to use it but for some reason it didn’t work, apparently I made some mistake. But still I can’t run it:
2024-02-12 14:45:17.901981929 E RKNN: [14:45:17.901] failed to open rknpu module, need to insmod rknpu dirver! 2024-02-12 14:45:17.902072636 E RKNN: [14:45:17.902] failed to open rknn device! 2024-02-12 14:45:18.003171540 E Catch exception when init runtime! 2024-02-12 14:45:18.006222319 E Traceback (most recent call last): 2024-02-12 14:45:18.006239818 File "/usr/local/lib/python3.9/dist-packages/rknnlite/api/rknn_lite.py", line 148, in init_runtime 2024-02-12 14:45:18.006260526 self.rknn_runtime.build_graph(self.rknn_data, self.load_model_in_npu) 2024-02-12 14:45:18.006272484 File "rknnlite/api/rknn_runtime.py", line 875, in rknnlite.api.rknn_runtime.RKNNRuntime.build_graph 2024-02-12 14:45:18.006350941 Exception: RKNN init failed. error code: RKNN_ERR_FAIL
rknn detector works using these options:
--security-opt systempaths=unconfined --security-opt apparmor=unconfined --device /dev/dri:/dev/dri
Using these options rknpu and ffmpeg using hardware works:
--security-opt systempaths=unconfined --security-opt apparmor=unconfined --device /dev/dri:/dev/dri --device /dev/dma_heap:/dev/dma_heap --device /dev/rga:/dev/rga --device /dev/mpp_service:/dev/mpp_service
Moreover, Home Assistant Addons already support the device
option. So, we only need a way to add --security-opt systempaths=unconfined
and everything should work.
@MarcA711 So what is better to use for testing, leave the rockchip addon for homeassistant or build a container in docker from your repository? And configure it in the home assistant via a proxy frigate?
The Home Assistant Addon will not work until this got fixed: https://github.com/home-assistant/supervisor/issues/4863
Using these options rknpu and ffmpeg using hardware works:
--security-opt systempaths=unconfined --security-opt apparmor=unconfined --device /dev/dri:/dev/dri --device /dev/dma_heap:/dev/dma_heap --device /dev/rga:/dev/rga --device /dev/mpp_service:/dev/mpp_service
doesn't work unfortunately, same error
2024-02-12 19:35:55.670507576 E RKNN: [19:35:55.670] failed to open rknpu module, need to insmod rknpu dirver! 2024-02-12 19:35:55.670597116 E RKNN: [19:35:55.670] failed to open rknn device! 2024-02-12 19:35:55.771905809 E Catch exception when init runtime! 2024-02-12 19:35:55.775398747 E Traceback (most recent call last): 2024-02-12 19:35:55.775412746 File "/usr/local/lib/python3.9/dist-packages/rknnlite/api/rknn_lite.py", line 148, in init_runtime 2024-02-12 19:35:55.775469329 self.rknn_runtime.build_graph(self.rknn_data, self.load_model_in_npu) 2024-02-12 19:35:55.775485953 File "rknnlite/api/rknn_runtime.py", line 875, in rknnlite.api.rknn_runtime.RKNNRuntime.build_graph 2024-02-12 19:35:55.775591535 Exception: RKNN init failed. error code: RKNN_ERR_FAIL 2024-02-12 19:35:55.775595618 2024-02-12 19:35:55.781402015 [2024-02-12 19:35:55] frigate.detectors.plugins.rknn E : Error initializing rknn runtime. Do you run docker in privileged mode?
failed to open rknpu module, need to insmod rknpu dirver!
Looks like there are not enough permissions, privileged mode makes changes to cgroup
Maybe the problem is a missing capability. Try adding: --cap-add SYS_MODULE
. This allows the container to load kernel modules.
If this does not work, try adding --cap-add ALL
. This adds all capabilities. This way we find out, if the reason is a missing capability or not.
Maybe the problem is a missing capability. Try adding:
--cap-add SYS_MODULE
. This allows the container to load kernel modules.If this does not work, try adding
--cap-add ALL
. This adds all capabilities. This way we find out, if the reason is a missing capability or not.
I already tried this, it didn't help. Exactly the same error is caused by launching rknpu without sudo
This issue was accidently closed, but the problem has still not been solved. It should be reopened.
This issue depends on https://github.com/home-assistant/supervisor/issues/4863. Unfortunately, I don't know if the developers have time to add this feature in the near future. This blocks the Rockchip addon.
I think the fix is quite simple. We need to add an option systempaths: false/ true
to the config.yml
of the addon. Default value should be false to ensure backward compatibility. If set to true, the option --security-opt systempaths=unconfined
should be added to the container.
I am not very familiar with the Home Assistant source code, and I have currently no time to work on this. Maybe someone is interested in contributing this feature. This could perhaps speed things up a bit.
I am not sure if this solves the issue completely (maybe there are some other changes necessary) but I am quite convinced that this would be a big step in finally having a rockchip addon in HA.
Description
I can`t start the add on on 13 beta-6 version when testing the new rknn detector. I am using an orangepi 5 with a Rockchip RK3588S. But i gives the following error when starting.
The file does exists from a ssh connection to the orangepi, but i don`t know if is really a bug or i am missing any configuration to make the docker container access this file (I already disabled protection mode).