search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.52k stars 1.68k forks source link

[Detector Support]: Intel NCS2 on a Khadas Vim 3 #9371

Closed alucryd closed 7 months ago

alucryd commented 8 months ago

Describe the problem you are having

Trying to use the NSC2 on a Khadas Vim3 running Armbian, I can't get the thing to work. The same stick works fine on an Intel server, using the exact same configuration.

Version

0.12.1-367d724

Frigate config file

detectors:
#  cpu1:
#    type: cpu
#    num_threads: 2
#  cpu2:
#    type: cpu
#    num_threads: 2
  ncs2:
    type: openvino
    device: MYRIAD
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

docker-compose file or Docker CLI command

docker run --rm -it --volume /etc/localtime:/etc/localtime:ro --volume /var/lib/frigate:/config --volume /mnt/public/Videos/Cameras:/media/frigate --device /dev/dri:/dev/dri --device /dev/video0:/dev/video0 --device-cgroup-rule='c 189:* rmw' --volume /dev/bus/usb:/dev/bus/usb --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 --publish 5000:5000 ghcr.io/blakeblackshear/frigate:stable

Relevant log output

2024-01-17 19:50:00.625753751  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2024-01-17 19:50:00.746076239  [INFO] Starting go2rtc...
2024-01-17 19:50:00.868000317  19:50:00.867 INF go2rtc version 1.2.0 linux/arm64
2024-01-17 19:50:00.868714028  19:50:00.868 INF [api] listen addr=:1984
2024-01-17 19:50:00.869871908  19:50:00.869 INF [rtsp] listen addr=:8554
2024-01-17 19:50:00.870909037  19:50:00.870 INF [srtp] listen addr=:8443
2024-01-17 19:50:00.871515664  19:50:00.871 INF [webrtc] listen addr=:8555
2024-01-17 19:50:02.718287405  [2024-01-17 19:50:02] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2024-01-17 19:50:02.786812058  [2024-01-17 19:50:02] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2024-01-17 19:50:02.813813460  [2024-01-17 19:50:02] peewee_migrate                 INFO    : Starting migrations
2024-01-17 19:50:02.822090201  [2024-01-17 19:50:02] peewee_migrate                 INFO    : There is nothing to migrate
2024-01-17 19:50:02.856205923  [2024-01-17 19:50:02] detector.ncs2                  INFO    : Starting detection process: 360
2024-01-17 19:50:02.862526907  [2024-01-17 19:50:02] frigate.app                    INFO    : Output process started: 362
2024-01-17 19:50:02.877594343  [2024-01-17 19:50:02] frigate.app                    INFO    : Camera processor started for living_room: 364
2024-01-17 19:50:02.889373224  [2024-01-17 19:50:02] frigate.app                    INFO    : Camera processor started for master_bedroom: 366
2024-01-17 19:50:02.900176101  [2024-01-17 19:50:02] frigate.app                    INFO    : Camera processor started for doorbell: 367
2024-01-17 19:50:02.912440526  [2024-01-17 19:50:02] frigate.app                    INFO    : Camera processor started for front_yard: 368
2024-01-17 19:50:02.922145232  [2024-01-17 19:50:02] frigate.app                    INFO    : Capture process started for living_room: 370
2024-01-17 19:50:02.934762908  [2024-01-17 19:50:02] frigate.app                    INFO    : Capture process started for master_bedroom: 373
2024-01-17 19:50:02.946493289  [2024-01-17 19:50:02] frigate.app                    INFO    : Capture process started for doorbell: 375
2024-01-17 19:50:02.971710017  [2024-01-17 19:50:02] frigate.app                    INFO    : Capture process started for front_yard: 383
2024-01-17 19:50:10.144535820  [INFO] Starting go2rtc healthcheck service...
2024-01-17 19:50:25.940100204  E: [ncAPI] [    425940] [frigate.detecto] ncDeviceOpen:1013  Failed to find booted device after boot
2024-01-17 19:50:25.943939593  Process detector:ncs2:
2024-01-17 19:50:25.945690142  Traceback (most recent call last):
2024-01-17 19:50:25.945731642    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-01-17 19:50:25.945734142      self.run()
2024-01-17 19:50:25.945736808    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-01-17 19:50:25.945747267      self._target(*self._args, **self._kwargs)
2024-01-17 19:50:25.945750684    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2024-01-17 19:50:25.945805475      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-01-17 19:50:25.945808350    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2024-01-17 19:50:25.945811475      self.detect_api = create_detector(detector_config)
2024-01-17 19:50:25.945813684    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2024-01-17 19:50:25.945815434      return api(detector_config)
2024-01-17 19:50:25.945817684    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__
2024-01-17 19:50:25.945869517      self.interpreter = self.ov_core.compile_model(
2024-01-17 19:50:25.945872392    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 387, in compile_model
2024-01-17 19:50:25.945895434      super().compile_model(model, device_name, {} if config is None else config),
2024-01-17 19:50:25.945948184  RuntimeError: Failed to allocate graph: MYRIAD device is not opened.
2024-01-17 19:50:33.571318349  [2024-01-17 19:50:33] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting Frigate...

Operating system

Other Linux

Install method

Docker CLI

Coral version

Other

Any other information that may be helpful

Tried podman as well (which I'm using on the main server) to no avail. Also disabled AppArmor on the SBC just in case, but it didn't help.

I can see the stick changing descriptors in the dmesg output, but it gets disconnected fairly quickly.

NickM-27 commented 8 months ago

/dev/bus/usb should be passed as a device not volume

alucryd commented 8 months ago

I'm quite sure it should be a volume, it works as a volume on my server, doesn't work when passed as a device. But I can give it a try.

Official instructions also pass it as a volume: https://docs.frigate.video/configuration/detectors#intel-ncs2-vpu-and-myriad-x-setup

NickM-27 commented 8 months ago

ah I see, interesting

alucryd commented 8 months ago

Intel has documentation on how to run it on ARM 64 devices (https://www.intel.com/content/www/us/en/developer/articles/technical/arm64-sbc-and-ncs2.html) so in theory it should be able to work. Maybe it's just that the aarch64 frigate docker image does not ship with the OpenVINO SDK? I'll check that tomorrow.

NickM-27 commented 8 months ago

it does ship with it, there were users testing it on raspberry pis https://github.com/blakeblackshear/frigate/discussions/3797

alucryd commented 8 months ago

I do have a pi lying around, I'll give it a try there.

alucryd commented 8 months ago

Sadly I can reproduce on a Pi 4 freshly flashed with Rpi OS Bookworm aarch64. I'll try poking people on the old thread, see if I'm alone or not.

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

alucryd commented 7 months ago

Problem is still very much there, please don't close.

NickM-27 commented 7 months ago

I don't think this is going to be fixed. Intel has ended support for the NCS2 last year and openvino library does not work with it any more

alucryd commented 7 months ago

Oh, didn't know the NCS2 had been abandoned by Intel, it's sad but I get it. Maybe someone smarter than me with the will and free time will find a way to get it working again. I'll place my hopes on the Khadas Vim 3 TPU getting its way into the linux kernel and become a valid alternative. Or just get a coral, they seem to finally be back in stock.