Closed aeozyalcin closed 1 year ago
I am on J5005 too. The inference about 40-50ms with yolox-tiny 416x416. I tried the yolov8n 416x416, it has ~30ms, but significantly worst the detection confidence, so switched back to yolox. Maybe I do something wrong with the convertation.
Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?
I am on a J5005 getting 32-34ms on yolov8n 320x320 - but need to tweak detection thresholds. What is your config.yml?
I am using a custom trained model, so my confidence threshold probably wouldn't apply. But in my case, min score is 0.55 and threshold is 0.7. So not that different from the defaults.
In my case, my model positively identifies subjects with 90%+ confidence, so I can bring up the min score and thresholds to eliminate false positives, if needed. So far so good though.
I am also surprised your inference still takes ~30ms with a 320x320 model. I would have expected more like 20-25ms. Did you export the model as a 320x320 from the Google Collab notebook?
Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?
Synology with Intel Celeron J4125 processor Performance yolon 416x416 about 40ms Perfomance yolox 416x416 about 10 times higher Performance default 300*300 stock OpenVino model around 20ms
Haven't used yolox due to too high latency.
Accuracy seems very good on yolon. But the most interesting part is that I see many more power usage spikes. Almost 10Watt spikes when there is a lot of movement, which I have never seen on a Google Coral
or on the stock SSD OpenVino model
. So I am hesitating now to go back to the stock model.
Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?
Synology with Intel Celeron J4125 processor Performance yolon 416x416 about 40ms Perfomance yolox 416x416 about 10 times higher Performance default 300*300 stock OpenVino model around 20ms
Haven't used yolox due to too high latency. Accuracy seems very good on yolon. But the most interesting part is that I see many more power usage spikes. Almost 10Watt spikes when there is a lot of movement, which I have never seen on a
Google Coral
or on thestock SSD OpenVino model
. So I am hesitating now to go back to the stock model.
YOLOx at 416x416 shouldn't be 10x higher inference time than YOLOv8n. When exporting your YOLOx model, make sure to export it in FP16 precision.
Let's take the guessing out of the game. I have made this Colab for you guys. Give this a try and let me know. Convert YOLOv8 to OpenVino for Frigate.ipynb
@aeozyalcin I'm trying to use a non-square input size (640 width, 480 height) in your colab, and it's not working with Frigate. I did have it working with a square size, FYI. I figured it would have working since you included the width and height variables instead of just a imgsz scalar. Here is the error from the Frigate log:
2023-07-27 17:26:02.999457375 Process camera_processor:car_port: 2023-07-27 17:26:03.000789574 Traceback (most recent call last): 2023-07-27 17:26:03.001129326 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2023-07-27 17:26:03.001132707 self.run() 2023-07-27 17:26:03.001252905 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2023-07-27 17:26:03.001255325 self._target(*self._args, **self._kwargs) 2023-07-27 17:26:03.001337190 File "/opt/frigate/frigate/video.py", line 480, in track_camera 2023-07-27 17:26:03.001338960 process_frames( 2023-07-27 17:26:03.001412145 File "/opt/frigate/frigate/video.py", line 725, in process_frames 2023-07-27 17:26:03.001413449 detect( 2023-07-27 17:26:03.001474952 File "/opt/frigate/frigate/video.py", line 546, in detect 2023-07-27 17:26:03.001476399 region_detections = object_detector.detect(tensor_input) 2023-07-27 17:26:03.001532892 File "/opt/frigate/frigate/object_detection.py", line 208, in detect 2023-07-27 17:26:03.001543305 self.np_shm[:] = tensor_input[:] 2023-07-27 17:26:03.001634772 ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
I think I tried every combination of width and height order on the following lines, and that didn't work.
model.export(format="onnx", imgsz=[input_height,input_width], optimize=optimize_cpu)
!python3 /usr/local/bin/mo --input_model /content/{model_name}.onnx --model_name {model_name} -s 255 --reverse_input_channels --compress_to_fp16 --input_shape [1,3,{input_width},{input_height}] --output_dir /content/{zipname}
Here is how I set up my model, and I also tried playing with these values to no avail with a non-square input.
model: width: 640 height: 480 input_tensor: nchw input_pixel_format: bgr model_type: yolov8 labelmap_path: /yolov8s/coco_80cl.txt
Is this a bug in the Frigate code?
I don't believe YOLO models work with non-square dimensions. I don't think this is an issue with Frigate or Openvino. Why are you wanting non-square dimensions?
The Ultralytics model.export function supports exporting to a non-square size. https://docs.ultralytics.com/modes/export/
OpenVINO has this page about setting input shapes with examples of converting ONNX models with non-square shapes to OpenVINO. This page also gives the reason for doing so: "With model conversion API you can increase your model’s efficiency by providing an additional shape definition, with these two parameters: input_shape and static_shape." https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model.html
My input images usually have a non-square aspect ratio, so I want to use non-square dimensions to increase efficiency.
The error given by Frigate seems like a basic programming error when trying to transfer the tensor to memory, unrelated to YOLO. I could be mistaken, though.
tensor_input[:]
Looking into this closer, maybe there is something up on the frigate backend side that's causing that error. That error happens before the openvino plugin is ever called. I suspect there could be a bug with handling non-square dimensions on the Frigate side.
@blakeblackshear any clues?
@drolex2 have you tried any other model (not yolo, or not openvino at all) with non-square dimensions, and gotten it to work?
I would suggest trying to run this on 0.13, I believe we have had some changes / refactoring that may have fixed this
Any ETA on when v13 RC builds are getting released?
Beta is coming soon™
Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?
core i3 8100 8gb ram model size 416416 9cameras 4MP average raw motion recording 640480 for detect yolov8n: inference speed <20ms 20% cpu yolov8s: inference speed <45ms 20% cpu yolov8m: inference speed <100ms 80% cpu
gpu still a few % only (integrated graphics card)
I can not say about accuracy for now, it is a completely new setup for me (coming from ispy)
hello @aeozyalcin, I am trying to get J5005/openvino working with frigate and struggling to get inference speed lower than 148ms for the bundled ssdlite_mobilenet_v2.xml model.
Is it possible for you to provide some insights and sample config you are using and the inference you are getting with J5005?
thanks!
@sumitc96 probably your openvino config is wrong in some way? i am using i5-6500T and i can get < 10ms inference speed with the default model. try to post your config.
@ivanjx Here's my config, nothing special, also if I set the openvino device as GPU, frigate crashes.
I wonder whether there is something wrong in my GPU acceleration config and openvino is defaulting to CPU.
I am trying to get this run in docker running in a proxmox LXC.
`database: path: /db/frigate.db mqtt: # If you have a MQTT server then update this section, if not delete it. host: 192.168.55.2 user: x password: x detectors: openvino: type: openvino device: AUTO model: path: /openvino-model/ssdlite_mobilenet_v2.xml model: width: 300 height: 300 input_tensor: nhwc input_pixel_format: bgr labelmap_path: /openvino-model/coco_91cl_bkgr.txt cameras: doorbell: # <------ Name the camera ffmpeg: hwaccel_args: preset-vaapi inputs:
path: rtsp://x:x@192.168.43.179:554/h264Preview_01_main # Update with your camera info (Normally the higher res stream, used for actual recording) roles:
record: # what you want Frigate to record enabled: True
events: # this is the default recording mode where events are recorded, the number of days it saves events for is 10. retain: mode: active_objects default: 10
snapshots: enabled: True retain: default: 10 objects: # Object to detect - https://docs.frigate.video/configuration/objects track:
What kernel version are you on? I upgraded to the most recent proxmox kernel on my j5004 recently and Frigate/openvino can no longer find the GPU, despite hours messing around with drivers. I migrate the lxc to my 6500-T or my Nuc 11 and it works instantly. Seems to be a bug with the J5005 and latest Linux kernels support for Gemini Lake. I didn't have a solution and gave up (it runs fine on my 6500-T) but flagging in case it's the same issue you are having
I am using kernel 6.2, came default with latest proxmox 8.0. I am hoping I can get a working configuration for J5005 as even the documentation suggests 25ms inference speeds with Gemini Lake CPUs.
https://docs.frigate.video/frigate/hardware/#:~:text=on%20host%20device-,Intel,-Celeron%20J4105
on intel cpu openvino will use the cpu but it is very optimized compared to running it on cpu without openvino so gpu will not work there at least in my case. maybe try to remove the model
section on your config @sumitc96
currently i am using openvino with yolov8 but when i tested for the first time i dont specify any model and frigate picks up the default model for openvino and my inference speed went from 500ms+ down to < 10ms.
I am using kernel 6.2, came default with latest proxmox 8.0. I am hoping I can get a working configuration for J5005 as even the documentation suggests 25ms inference speeds with Gemini Lake CPUs.
https://docs.frigate.video/frigate/hardware/#:~:text=on%20host%20device-,Intel,-Celeron%20J4105
This is the error I get trying to use GPU on my j5005, using the same set up as yours. It started after upgrading to PVE8.
2023-12-03 18:07:33.234865475 Process detector:ov_detector: 2023-12-03 18:07:33.240858880 Traceback (most recent call last): 2023-12-03 18:07:33.240935286 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2023-12-03 18:07:33.240943288 self.run() 2023-12-03 18:07:33.241006584 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2023-12-03 18:07:33.241014931 self._target(*self._args, **self._kwargs) 2023-12-03 18:07:33.241020676 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector 2023-12-03 18:07:33.241031588 object_detector = LocalObjectDetector(detector_config=detector_config) 2023-12-03 18:07:33.241105895 File "/opt/frigate/frigate/object_detection.py", line 53, in init 2023-12-03 18:07:33.241114468 self.detect_api = create_detector(detector_config) 2023-12-03 18:07:33.241120346 File "/opt/frigate/frigate/detectors/init.py", line 18, in create_detector 2023-12-03 18:07:33.241124900 return api(detector_config) 2023-12-03 18:07:33.241130653 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in init 2023-12-03 18:07:33.241215915 self.interpreter = self.ov_core.compile_model( 2023-12-03 18:07:33.241301938 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model 2023-12-03 18:07:33.241311021 super().compile_model(model, device_name, {} if config is None else config), 2023-12-03 18:07:33.241316703 RuntimeError: cldnn program build failed! [GPU] out of GPU resources
I haven't tried CPU as it was easier for me to migrate to another node, but posting here in case anyone else is having the same issue (as I suspect its related to yours). I have tested with both i915, i965, Vaapi and QSV.
@nickp27 Thanks so much for reporting this. Yes, hopefully others experiencing the same can report this so that this issue can gain adequate traction/attention and hopefully someone more experienced than me can help figure out a solution.
Thus at this time looks like J5005 with proxmox8 doesn't play well with openvino/frigate.
@nickp27 btw the error I got when selecting GPU as openvino device is different than what you are seeing..
2023-12-06 03:41:42.248822993 Process detector:openvino: 2023-12-06 03:41:42.252603217 Traceback (most recent call last): 2023-12-06 03:41:42.252630951 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2023-12-06 03:41:42.252633215 self.run() 2023-12-06 03:41:42.252635008 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2023-12-06 03:41:42.252640388 self._target(*self._args, **self._kwargs) 2023-12-06 03:41:42.252643721 File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector 2023-12-06 03:41:42.252679528 object_detector = LocalObjectDetector(detector_config=detector_config) 2023-12-06 03:41:42.252697655 File "/opt/frigate/frigate/object_detection.py", line 52, in __init__ 2023-12-06 03:41:42.252700026 self.detect_api = create_detector(detector_config) 2023-12-06 03:41:42.252701677 File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector 2023-12-06 03:41:42.252702965 return api(detector_config) 2023-12-06 03:41:42.252704649 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__ 2023-12-06 03:41:42.252706142 self.interpreter = self.ov_core.compile_model( 2023-12-06 03:41:42.252708021 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 387, in compile_model 2023-12-06 03:41:42.252710863 super().compile_model(model, device_name, {} if config is None else config), 2023-12-06 03:41:42.252874663 RuntimeError: cldnn program build failed! clWaitForEvents, error code: -14
@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU.
If you would like to give your J5005 a try again, here's the steps:
kernel downgrade instructions:
add pve 7(bullseye) in sources.list root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
install the old kernel.
apt install pve-kernel-5.11.22-7-pve
pin the kernel (i think boot by default)
pve-efiboot-tool kernel pin 5.11.22-7-pve
4.reboot
I am super late to this party as I am only starting frigate now. IS there someone that can perhaps guide me on how to get Yolo working?
@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU.
If you would like to give your J5005 a try again, here's the steps:
kernel downgrade instructions:
1. add pve 7(bullseye) in sources.list root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription 2. install the old kernel. `apt install pve-kernel-5.11.22-7-pve` 3. pin the kernel (i think boot by default) `pve-efiboot-tool kernel pin 5.11.22-7-pve`
4.reboot
5. now i am able to use GPU in openvino with inference time lowered to 35ms! detectors: openvino: type: openvino device: GPU
Were you able to identify the cause of this? Has it been upstream at all? Given how popular the J5005 is for Frigate, should we be raising an issue or flagging in the docs to avoid PVE8?
@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU. If you would like to give your J5005 a try again, here's the steps: kernel downgrade instructions:
1. add pve 7(bullseye) in sources.list root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription 2. install the old kernel. `apt install pve-kernel-5.11.22-7-pve` 3. pin the kernel (i think boot by default) `pve-efiboot-tool kernel pin 5.11.22-7-pve`
4.reboot
5. now i am able to use GPU in openvino with inference time lowered to 35ms! detectors: openvino: type: openvino device: GPU
Were you able to identify the cause of this? Has it been upstream at all? Given how popular the J5005 is for Frigate, should we be raising an issue or flagging in the docs to avoid PVE8?
No, unfortunately I am not that techsavvy to further isolate/identify the problem. Once GPU detector started working after kernel downgrade I stopped researching this.
@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU. If you would like to give your J5005 a try again, here's the steps: kernel downgrade instructions:
1. add pve 7(bullseye) in sources.list root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription 2. install the old kernel. `apt install pve-kernel-5.11.22-7-pve` 3. pin the kernel (i think boot by default) `pve-efiboot-tool kernel pin 5.11.22-7-pve`
4.reboot
5. now i am able to use GPU in openvino with inference time lowered to 35ms! detectors: openvino: type: openvino device: GPU
Were you able to identify the cause of this? Has it been upstream at all? Given how popular the J5005 is for Frigate, should we be raising an issue or flagging in the docs to avoid PVE8?
No, unfortunately I am not that techsavvy to further isolate/identify the problem. Once GPU detector started working after kernel downgrade I stopped researching this.
It looks like there are a couple of issues dealing with it, but in the meantime, I have gone with kernel 5.15, with some grub tweaks.
Hi. Can you please help me with yolo models? I've tried to train my own accornig @aeozyalcin notebook and convert it. I even tried to just download default yolov8* model according this notebook Convert YOLOv8 to OpenVino for Frigate.ipynb
Every time i'm getting an error from frigate on startup:
2024-02-28 10:21:18.665358694 [2024-02-28 13:21:18] frigate.detectors.plugins.openvino INFO : Model Input Shape: [1,3,320,320]
2024-02-28 10:21:18.665403574 [2024-02-28 13:21:18] frigate.detectors.plugins.openvino INFO : Model Output-0 Shape: [1,84,2100]
2024-02-28 10:21:18.665442681 [2024-02-28 13:21:18] frigate.detectors.plugins.openvino INFO : Model has 1 Output Tensors
2024-02-28 10:21:19.988495509 Process detector:ov:
2024-02-28 10:21:19.988500747 Traceback (most recent call last):
2024-02-28 10:21:19.988502267 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-02-28 10:21:19.988503394 self.run()
2024-02-28 10:21:19.988504666 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-02-28 10:21:19.988507373 self._target(*self._args, **self._kwargs)
2024-02-28 10:21:19.988512096 File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
2024-02-28 10:21:19.988513470 detections = object_detector.detect_raw(input_frame)
2024-02-28 10:21:19.988547388 File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
2024-02-28 10:21:19.988548949 return self.detect_api.detect_raw(tensor_input=tensor_input)
2024-02-28 10:21:19.988550325 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 92, in detect_raw
2024-02-28 10:21:19.988589437 if object_detected[0] != -1:
2024-02-28 10:21:19.988591040 IndexError: invalid index to scalar variable.
my config is:
detectors:
ov:
type: openvino
device: AUTO
model:
path: /media/frigate/openvino/custom/yolov8x.xml
model:
width: 320
height: 320
input_tensor: nchw
input_pixel_format: bgr
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
I really not clue what i'm doing wrong but it seems that i can't launch any YOLO model instead of frigate's OpenVINO default model.
Can you please point where is the problem? Frigate version: 0.13.2
model_type: yolox
sorry... Forgot to set model type.
Now iI have another problem with detection Detected object labeling worng. I.e. now car is a person, person is a car and dog is someone else. And sometimes random.
Any ideas about this issue?
Now iI have another problem with detection Detected object labeling worng. I.e. now car is a person, person is a car and dog is someone else. And sometimes random.
Any ideas about this issue?
Your labelmap_path seems to be the wrong one. For YOLO you need to use coco_80cl.txt. Hope that helps with object labeling.
Describe the problem you are having
Awesome job getting OpenVINO into Frigate @NateMeyer! I just got a thin client with a J5005, and OpenVINO support was perfectly timed!
I tried using the
yolox-tiny
model with OpenVINO, and got the following error:Looks like there is an incompatibility between how Frigate sends frames to the yolo model, and what the model expects as its input. When I use the included
ssdlite_mobilenet_v2
model, object detection is working as expected.P.S.: I also tried Intel's own person-detection-0201 model. Frigate ran without errors, but no detections were showing up. I don't think that's related, but just a data point.
Version
0.12.0-12D51D3
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Other Linux
Install method
HassOS Addon
Coral version
Other
Network connection
Wired
Camera make and model
Reolink RLC-810A
Any other information that may be helpful
No response