[Support]: OpenVINO Does not work with YOLOX model

aeozyalcin commented 1 year ago

Describe the problem you are having

Awesome job getting OpenVINO into Frigate @NateMeyer! I just got a thin client with a J5005, and OpenVINO support was perfectly timed!

I tried using the yolox-tiny model with OpenVINO, and got the following error:

2023-01-21 20:32:53.946348978  [2023-01-21 12:32:53] detector.ov                    INFO    : Starting detection process: 376
2023-01-21 20:32:55.017048607  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 3, 416, 416}
2023-01-21 20:32:55.017056288  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 3549, 85}
2023-01-21 20:32:55.017058803  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2023-01-21 20:32:55.019078340  Process detector:ov:
2023-01-21 20:32:55.042640498  Traceback (most recent call last):
2023-01-21 20:32:55.042646939    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-01-21 20:32:55.042648687      self.run()
2023-01-21 20:32:55.042650805    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-01-21 20:32:55.042652584      self._target(*self._args, **self._kwargs)
2023-01-21 20:32:55.042654630    File "/opt/frigate/frigate/object_detection.py", line 120, in run_detector
2023-01-21 20:32:55.042656489      detections = object_detector.detect_raw(input_frame)
2023-01-21 20:32:55.042658500    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2023-01-21 20:32:55.042660363      return self.detect_api.detect_raw(tensor_input=tensor_input)
2023-01-21 20:32:55.042662445    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 45, in detect_raw
2023-01-21 20:32:55.042664053      infer_request.infer([tensor_input])
2023-01-21 20:32:55.042666257    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 151, in infer
2023-01-21 20:32:55.042667807      return super().infer(
2023-01-21 20:32:55.042670418  RuntimeError: Can't set input blob with name: images, because model input (shape={1,3,416,416}) and blob (shape=(1.416.416.3)) are incompatible

Looks like there is an incompatibility between how Frigate sends frames to the yolo model, and what the model expects as its input. When I use the included ssdlite_mobilenet_v2 model, object detection is working as expected.

P.S.: I also tried Intel's own person-detection-0201 model. Frigate ran without errors, but no detections were showing up. I don't think that's related, but just a data point.

Version

0.12.0-12D51D3

Frigate config file

mqtt:
  enabled: False

detectors:
  ov:
    type: openvino
    device: GPU
    model:
      path: /media/frigate/openvino-model/yolox-tiny.xml   

model:
  width: 416
  height: 416
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /media/frigate/openvino-model/coco_80cl.txt 

cameras:
  front_driveway:
    objects:
      track:
        - person
    ffmpeg:
      inputs:
        - path: http://192.168.0.241/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password=password
          roles:
            - detect
            - restream
          hwaccel_args: preset-intel-qsv-h264
          input_args:
            - -strict
            - experimental
            - -analyzeduration
            - 1000M
            - -probesize
            - 1000M
            - -rw_timeout
            - "5000000"

    detect:
      enabled: True
      width: 896
      height: 512
      fps: 10

    zones:

      lower_driveway:
        coordinates: 480,191,486,81,320,92,191,136,185,178,0,242,0,377
        objects:
          - person
        filters:
          person:
            min_area: 1000
            max_area: 10000
      middle_driveway:
        coordinates: 83,400,704,326,784,228,798,92,478,69,474,182,0,372
        objects:
          - person
        filters:
          person:
            min_area: 2000
            max_area: 20000          
      upper_driveway:
        coordinates: 896,512,896,123,783,89,776,227,703,326,83,400,0,372,0,512
        objects:
          - person
        filters:
          person:
            min_area: 4500
            max_area: 100000   

    motion:
      mask:
        - 201,113,398,36,896,38,896,36,896,0,0,0,0,165
      improve_contrast: True
      contour_area: 25

    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: save a clean PNG copy of the snapshot image (default: shown below)
      clean_copy: True
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: False
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: True
      # Optional: crop the snapshot (default: shown below)
      crop: False
      required_zones:
        - upper_driveway
        - middle_driveway
        - lower_driveway
      # Optional: height to resize the snapshot to (default: original size)
      # height: 175
      # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
        # - driveway_entrance
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 2
        # objects:
        #   car: 7
    record:
      # Optional: Enable recording (default: global setting)
      enabled: True
      # Optional: Number of days to retain (default: global setting)
      retain:
        days: 1
      events:
        objects:
          - person
        required_zones:
          - upper_driveway
          - middle_driveway
          - lower_driveway
        retain:
          default: 2

Relevant log output

2023-01-21 20:32:53.946348978  [2023-01-21 12:32:53] detector.ov                    INFO    : Starting detection process: 376
2023-01-21 20:32:55.017048607  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 3, 416, 416}
2023-01-21 20:32:55.017056288  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 3549, 85}
2023-01-21 20:32:55.017058803  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2023-01-21 20:32:55.019078340  Process detector:ov:
2023-01-21 20:32:55.042640498  Traceback (most recent call last):
2023-01-21 20:32:55.042646939    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-01-21 20:32:55.042648687      self.run()
2023-01-21 20:32:55.042650805    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-01-21 20:32:55.042652584      self._target(*self._args, **self._kwargs)
2023-01-21 20:32:55.042654630    File "/opt/frigate/frigate/object_detection.py", line 120, in run_detector
2023-01-21 20:32:55.042656489      detections = object_detector.detect_raw(input_frame)
2023-01-21 20:32:55.042658500    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2023-01-21 20:32:55.042660363      return self.detect_api.detect_raw(tensor_input=tensor_input)
2023-01-21 20:32:55.042662445    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 45, in detect_raw
2023-01-21 20:32:55.042664053      infer_request.infer([tensor_input])
2023-01-21 20:32:55.042666257    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 151, in infer
2023-01-21 20:32:55.042667807      return super().infer(
2023-01-21 20:32:55.042670418  RuntimeError: Can't set input blob with name: images, because model input (shape={1,3,416,416}) and blob (shape=(1.416.416.3)) are incompatible

FFprobe output from your camera

"[\n  {\n    \"return_code\": 0,\n    \"stderr\": {},\n    \"stdout\": {\n      \"programs\": [],\n      \"streams\": [\n        {\n          \"avg_frame_rate\": \"19/1\",\n          \"codec_long_name\": \"H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10\",\n          \"height\": 512,\n          \"width\": 896\n        },\n        {\n          \"avg_frame_rate\": \"0/0\",\n          \"codec_long_name\": \"AAC (Advanced Audio Coding)\"\n        }\n      ]\n    }\n  }\n]"

Frigate stats

No response

Operating system

Other Linux

Install method

HassOS Addon

Coral version

Other

Network connection

Wired

Camera make and model

Reolink RLC-810A

Any other information that may be helpful

No response

KohajdaG commented 1 year ago

I am on J5005 too. The inference about 40-50ms with yolox-tiny 416x416. I tried the yolov8n 416x416, it has ~30ms, but significantly worst the detection confidence, so switched back to yolox. Maybe I do something wrong with the convertation.

aeozyalcin commented 1 year ago

Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?

I am on a J5005 getting 32-34ms on yolov8n 320x320 - but need to tweak detection thresholds. What is your config.yml?

I am using a custom trained model, so my confidence threshold probably wouldn't apply. But in my case, min score is 0.55 and threshold is 0.7. So not that different from the defaults.

In my case, my model positively identifies subjects with 90%+ confidence, so I can bring up the min score and thresholds to eliminate false positives, if needed. So far so good though.

I am also surprised your inference still takes ~30ms with a 320x320 model. I would have expected more like 20-25ms. Did you export the model as a 320x320 from the Google Collab notebook?

Caros2017 commented 1 year ago

Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?

Synology with Intel Celeron J4125 processor Performance yolon 416x416 about 40ms Perfomance yolox 416x416 about 10 times higher Performance default 300*300 stock OpenVino model around 20ms

Haven't used yolox due to too high latency. Accuracy seems very good on yolon. But the most interesting part is that I see many more power usage spikes. Almost 10Watt spikes when there is a lot of movement, which I have never seen on a Google Coral or on the stock SSD OpenVino model. So I am hesitating now to go back to the stock model.

aeozyalcin commented 1 year ago

Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?

Synology with Intel Celeron J4125 processor Performance yolon 416x416 about 40ms Perfomance yolox 416x416 about 10 times higher Performance default 300*300 stock OpenVino model around 20ms

Haven't used yolox due to too high latency. Accuracy seems very good on yolon. But the most interesting part is that I see many more power usage spikes. Almost 10Watt spikes when there is a lot of movement, which I have never seen on a Google Coral or on the stock SSD OpenVino model. So I am hesitating now to go back to the stock model.

YOLOx at 416x416 shouldn't be 10x higher inference time than YOLOv8n. When exporting your YOLOx model, make sure to export it in FP16 precision.

drolex2 commented 1 year ago

Let's take the guessing out of the game. I have made this Colab for you guys. Give this a try and let me know. Convert YOLOv8 to OpenVino for Frigate.ipynb

@aeozyalcin I'm trying to use a non-square input size (640 width, 480 height) in your colab, and it's not working with Frigate. I did have it working with a square size, FYI. I figured it would have working since you included the width and height variables instead of just a imgsz scalar. Here is the error from the Frigate log:

2023-07-27 17:26:02.999457375 Process camera_processor:car_port: 2023-07-27 17:26:03.000789574 Traceback (most recent call last): 2023-07-27 17:26:03.001129326 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2023-07-27 17:26:03.001132707 self.run() 2023-07-27 17:26:03.001252905 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2023-07-27 17:26:03.001255325 self._target(*self._args, **self._kwargs) 2023-07-27 17:26:03.001337190 File "/opt/frigate/frigate/video.py", line 480, in track_camera 2023-07-27 17:26:03.001338960 process_frames( 2023-07-27 17:26:03.001412145 File "/opt/frigate/frigate/video.py", line 725, in process_frames 2023-07-27 17:26:03.001413449 detect( 2023-07-27 17:26:03.001474952 File "/opt/frigate/frigate/video.py", line 546, in detect 2023-07-27 17:26:03.001476399 region_detections = object_detector.detect(tensor_input) 2023-07-27 17:26:03.001532892 File "/opt/frigate/frigate/object_detection.py", line 208, in detect 2023-07-27 17:26:03.001543305 self.np_shm[:] = tensor_input[:] 2023-07-27 17:26:03.001634772 ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)

I think I tried every combination of width and height order on the following lines, and that didn't work. model.export(format="onnx", imgsz=[input_height,input_width], optimize=optimize_cpu) !python3 /usr/local/bin/mo --input_model /content/{model_name}.onnx --model_name {model_name} -s 255 --reverse_input_channels --compress_to_fp16 --input_shape [1,3,{input_width},{input_height}] --output_dir /content/{zipname}

Here is how I set up my model, and I also tried playing with these values to no avail with a non-square input. model: width: 640 height: 480 input_tensor: nchw input_pixel_format: bgr model_type: yolov8 labelmap_path: /yolov8s/coco_80cl.txt

Is this a bug in the Frigate code?

aeozyalcin commented 1 year ago

I don't believe YOLO models work with non-square dimensions. I don't think this is an issue with Frigate or Openvino. Why are you wanting non-square dimensions?

drolex2 commented 1 year ago

The Ultralytics model.export function supports exporting to a non-square size. https://docs.ultralytics.com/modes/export/

OpenVINO has this page about setting input shapes with examples of converting ONNX models with non-square shapes to OpenVINO. This page also gives the reason for doing so: "With model conversion API you can increase your model’s efficiency by providing an additional shape definition, with these two parameters: input_shape and static_shape." https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model.html

My input images usually have a non-square aspect ratio, so I want to use non-square dimensions to increase efficiency.

The error given by Frigate seems like a basic programming error when trying to transfer the tensor to memory, unrelated to YOLO. I could be mistaken, though.

aeozyalcin commented 1 year ago

tensor_input[:]

Looking into this closer, maybe there is something up on the frigate backend side that's causing that error. That error happens before the openvino plugin is ever called. I suspect there could be a bug with handling non-square dimensions on the Frigate side.

@blakeblackshear any clues?

@drolex2 have you tried any other model (not yolo, or not openvino at all) with non-square dimensions, and gotten it to work?

NickM-27 commented 1 year ago

I would suggest trying to run this on 0.13, I believe we have had some changes / refactoring that may have fixed this

aeozyalcin commented 1 year ago

Any ETA on when v13 RC builds are getting released?

NickM-27 commented 1 year ago

Beta is coming soon™

xavan83 commented 9 months ago

Out of curiosity, what inference time are people getting, and on what hardware are you running it on? And how do you feel the YOLOv8 accuracy/performance is compared to the stock SSD OpenVINO model?

core i3 8100 8gb ram model size 416416 9cameras 4MP average raw motion recording 640480 for detect yolov8n: inference speed <20ms 20% cpu yolov8s: inference speed <45ms 20% cpu yolov8m: inference speed <100ms 80% cpu

gpu still a few % only (integrated graphics card)

I can not say about accuracy for now, it is a completely new setup for me (coming from ispy)

sumitc96 commented 8 months ago

hello @aeozyalcin, I am trying to get J5005/openvino working with frigate and struggling to get inference speed lower than 148ms for the bundled ssdlite_mobilenet_v2.xml model.

Is it possible for you to provide some insights and sample config you are using and the inference you are getting with J5005?

thanks!

ivanjx commented 8 months ago

@sumitc96 probably your openvino config is wrong in some way? i am using i5-6500T and i can get < 10ms inference speed with the default model. try to post your config.

sumitc96 commented 8 months ago

@ivanjx Here's my config, nothing special, also if I set the openvino device as GPU, frigate crashes.

I wonder whether there is something wrong in my GPU acceleration config and openvino is defaulting to CPU.

I am trying to get this run in docker running in a proxmox LXC.

`database: path: /db/frigate.db mqtt: # If you have a MQTT server then update this section, if not delete it. host: 192.168.55.2 user: x password: x detectors: openvino: type: openvino device: AUTO model: path: /openvino-model/ssdlite_mobilenet_v2.xml model: width: 300 height: 300 input_tensor: nhwc input_pixel_format: bgr labelmap_path: /openvino-model/coco_91cl_bkgr.txt cameras: doorbell: # <------ Name the camera ffmpeg: hwaccel_args: preset-vaapi inputs:

path: rtsp://x:x@192.168.43.179:554/h264Preview_01_sub # Update with your camera info (Normally the lower res stream, 1280*720 would be a good detection res) roles:
- detect
path: rtsp://x:x@192.168.43.179:554/h264Preview_01_main # Update with your camera info (Normally the higher res stream, used for actual recording) roles:
- record output_args: record: preset-record-generic-audio-copy
  detect: width: 648 # <---- update for your camera's resolution height: 480 # <---- update for your camera's resolution
  width: 640
  
  height: 480
  
  record: # what you want Frigate to record enabled: True
  
  retain: # uncomment this section if you want Frigate to record this camera 24/7. This can use a lot of space, you have been warned!
  
  days: 1
  
  mode: all
  
  events: # this is the default recording mode where events are recorded, the number of days it saves events for is 10. retain: mode: active_objects default: 10
snapshots: enabled: True retain: default: 10 objects: # Object to detect - https://docs.frigate.video/configuration/objects track:
- person record: # how long to record before and after motion events: pre_capture: 7 post_capture: 10 1

nickp27 commented 8 months ago

What kernel version are you on? I upgraded to the most recent proxmox kernel on my j5004 recently and Frigate/openvino can no longer find the GPU, despite hours messing around with drivers. I migrate the lxc to my 6500-T or my Nuc 11 and it works instantly. Seems to be a bug with the J5005 and latest Linux kernels support for Gemini Lake. I didn't have a solution and gave up (it runs fine on my 6500-T) but flagging in case it's the same issue you are having

sumitc96 commented 8 months ago

I am using kernel 6.2, came default with latest proxmox 8.0. I am hoping I can get a working configuration for J5005 as even the documentation suggests 25ms inference speeds with Gemini Lake CPUs.

https://docs.frigate.video/frigate/hardware/#:~:text=on%20host%20device-,Intel,-Celeron%20J4105

ivanjx commented 8 months ago

on intel cpu openvino will use the cpu but it is very optimized compared to running it on cpu without openvino so gpu will not work there at least in my case. maybe try to remove the model section on your config @sumitc96

currently i am using openvino with yolov8 but when i tested for the first time i dont specify any model and frigate picks up the default model for openvino and my inference speed went from 500ms+ down to < 10ms.

nickp27 commented 8 months ago

I am using kernel 6.2, came default with latest proxmox 8.0. I am hoping I can get a working configuration for J5005 as even the documentation suggests 25ms inference speeds with Gemini Lake CPUs.

https://docs.frigate.video/frigate/hardware/#:~:text=on%20host%20device-,Intel,-Celeron%20J4105

This is the error I get trying to use GPU on my j5005, using the same set up as yours. It started after upgrading to PVE8.

2023-12-03 18:07:33.234865475 Process detector:ov_detector: 2023-12-03 18:07:33.240858880 Traceback (most recent call last): 2023-12-03 18:07:33.240935286 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2023-12-03 18:07:33.240943288 self.run() 2023-12-03 18:07:33.241006584 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2023-12-03 18:07:33.241014931 self._target(*self._args, **self._kwargs) 2023-12-03 18:07:33.241020676 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector 2023-12-03 18:07:33.241031588 object_detector = LocalObjectDetector(detector_config=detector_config) 2023-12-03 18:07:33.241105895 File "/opt/frigate/frigate/object_detection.py", line 53, in init 2023-12-03 18:07:33.241114468 self.detect_api = create_detector(detector_config) 2023-12-03 18:07:33.241120346 File "/opt/frigate/frigate/detectors/init.py", line 18, in create_detector 2023-12-03 18:07:33.241124900 return api(detector_config) 2023-12-03 18:07:33.241130653 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in init 2023-12-03 18:07:33.241215915 self.interpreter = self.ov_core.compile_model( 2023-12-03 18:07:33.241301938 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model 2023-12-03 18:07:33.241311021 super().compile_model(model, device_name, {} if config is None else config), 2023-12-03 18:07:33.241316703 RuntimeError: cldnn program build failed! [GPU] out of GPU resources

I haven't tried CPU as it was easier for me to migrate to another node, but posting here in case anyone else is having the same issue (as I suspect its related to yours). I have tested with both i915, i965, Vaapi and QSV.

sumitc96 commented 8 months ago

@nickp27 Thanks so much for reporting this. Yes, hopefully others experiencing the same can report this so that this issue can gain adequate traction/attention and hopefully someone more experienced than me can help figure out a solution.

Thus at this time looks like J5005 with proxmox8 doesn't play well with openvino/frigate.

sumitc96 commented 8 months ago

@nickp27 btw the error I got when selecting GPU as openvino device is different than what you are seeing..

2023-12-06 03:41:42.248822993 Process detector:openvino: 2023-12-06 03:41:42.252603217 Traceback (most recent call last): 2023-12-06 03:41:42.252630951 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2023-12-06 03:41:42.252633215 self.run() 2023-12-06 03:41:42.252635008 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2023-12-06 03:41:42.252640388 self._target(*self._args, **self._kwargs) 2023-12-06 03:41:42.252643721 File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector 2023-12-06 03:41:42.252679528 object_detector = LocalObjectDetector(detector_config=detector_config) 2023-12-06 03:41:42.252697655 File "/opt/frigate/frigate/object_detection.py", line 52, in __init__ 2023-12-06 03:41:42.252700026 self.detect_api = create_detector(detector_config) 2023-12-06 03:41:42.252701677 File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector 2023-12-06 03:41:42.252702965 return api(detector_config) 2023-12-06 03:41:42.252704649 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__ 2023-12-06 03:41:42.252706142 self.interpreter = self.ov_core.compile_model( 2023-12-06 03:41:42.252708021 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 387, in compile_model 2023-12-06 03:41:42.252710863 super().compile_model(model, device_name, {} if config is None else config), 2023-12-06 03:41:42.252874663 RuntimeError: cldnn program build failed! clWaitForEvents, error code: -14

sumitc96 commented 8 months ago

@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU.

If you would like to give your J5005 a try again, here's the steps:

kernel downgrade instructions:

add pve 7(bullseye) in sources.list root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
install the old kernel. apt install pve-kernel-5.11.22-7-pve
pin the kernel (i think boot by default) pve-efiboot-tool kernel pin 5.11.22-7-pve

4.reboot

now i am able to use GPU in openvino with inference time lowered to 35ms! detectors: openvino: type: openvino device: GPU

Frikkie50 commented 7 months ago

I am super late to this party as I am only starting frigate now. IS there someone that can perhaps guide me on how to get Yolo working?

nickp27 commented 7 months ago

@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU.

If you would like to give your J5005 a try again, here's the steps:

kernel downgrade instructions:
1. add pve 7(bullseye) in sources.list
   root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list
   deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

2. install the old kernel.
   `apt install pve-kernel-5.11.22-7-pve`

3. pin the kernel (i think boot by default)
   `pve-efiboot-tool kernel pin 5.11.22-7-pve`
4.reboot
5. now i am able to use GPU in openvino with inference time lowered to 35ms!
   detectors:
   openvino:
   type: openvino
   device: GPU

Were you able to identify the cause of this? Has it been upstream at all? Given how popular the J5005 is for Frigate, should we be raising an issue or flagging in the docs to avoid PVE8?

sumitc96 commented 6 months ago

@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU. If you would like to give your J5005 a try again, here's the steps: kernel downgrade instructions:
1. add pve 7(bullseye) in sources.list
   root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list
   deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

2. install the old kernel.
   `apt install pve-kernel-5.11.22-7-pve`

3. pin the kernel (i think boot by default)
   `pve-efiboot-tool kernel pin 5.11.22-7-pve`
4.reboot
5. now i am able to use GPU in openvino with inference time lowered to 35ms!
   detectors:
   openvino:
   type: openvino
   device: GPU
Were you able to identify the cause of this? Has it been upstream at all? Given how popular the J5005 is for Frigate, should we be raising an issue or flagging in the docs to avoid PVE8?

No, unfortunately I am not that techsavvy to further isolate/identify the problem. Once GPU detector started working after kernel downgrade I stopped researching this.

nickp27 commented 6 months ago

@nickp27 I finally was able to get GPU device working in frigate, As you said it was indeed a kernel bug causing the issue. I downgraded my kernel, kept proxmox 8 and finally openvino is working with GPU. If you would like to give your J5005 a try again, here's the steps: kernel downgrade instructions:
1. add pve 7(bullseye) in sources.list
   root@pve:/etc/apt/sources.list.d# cat proxmoxve7.xno-subscription.list
   deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

2. install the old kernel.
   `apt install pve-kernel-5.11.22-7-pve`

3. pin the kernel (i think boot by default)
   `pve-efiboot-tool kernel pin 5.11.22-7-pve`
4.reboot
5. now i am able to use GPU in openvino with inference time lowered to 35ms!
   detectors:
   openvino:
   type: openvino
   device: GPU
Were you able to identify the cause of this? Has it been upstream at all? Given how popular the J5005 is for Frigate, should we be raising an issue or flagging in the docs to avoid PVE8?
No, unfortunately I am not that techsavvy to further isolate/identify the problem. Once GPU detector started working after kernel downgrade I stopped researching this.

It looks like there are a couple of issues dealing with it, but in the meantime, I have gone with kernel 5.15, with some grub tweaks.

https://github.com/intel/compute-runtime/issues/679

https://github.com/intel/compute-runtime/issues/697

cheese202 commented 6 months ago

Hi. Can you please help me with yolo models? I've tried to train my own accornig @aeozyalcin notebook and convert it. I even tried to just download default yolov8* model according this notebook Convert YOLOv8 to OpenVino for Frigate.ipynb

Every time i'm getting an error from frigate on startup:

2024-02-28 10:21:18.665358694  [2024-02-28 13:21:18] frigate.detectors.plugins.openvino INFO    : Model Input Shape: [1,3,320,320]
2024-02-28 10:21:18.665403574  [2024-02-28 13:21:18] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: [1,84,2100]
2024-02-28 10:21:18.665442681  [2024-02-28 13:21:18] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2024-02-28 10:21:19.988495509  Process detector:ov:
2024-02-28 10:21:19.988500747  Traceback (most recent call last):
2024-02-28 10:21:19.988502267    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-02-28 10:21:19.988503394      self.run()
2024-02-28 10:21:19.988504666    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-02-28 10:21:19.988507373      self._target(*self._args, **self._kwargs)
2024-02-28 10:21:19.988512096    File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
2024-02-28 10:21:19.988513470      detections = object_detector.detect_raw(input_frame)
2024-02-28 10:21:19.988547388    File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
2024-02-28 10:21:19.988548949      return self.detect_api.detect_raw(tensor_input=tensor_input)
2024-02-28 10:21:19.988550325    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 92, in detect_raw
2024-02-28 10:21:19.988589437      if object_detected[0] != -1:
2024-02-28 10:21:19.988591040  IndexError: invalid index to scalar variable.

my config is:

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /media/frigate/openvino/custom/yolov8x.xml

model:
  width: 320
  height: 320
  input_tensor: nchw
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

I really not clue what i'm doing wrong but it seems that i can't launch any YOLO model instead of frigate's OpenVINO default model.

Can you please point where is the problem? Frigate version: 0.13.2

cheese202 commented 6 months ago

model_type: yolox

sorry... Forgot to set model type.

cheese202 commented 6 months ago

Now iI have another problem with detection Detected object labeling worng. I.e. now car is a person, person is a car and dog is someone else. And sometimes random.

Any ideas about this issue?

halk1982 commented 5 months ago

Now iI have another problem with detection Detected object labeling worng. I.e. now car is a person, person is a car and dog is someone else. And sometimes random.

Any ideas about this issue?

Your labelmap_path seems to be the wrong one. For YOLO you need to use coco_80cl.txt. Hope that helps with object labeling.

blakeblackshear / frigate