search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.21k stars 1.76k forks source link

[Support]: System crashes #5799

Closed stevenbrown1862 closed 1 year ago

stevenbrown1862 commented 1 year ago

Describe the problem you are having

Hello. I don't know if anyone has any thoughts or have seen similar but I'm running in docker on an i3 6100t and the system crashes every day or so...I initially assumed it was a hardware issue but then I started noticing the following popping up repeatedly in the system log.

Mar 19 13:52:13 steve-HP-EliteDesk-800-G2-DM-35W kernel: [164137.340296] i915 0000:00:02.0: [drm] Resetting rcs0 for preemption time out Mar 19 13:52:13 steve-HP-EliteDesk-800-G2-DM-35W kernel: [164137.342911] i915 0000:00:02.0: [drm] GPU HANG: ecode 9:1:8ed97ff2, in frigate.detecto [475124].

I've had a look in the portainer logs to see if I can see anything relevant and not seen anything obvious. Tonights crash had loads of lines that looked like this just before the crash but not sure whether its relevant:

2023-03-21 19:19:01.876039553 192.168.1.191 - - [21/Mar/2023:19:19:01 +0000] "GET /api/stats HTTP/1.1" 200 2844 "-" "HomeAssistant/2023.3.5 aiohttp/3.8.4 Python/3.10" "-"

Most of the crashes seem to happen when an event is being logged (although there's no indication of an event on tonight's crash).

Version

Beta 10

Frigate config file

go2rtc:

  streams:
    new_doorbell: 
      - echo:bash /config/get_stream.sh camera.front_door
    livingroom: 
      - echo:bash /config/get_stream.sh camera.living_room
   # front_drive: 
    #  - rtsp://admin:NWSXNR@192.168.1.83:554/H.264

  #  back_garden:
    #  - ffmpeg:rtsp://192.168.1.121:554/live/ch0#video=h264#hardware=vaapi
    #  - rtsp://192.168.1.121:554/live/ch0
  #  doorbell:
  #    - rtsp://192.168.1.191:8554/new_doorbell
  #    - "ffmpeg:rtsp_cam#audio=opus"
  #  living_room:
   #   - ffmpeg:rtsp://192.168.1.191:8554/livingroom#video=copy
database:
  path: /db/frigate.db      

timestamp_style:
  # Optional: Position of the timestamp (default: shown below)
  #           "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
  position: "tl"
mqtt:
  host: 192.168.1.191
  user: mqttbroker
  password: mqttbroker
birdseye:
  enabled: True
  mode: motion
  restream: true
detectors:
  ov:
    type: openvino
    device: CPU
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml
model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt
record:
  enabled: True
  retain:
    days: 5
    mode: motion
  events:
    retain:
      default: 21
      mode: active_objects
    pre_capture: 10

    post_capture: 35
#ffmpeg:
#  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
#  global_args: -hide_banner -loglevel warning
#  output_args:
#    detect: -f rawvideo -pix_fmt yuv420p

cameras:
  back_garden:
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264 # <------ Name the camera
    ffmpeg:

      inputs:
        - path: rtsp://192.168.1.121:554/live/ch0

          input_args: preset-rtsp-generic

          roles:
            - detect
            - record

    snapshots: # <----- Enable recording
        enabled: True

    objects:
      track: 
        - person
        - cat
        - dog
        - bird
      filters:
        bird:
          min_area: 7000
        person:
          min_area: 10000
          max_area: 90000

    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
   #   fps: 5
    motion:
    #  contour_area: 50
      threshold: 30

     # delta_alpha: 0.1
      mask:  

        - 1280,51,1280,0,0,0,0,151,283,48

        - 699,436,216,720,37,546,421,362        
        - 791,399,897,556,1089,511,943,218,828,143,721,163,610,232,550,328

  front_drive: # <------ Name the camera
    ffmpeg:

      hwaccel_args: preset-intel-qsv-h264
   #   hwaccel_args: preset-vaapi
      output_args: 
          record: preset-record-generic-audio-copy

      inputs:
     #   - path: rtsp://192.168.1.191:8554/front_drive
      #    input_args: preset-rtsp-restream-low-latency
        - path: rtsp://admin:NWSXNR@192.168.1.83:554/H.264
          input_args: preset-rtsp-generic 

          roles:
            - detect

            - record
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500   

    snapshots: # <----- Enable recording
        enabled: True
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 960 #1920 #960 # <---- update for your camera's resolution
      height: 540 #1080 #540 # <---- update for your camera's resolution

    objects:
      track: 
        - car
        - person
        - bicycle
        - cat
        - dog
        - bird
      filters:
        car:
          min_area: 8000
          mask: 

            - 0,540,0,0,960,246,960,540

            - 805,128,960,192,960,0,275,0,371,48

           # - 0,1080,1920,1080,1920,302,1364,261,921,279,226,0,0,0

            #- 0,720,1280,720,1280,244,560,295,226,0,0,0
            #- 1722,239,1920,757,1920,0,1462,0,1450,188

            #- 1280,207,1280,0,930,0,884,75
        person:
          min_area: 2700
          max_area: 64000
          threshold: 0.75
        #  min_ratio: 0.3
      # Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
          max_ratio: 0.8
          mask:
             - 473,67,791,107,960,187,960,0,345,0,0,0,0,126
          #  - 1456,169,1920,351,1920,0,1466,27
       #     - 1280,207,1280,0,807,0,803,82

    motion:

   #   improve_contrast: True
      delta_alpha: 0.3
      threshold: 25

      mask:
       # - 0,334,456,133,1249,125,1920,304,1920,0,0,0
       # - 677,1080,77,1080,89,998,665,995
        - 0,191,446,34,845,132,960,184,960,0,0,0

        - 331,540,335,496,24,497,26,540

  doorbell: # <------ Name the camera
    ffmpeg:
  #    hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p

      hwaccel_args: preset-intel-qsv-h264
   #   hwaccel_args: preset-vaapi
    #  output_args:
     #   record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
      output_args: 
          record: preset-record-generic-audio-copy

      inputs:

        - path: rtsp://192.168.1.191:8554/new_doorbell
          input_args: preset-rtsp-generic

      #  - path: rtsp://192.168.1.191:8554/new_doorbell

          roles:
            - detect

            - record
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500 

    snapshots: # <----- Enable recording
        enabled: True
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1200 # <---- update for your camera's resolution
      height: 900 # <---- update for your camera's resolution

    objects:
      filters:
        person:
          min_area: 5000
      track: 
        - person
        - cat
        - dog
        - bird
    motion:
      improve_contrast: True
      contour_area: 20
      mask:
        - 331,362,770,366,907,176,1200,0,0,0,0,304

    #    - 1373,0,1049,612,363,616,338,666,0,669,0,0
     #   - 539,727,1041,783,1050,719,778,700,571,687
      #  - 0,815,129,798,107,720,0,736
       # - 1415,748,1264,1200,1600,1200,1600,0,1361,0

  #    mask:
   #     - 800,0,800,68,620,212,171,185,0,290,0,0
    #    - 539,727,1041,783,1050,719,778,700,571,687
     #   - 0,815,129,798,107,720,0,736
      #  - 1415,748,1264,1200,1600,1200,1600,0,1361,0

  living_room: # <------ Name the camera
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264

   #   hwaccel_args: preset-vaapi             
      inputs:
      # - path: rtsp://192.168.1.191:8554/living_room
       #  input_args: preset-rtsp-restream
       - path: rtsp://192.168.1.191:8554/livingroom
         input_args: preset-rtsp-generic

         roles:
            - detect

            - record

    snapshots: # <----- Enable recording
        enabled: True
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1152 # <---- update for your camera's resolution
      height: 864 # <---- update for your camera's resolution 
    motion:
    #  threshold: 37
      mask:
        - 1115,372,978,864,1152,864,1152,0,0,0,0,388
      #  - 514,150,515,432,576,432,576,0,0,0,0,155
    birdseye:
      enabled: False   
    objects:
      filters:
        person:
          mask:

            - 0,864,293,864,443,0,0,0
         #   - 0,432,170,432,146,0,0,0

Relevant log output

See log entries in question.

FFprobe output from your camera

N/A

Frigate stats

I'm not sure how to get this.

Operating system

Debian

Install method

Docker Compose

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

Various generic chines, Ezviz and 2 nest cameras

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

More info will be needed, if the GPU is hanging then there is some issue on the host most likely. If you are using the integration then /api/stats gets called every 60 seconds so that does not really mean anything

stevenbrown1862 commented 1 year ago

Thanks for the quick response Nick. What more info will be helpful ? When you say issue on the host then are we talking about hardware (bad RAM etc.) or with the operating system ?

More info will be needed, if the GPU is hanging then there is some issue on the host most likely. If you are using the integration then /api/stats gets called every 60 seconds so that does not really mean anything

javydekoning commented 1 year ago

Looks like https://gitlab.freedesktop.org/drm/intel/-/issues/673.

What is your Linux kernel version? uname -r?

Have you installed the latest firmware-linux package (this includes non-free i915 driver)

stevenbrown1862 commented 1 year ago

Looks like https://gitlab.freedesktop.org/drm/intel/-/issues/673.

What is your Linux kernel version? uname -r?

Have you installed the latest firmware-linux package (this includes non-free i915 driver)

I had a similar thought about the kernel after posting the issue so have updated to the latest kernel that Linux mint offered me which is 5.19.0-35-generic. Going from 5.15.0-67.

How would I install the "latest firmware-linux package (this includes non-free i915 driver)" ?

javydekoning commented 1 year ago

Not a Mint user myself, but I guess sudo apt install linux-firmware linux-firmware-nonfree

stevenbrown1862 commented 1 year ago

I've now gone 6 days without a crash since updating the kernel version so it looks like that was the problem. Due to when it started happening and the fact it was always linked to detection/motion events I presume its something to do with the OpenVINO detection and the GPU driver in that version of the kernel but thats just an uneducated guess. Thanks for the suggestion @javydekoning .

NickM-27 commented 1 year ago

Thanks for coming back and updating. I'll go ahead and close this then, feel free to create a new issue if something else comes up

Shponzo commented 1 year ago

@stevenbrown1862 To which kernel version did you upgrade? I'm using Ubuntu 22.04 and had kernel 5.15.0-67-generic and was experiencing the same problem. I tried upgrading to v5.19.17-051917 and now die PCIe Coral is not found anymore...

stevenbrown1862 commented 1 year ago

@stevenbrown1862 To which kernel version did you upgrade? I'm using Ubuntu 22.04 and had kernel 5.15.0-67-generic and was experiencing the same problem. I tried upgrading to v5.19.17-051917 and now die PCIe Coral is not found anymore...

I'm using 5.19.0-35-generic and so far so good...7 days without a crash. I had assumed my issue was related to me using the GPU for the OpenVino detector which presumably you aren't using given that you have a Coral.

Shponzo commented 1 year ago

@stevenbrown1862

I just changed to kernel v5.15.104 and this one does work with my Coral. Maybe a stupid question, but can I just upgrade to v6.2.8? Or how do I decide which version to choose to upgrade to?