search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.24k stars 1.76k forks source link

[Support]: ffmpeg causing out of memory crash on host #6625

Closed kirsch33 closed 1 year ago

kirsch33 commented 1 year ago

Describe the problem you are having

so i have recently added 2 wireless cameras to my config recently, a cheap Wyze Pan v1 with Dafang firmware and a new Amcrest link here. My other cameras are hardwired Amcrest that i've not ever previously had any issues with

since adding these cameras, ive noticed a steady stream of ffmpeg errors in the frigate logs. it seems that most of the time these errors are inconsequential and recoverable since the streams are available 90% of the time (occasionally have to reload a couple times) but otherwise i would only know about these errors from always keeping an eye on the log file.

however, about every 24-36 hours i have been noticing the entire host that Frigate is on will lock up entirely for about 5 minutes or so (which affects almost every other service on my network) and i can trace back the error to ffmpeg out of memory:

May 26 11:09:26 gws1-docker kernel: [88580.665143] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-timesyncd.service,mems_allowed=0-1,global_oom,task_memcg=/system.slice/docker-358b6d41b6b4bb837545709fd5cca9d8d27748f1169c03463ccdcf7c469dd016.scope,task=ffmpeg,pid=917557,uid=0

May 26 11:09:26 gws1-docker kernel: [88580.665187] Out of memory: Killed process 917557 (ffmpeg) total-vm:47404240kB, anon-rss:37944796kB, file-rss:81536kB, shmem-rss:10240kB, UID:0 pgtables:83032kB oom_score_adj:0

this is a huge issue because it interrupts every VM that this host runs. the VM that runs Frigate has been given 48GB memory so I dont think its an actual resource issue.

i have tried using a custom go2rtc version and the results are the same. i currently use 1.5.0

Version

0.13.0-ca7790f

Frigate config file

mqtt:
  host: 192.168.42.13
  port: 1883
  user: homeassistant
  password: pass

detectors:
  coral_pci:
    type: edgetpu
    device: pci

ffmpeg:
  hwaccel_args: preset-nvidia-h264

go2rtc:
  streams:
    back: 
      - rtsp://user:pass@192.168.72.83:554/cam/realmonitor?channel=1&subtype=0 #wired Amcrest 
    front: 
      - rtsp://user:pass@192.168.72.84:554/cam/realmonitor?channel=1&subtype=0 #wired Amcrest 
    iot-cam1:
      - rtsp://user:pass@192.168.62.100:554/unicast#video=h264#rotate=180 #wireless Wyze Pan v1 w/ Dafang
    iot-cam2: 
      - rtsp://user:pass@192.168.62.101:554/cam/realmonitor?channel=1&subtype=0 #wireless Amcrest indoor
  webrtc:
    listen: ":8555"
    candidates:
      - 192.168.42.12:8555
      - stun:8555

cameras:
  back:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://192.168.42.12:8554/back
          input_args: preset-rtsp-restream-low-latency
          roles:
            - record
            - detect
    zones:
      back_yard:
        coordinates: 0,1080,1117,1080,1349,925,1699,601,1656,484,1730,384,1381,218,1455,157,883,0,747,0,533,56,225,161,0,295
        objects:
          - person
    mqtt:
      required_zones:
        - back_yard
    record:
      retain:
        days: 30
        mode: all                 
      events:
        required_zones:
          - back_yard
    snapshots:
      required_zones:
        - back_yard     
    detect:
      width: 1920
      height: 1080               

  front:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://192.168.42.12:8554/front
          input_args: preset-rtsp-restream-low-latency
          roles:
            - record
            - detect
    objects:
      filters:
        car:
          mask:
            - 395,144,671,62,959,0,0,0,0,356,173,247
            - 1293,152,1648,243,1920,355,1920,0,1584,0,1358,47,1077,119
            - 0,1080,1920,1080,1920,612,1420,416,1136,322,906,265,737,241,511,341,258,477,0,647
    zones:
      front_yard:
        coordinates: 0,1080,1339,1080,1553,875,1792,444,1844,179,1426,90,1178,166,872,291,438,513,0,824
        objects:
          - person
      front_driveway:
        coordinates: 1148,150,1012,133,692,247,929,262,1063,294,1266,177
        objects:
          - car
    mqtt:
      required_zones:
        - front_yard
        - front_driveway
    record:
      retain:
        days: 30
        mode: all           
      events:
        required_zones:
         - front_yard
         - front_driveway
    snapshots:
      required_zones:
        - front_yard
        - front_driveway  
    detect:
      width: 1920
      height: 1080       

  iot-cam1:
    ffmpeg:
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp://192.168.42.12:8554/iot-cam1
          input_args: preset-rtsp-restream-low-latency
          roles:
            - record
    record:
      retain:
        days: 10
        mode: motion     
    detect:
      enabled: False
    onvif:
      host: 192.168.62.100
      port: 8080
      user: user
      password: pass        

  iot-cam2:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://192.168.42.12:8554/iot-cam2
          input_args: preset-rtsp-restream-low-latency
          roles:
            - record
    record:
      retain:
        days: 10
        mode: motion  
    detect:
      enabled: False             
    onvif:
      host: 192.168.62.101
      port: 80
      user: user
      password: pass

birdseye:
  enabled: False

record:
  enabled: True
  retain:
    days: 30
    mode: all
  events:
    pre_capture: 15
    post_capture: 15

snapshots:
  enabled: True
  bounding_box: True
  retain:
    default: 45

objects:
  track:
    - person
    - car
  filters:
    person:
      min_area: 4000
      max_area: 80000
      min_score: 0.55
      threshold: 0.70
    car:
      max_area: 120000
      min_score: 0.55
      threshold: 0.70
    dog:
      min_score: 0.55
      threshold: 0.70

detect:
  fps: 5
  max_disappeared: 25
  stationary:
    threshold: 50

motion:
  contour_area: 100

ui:
  live_mode: webrtc
  use_experimental: false

live:
  height: 1080

Relevant log output

2023-05-26 11:02:06.052797630  [2023-05-26 11:02:06] watchdog.iot-cam1              ERROR   : Ffmpeg process crashed unexpectedly for iot-cam1.
2023-05-26 11:02:06.053059028  [2023-05-26 11:02:06] watchdog.iot-cam1              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-05-26 11:02:06.053328263  [2023-05-26 11:02:06] ffmpeg.iot-cam1.detect         ERROR   : [rtsp @ 0x56348a149140] RTP: PT=60: bad cseq cfc5 expected=1231
2023-05-26 11:02:06.053597515  [2023-05-26 11:02:06] ffmpeg.iot-cam1.detect         ERROR   : [h264 @ 0x56348a15df00] Increasing reorder buffer to 1
2023-05-26 11:02:06.053857893  [2023-05-26 11:02:06] ffmpeg.iot-cam1.detect         ERROR   : [out_1_0 @ 0x56348adfd1c0] 100 buffers queued in out_1_0, something may be wrong.
2023-05-26 11:02:06.054118135  [2023-05-26 11:02:06] ffmpeg.iot-cam1.detect         ERROR   : [out_1_0 @ 0x56348adfd1c0] 1000 buffers queued in out_1_0, something may be wrong.
2023-05-26 11:06:16.077151969  [2023-05-26 11:06:16] watchdog.iot-cam1              INFO    : iot-cam1 exceeded fps limit. Exiting ffmpeg...
2023-05-26 11:06:16.077806478  [2023-05-26 11:06:16] watchdog.iot-cam1              INFO    : Waiting for ffmpeg to exit gracefully...
2023-05-26 11:09:20.874908261  [2023-05-26 11:09:20] watchdog.front                 INFO    : No frames received from front in 20 seconds. Exiting ffmpeg...
2023-05-26 11:09:20.875033053  [2023-05-26 11:09:20] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-05-26 11:09:21.619565273  [2023-05-26 11:09:21] frigate.video                  ERROR   : iot-cam1: Unable to read frames from ffmpeg process.
2023-05-26 11:09:21.619682760  [2023-05-26 11:09:21] frigate.video                  ERROR   : iot-cam1: ffmpeg process is not running. exiting capture thread...
2023-05-26 11:09:27.074278780  [2023-05-26 11:09:27] frigate.video                  ERROR   : front: Unable to read frames from ffmpeg process.
2023-05-26 11:09:27.074716903  [2023-05-26 11:09:27] frigate.video                  ERROR   : front: ffmpeg process is not running. exiting capture thread...
2023-05-26 11:09:30.463568035  [2023-05-26 11:09:30] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-05-26 11:09:30.567535992  [2023-05-26 11:09:30] root                           INFO    : Waiting for detection process to exit gracefully...
2023-05-26 11:09:30.639776647  [2023-05-26 11:09:30] detector.coral_pci             INFO    : Signal to exit detection process...
2023-05-26 11:09:31.226746062  [2023-05-26 11:09:31] detector.coral_pci             INFO    : Exited detection process...
2023-05-26 11:09:31.312123458  [2023-05-26 11:09:31] root                           INFO    : Detection process has exited...
2023-05-26 11:09:31.538437193  [2023-05-26 11:09:31] detector.coral_pci             INFO    : Starting detection process: 119674
2023-05-26 11:09:31.579324153  [2023-05-26 11:09:31] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2023-05-26 11:09:31.617750435  [2023-05-26 11:09:31] watchdog.iot-cam1              ERROR   : Ffmpeg process crashed unexpectedly for iot-cam1.
2023-05-26 11:09:31.618395117  [2023-05-26 11:09:31] watchdog.iot-cam1              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-05-26 11:09:31.618769962  [2023-05-26 11:09:31] ffmpeg.iot-cam1.detect         ERROR   : [h264 @ 0x5613fa8824c0] Increasing reorder buffer to 1
2023-05-26 11:09:31.619224273  [2023-05-26 11:09:31] ffmpeg.iot-cam1.detect         ERROR   : [rtsp @ 0x5613fa84f140] RTP: PT=60: bad cseq 6137 expected=2c7b
2023-05-26 11:09:31.620628741  [2023-05-26 11:09:31] ffmpeg.iot-cam1.detect         ERROR   : [segment @ 0x5613fa88a840] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-05-26 11:09:31.621048913  [2023-05-26 11:09:31] ffmpeg.iot-cam1.detect         ERROR   : [out_1_0 @ 0x5613fb50ae80] 100 buffers queued in out_1_0, something may be wrong.
2023-05-26 11:09:31.621470095  [2023-05-26 11:09:31] ffmpeg.iot-cam1.detect         ERROR   : [out_1_0 @ 0x5613fb50ae80] 1000 buffers queued in out_1_0, something may be wrong.
2023-05-26 11:09:31.621913653  [2023-05-26 11:09:31] ffmpeg.iot-cam1.detect         ERROR   : [out_1_0 @ 0x5613fb50ae80] 10000 buffers queued in out_1_0, something may be wrong.
2023-05-26 11:09:31.858159761  [2023-05-26 11:09:31] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-05-26 11:09:31.933043733  [2023-05-26 11:09:31] frigate.detectors.plugins.edgetpu_tfl INFO    : Model Input Shape: [  1 320 320   3]
2023-05-26 11:09:36.303640533  [2023-05-26 11:09:36] watchdog.iot-cam2              INFO    : iot-cam2 exceeded fps limit. Exiting ffmpeg...
2023-05-26 11:09:36.303934805  [2023-05-26 11:09:36] watchdog.iot-cam2              INFO    : Waiting for ffmpeg to exit gracefully...
2023-05-26 11:09:36.846720268  [2023-05-26 11:09:36] frigate.video                  ERROR   : iot-cam2: Unable to read frames from ffmpeg process.
2023-05-26 11:09:36.846813360  [2023-05-26 11:09:36] frigate.video                  ERROR   : iot-cam2: ffmpeg process is not running. exiting capture thread...
2023-05-26 11:09:37.081891219  [2023-05-26 11:09:37] watchdog.front                 ERROR   : Ffmpeg process crashed unexpectedly for front.
2023-05-26 11:09:37.092826801  [2023-05-26 11:09:37] watchdog.front                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-05-26 11:09:40.942174276  [2023-05-26 11:09:40] watchdog.back                  INFO    : back exceeded fps limit. Exiting ffmpeg...
2023-05-26 11:09:40.943447807  [2023-05-26 11:09:40] watchdog.back                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-05-26 11:09:41.687824388  [2023-05-26 11:09:41] frigate.video                  ERROR   : back: Unable to read frames from ffmpeg process.
2023-05-26 11:09:41.688101297  [2023-05-26 11:09:41] frigate.video                  ERROR   : back: ffmpeg process is not running. exiting capture thread...
2023-05-26 11:09:46.915808734  [2023-05-26 11:09:46] watchdog.iot-cam2              ERROR   : Ffmpeg process crashed unexpectedly for iot-cam2.
2023-05-26 11:09:46.916350844  [2023-05-26 11:09:46] watchdog.iot-cam2              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-05-26 11:09:46.916824653  [2023-05-26 11:09:46] ffmpeg.iot-cam2.detect         ERROR   : [rtsp @ 0x55982490c140] RTP: PT=60: bad cseq 9806 expected=7f68
2023-05-26 11:09:46.917281142  [2023-05-26 11:09:46] ffmpeg.iot-cam2.detect         ERROR   : [rtsp @ 0x55982490c140] CSeq 41 expected, 0 received.
2023-05-26 11:09:46.917649146  [2023-05-26 11:09:46] ffmpeg.iot-cam2.detect         ERROR   :     Last message repeated 4 times
2023-05-26 11:09:46.918009659  [2023-05-26 11:09:46] ffmpeg.iot-cam2.detect         ERROR   : [rtsp @ 0x55982490c140] RTP: PT=60: bad cseq def1 expected=5705
2023-05-26 11:09:51.699937277  [2023-05-26 11:09:51] watchdog.back                  ERROR   : Ffmpeg process crashed unexpectedly for back.
2023-05-26 11:09:51.707756662  [2023-05-26 11:09:51] watchdog.back                  ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-05-26 11:09:51.713522819  [2023-05-26 11:09:51] ffmpeg.back.detect             ERROR   : [rtsp @ 0x55c19816a140] RTP: PT=60: bad cseq dc8b expected=b161
2023-05-26 11:09:51.718964083  [2023-05-26 11:09:51] ffmpeg.back.detect             ERROR   : [rtsp @ 0x55c19816a140] RTP: PT=60: bad cseq 30c4 expected=f78f

go2rtc logs only show an i/o timeout around the same time as the logs above

Operating system

Proxmox

Install method

Docker Compose

Coral version

PCIe

NickM-27 commented 1 year ago

Wyze cams cause many problems due to their poor implementation of streaming (regardless of rtsp firmware / type). I'd highly suggest not using the wyze cam with go2rtc or at least not using the wyze cam restream as the source for frigate cameras section

kirsch33 commented 1 year ago

Wyze cams cause many problems due to their poor implementation of streaming (regardless of rtsp firmware / type). I'd highly suggest not using the wyze cam with go2rtc or at least not using the wyze cam restream as the source for frigate cameras section

but ultimately it is either go2rtc or frigate (not sure which) that is suffering from the runaway memory usage regarding the ffmpeg stream and causing the host to crash (no doubt caused from a faulty stream on a cheap camera), but is there anyway to identify when this issue is occurring and implement a failsafe?

NickM-27 commented 1 year ago

we have failsafes in place for many stream conditions but these conditions only do so much. At a certain point the failsafe is literally just to stop running the camera. Could be done but that logic doesn't exist currently.

peveleigh commented 1 year ago

Frigate kept crashing for me as well until I stopped using the built in go2rtc and switched back to running go2rtc in its own docker container. I'm not using Wyze cams but I am using Xiafang cameras (the camera Wyze is cloned from) flashed in dafang hacks.

NickM-27 commented 1 year ago

Most likely it's just that go2rtc was running a newer version. Many users have found updating the internal go2rtc to fix the issue

oguime commented 1 year ago

Are there directions on updating the internal go2rtc?

NickM-27 commented 1 year ago

Of course https://docs.frigate.video/configuration/advanced#custom-go2rtc-version

oguime commented 1 year ago

That was fast, thanks!

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.