search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.7k stars 1.7k forks source link

[Config Support]: Memory Spikes While Running Proxmox LXC #7256

Closed peveleigh closed 1 year ago

peveleigh commented 1 year ago

Describe the problem you are having

I'm trying to run Frigate in an LXC but I keep running into memory spike issues regardless of how much memory I assign the container.

Screenshot 2023-07-23 110420

Version

0.12.1-367d724

Frigate config file

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

mqtt:
  host: 192.168.X.XX
  port: 1883
  topic_prefix: frigate_test
  client_id: frigate_test
  user: hass
  password: XXXX

database:
  path: /config/frigate.db

record:
  enabled: False

snapshots:
  enabled: True

ffmpeg:
  hwaccel_args: preset-vaapi

cameras:
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.X.XX:8554/camera.driveway_ss
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://192.168.X.XX:8554/camera.driveway
          input_args: preset-rtsp-restream
          roles:
            - record

  front_walkway:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.X.XX:8554/camera.front_walkway_ss
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://192.168.X.XX:8554/camera.front_walkway
          input_args: preset-rtsp-restream
          roles:
            - record

  garage_side:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.X.XX:8554/camera.garage_side
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record

  garage_inside:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.X.XX:8554/camera.garage_inside
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record

  living_room:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.X.XX:8554/camera.living_room
          roles:
            - detect
            - record

Relevant log output

frigate    | 2023-07-23 09:43:13.322949721  [2023-07-23 07:13:13] watchdog.garage_inside         INFO    : garage_inside exceeded fps limit. Exiting ffmpeg...
frigate    | 2023-07-23 09:43:13.323103440  [2023-07-23 07:13:13] watchdog.garage_inside         INFO    : Waiting for ffmpeg to exit gracefully...
frigate    | 2023-07-23 09:45:07.232321416  [2023-07-23 07:13:45] watchdog.garage_inside         INFO    : FFmpeg did not exit. Force killing...
frigate    | 2023-07-23 09:45:07.232980329  [2023-07-23 07:15:07] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate    | 2023-07-23 09:45:07.232982505  [2023-07-23 07:15:07] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate    | 2023-07-23 09:45:07.238372675  [2023-07-23 07:15:07] watchdog.front_walkway         INFO    : No frames received from front_walkway in 20 seconds. Exiting ffmpeg...
frigate    | 2023-07-23 09:45:07.238524799  [2023-07-23 07:15:07] watchdog.front_walkway         INFO    : Waiting for ffmpeg to exit gracefully...
frigate    | 2023-07-23 09:45:07.242800263  [2023-07-23 07:15:07] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
frigate    | 2023-07-23 09:45:07.245710902  [2023-07-23 07:15:07] root                           INFO    : Waiting for detection process to exit gracefully...
frigate    | 2023-07-23 09:45:07.247195289  [2023-07-23 07:15:07] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 09:45:07.247199557  [2023-07-23 07:15:07] frigate.video                  ERROR   : garage_inside: ffmpeg process is not running. exiting capture thread...
frigate    | 2023-07-23 09:45:07.247200793  [2023-07-23 07:15:07] watchdog.garage_side           INFO    : No frames received from garage_side in 20 seconds. Exiting ffmpeg...
frigate    | 2023-07-23 09:45:07.252135964  [2023-07-23 07:15:07] watchdog.garage_side           INFO    : Waiting for ffmpeg to exit gracefully...
frigate    | 2023-07-23 09:45:07.256225558  [2023-07-23 07:15:07] frigate.video                  ERROR   : front_walkway: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 09:45:07.256228952  [2023-07-23 07:15:07] frigate.video                  ERROR   : front_walkway: ffmpeg process is not running. exiting capture thread...
frigate    | 2023-07-23 09:45:07.266030254  [2023-07-23 07:15:07] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 09:45:07.266251857  [2023-07-23 07:15:07] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate    | 2023-07-23 09:45:07.291061041  [2023-07-23 07:15:07] frigate.video                  ERROR   : garage_side: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 09:45:07.292051962  [2023-07-23 07:15:07] frigate.video                  ERROR   : garage_side: ffmpeg process is not running. exiting capture thread...
frigate    | 2023-07-23 09:45:07.300890630  [2023-07-23 07:15:07] detector.ov                    INFO    : Signal to exit detection process...
frigate    | 2023-07-23 09:45:07.303425222  [2023-07-23 07:15:07] detector.ov                    INFO    : Exited detection process...
frigate    | 2023-07-23 09:45:07.334665027  [2023-07-23 07:15:07] root                           INFO    : Detection process has exited...
frigate    | 2023-07-23 09:45:07.370097430  [2023-07-23 07:15:07] detector.ov                    INFO    : Starting detection process: 7199
frigate    | 2023-07-23 09:45:07.730045712  E: [xLinkUsb] [    507730] [frigate.detecto] usb_find_device_with_bcd:266 Library has not been initialized when loaded
frigate    | 2023-07-23 09:45:07.735927331  E: [xLinkUsb] [    507735] [frigate.detecto] usb_find_device_with_bcd:266 Library has not been initialized when loaded
frigate    | 2023-07-23 09:45:08.004954771  [2023-07-23 07:15:07] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 300, 300, 3}
frigate    | 2023-07-23 09:45:08.005094480  [2023-07-23 07:15:07] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 1, 100, 7}
frigate    | 2023-07-23 09:45:08.005200438  [2023-07-23 07:15:08] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
frigate    | 2023-07-23 09:45:09.239233600  [2023-07-23 07:15:09] frigate.video                  ERROR   : living_room: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 09:45:09.239742600  [2023-07-23 07:15:09] frigate.video                  ERROR   : living_room: ffmpeg process is not running. exiting capture thread...
frigate    | 2023-07-23 09:45:17.003405617  [2023-07-23 07:15:16] watchdog.living_room           ERROR   : Ffmpeg process crashed unexpectedly for living_room.
frigate    | 2023-07-23 09:45:17.003488257  [2023-07-23 07:15:16] watchdog.living_room           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate    | 2023-07-23 09:45:17.223158387  [2023-07-23 07:15:17] watchdog.garage_inside         ERROR   : Ffmpeg process crashed unexpectedly for garage_inside.
frigate    | 2023-07-23 09:45:17.223266688  [2023-07-23 07:15:17] watchdog.garage_inside         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate    | 2023-07-23 09:45:17.223927680  [2023-07-23 07:15:17] ffmpeg.garage_inside.detect    ERROR   : [rtsp @ 0x564509159e40] RTP: PT=60: bad cseq d644 expected=0d95
frigate    | 2023-07-23 09:45:17.224016476  [2023-07-23 07:15:17] ffmpeg.garage_inside.detect    ERROR   : [h264 @ 0x56450927fdc0] Increasing reorder buffer to 1
frigate    | 2023-07-23 09:45:17.224088016  [2023-07-23 07:15:17] ffmpeg.garage_inside.detect    ERROR   : [out_0_0 @ 0x56450922d000] 100 buffers queued in out_0_0, something may be wrong.
frigate    | 2023-07-23 09:45:17.224142972  [2023-07-23 07:15:17] ffmpeg.garage_inside.detect    ERROR   : [out_0_0 @ 0x56450922d000] 1000 buffers queued in out_0_0, something may be wrong.
frigate    | 2023-07-23 09:45:17.224206851  [2023-07-23 07:15:17] ffmpeg.garage_inside.detect    ERROR   : [out_0_0 @ 0x56450922d000] 10000 buffers queued in out_0_0, something may be wrong.
frigate    | 2023-07-23 09:45:17.259928852  [2023-07-23 07:15:17] watchdog.front_walkway         ERROR   : Ffmpeg process crashed unexpectedly for front_walkway.
frigate    | 2023-07-23 09:45:17.261909565  [2023-07-23 07:15:17] watchdog.front_walkway         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate    | 2023-07-23 09:45:17.270477943  [2023-07-23 07:15:17] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate    | 2023-07-23 09:45:17.270618640  [2023-07-23 07:15:17] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate    | 2023-07-23 09:45:17.300750825  [2023-07-23 07:15:17] watchdog.garage_side           ERROR   : Ffmpeg process crashed unexpectedly for garage_side.
frigate    | 2023-07-23 09:45:17.302168614  [2023-07-23 07:15:17] watchdog.garage_side           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate    | 2023-07-23 09:45:17.303789961  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.303916313  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.303994358  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.304064024  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.304115630  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.304164033  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.304219230  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.304267473  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.304318706  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.304365244  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.304414050  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.304459064  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.304505116  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.304555691  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.304597537  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.304643634  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.304692232  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.311223169  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.311300426  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312643669  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312645864  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312647023  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312648211  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312649263  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312650316  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312823929  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312842954  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312844195  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312845343  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312855381  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312856562  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312857787  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312858949  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312867745  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312868839  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312869902  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312870950  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312871923  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312883902  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312884921  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312885952  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312899208  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312900310  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312901271  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.312902304  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.312903268  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.312904277  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.312929635  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.313015768  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.313091746  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.313145678  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.313204706  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.313264890  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.313320956  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.313381849  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.313434647  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.313492475  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.313544771  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.313601812  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.313654117  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.313709142  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.313763806  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.313820761  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.313873571  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.313929218  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.313979849  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.314034801  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.314084829  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.314141858  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.314191331  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.314246341  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.314295897  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.314350612  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.314402076  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.314458396  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.314509529  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.314568054  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.314624068  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.314682297  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.314741470  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!
frigate    | 2023-07-23 09:45:17.314795674  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] non-existing PPS 0 referenced
frigate    | 2023-07-23 09:45:17.314955661  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   :     Last message repeated 1 times
frigate    | 2023-07-23 09:45:17.315009264  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] decode_slice_header error
frigate    | 2023-07-23 09:45:17.315067023  [2023-07-23 07:15:17] ffmpeg.garage_side.detect      ERROR   : [h264 @ 0x56221aca0b80] no frame!

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"1.6"},"1":{"cpu":"0.0","mem":"0.0"},"108":{"cpu":"0.0","mem":"0.0"},"117":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.0"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"137":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"151":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"166":{"cpu":"0.0","mem":"0.0"},"204":{"cpu":"0.0","mem":"0.0"},"209":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"413":{"cpu":"0.0","mem":"0.0"},"419":{"cpu":"0.3","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"422":{"cpu":"0.7","mem":"0.0"},"425":{"cpu":"0.0","mem":"0.0"},"426":{"cpu":"0.3","mem":"0.1"},"427":{"cpu":"0.3","mem":"0.1"},"428":{"cpu":"0.3","mem":"0.1"},"429":{"cpu":"0.3","mem":"0.1"},"430":{"cpu":"0.0","mem":"0.0"},"431":{"cpu":"0.3","mem":"0.1"},"432":{"cpu":"0.7","mem":"0.1"},"433":{"cpu":"0.0","mem":"0.0"},"434":{"cpu":"0.7","mem":"0.1"},"436":{"cpu":"1.0","mem":"0.1"},"438":{"cpu":"0.0","mem":"0.0"},"441":{"cpu":"0.7","mem":"0.1"},"443":{"cpu":"1.0","mem":"0.1"},"445":{"cpu":"0.0","mem":"0.0"},"458":{"cpu":"0.0","mem":"0.0"},"7199":{"cpu":"0.0","mem":"0.8"},"7212":{"cpu":"1.3","mem":"0.1"},"7214":{"cpu":"1.7","mem":"0.1"},"7216":{"cpu":"1.3","mem":"0.1"},"7218":{"cpu":"1.3","mem":"0.1"},"7220":{"cpu":"1.3","mem":"0.1"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"87":{"cpu":"0.0","mem":"0.0"},"9351":{"cpu":"0.0","mem":"0.0"},"9376":{"cpu":"0.0","mem":"0.0"},"96":{"cpu":"0.7","mem":"0.3"},"MiB":{"cpu":"14944.6","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"average:","mem":"0.54,"}},"detection_fps":4.6,"detectors":{"ov":{"detection_start":0.0,"inference_speed":18.58,"pid":7199}},"driveway":{"camera_fps":5.1,"capture_pid":432,"detection_enabled":1,"detection_fps":1.7,"ffmpeg_pid":7218,"pid":426,"process_fps":5.1,"skipped_fps":0.0},"front_walkway":{"camera_fps":5.1,"capture_pid":434,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":7216,"pid":427,"process_fps":5.1,"skipped_fps":0.0},"garage_inside":{"camera_fps":5.1,"capture_pid":441,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":7214,"pid":429,"process_fps":5.1,"skipped_fps":0.0},"garage_side":{"camera_fps":5.1,"capture_pid":436,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":7220,"pid":428,"process_fps":5.1,"skipped_fps":0.0},"gpu_usages":{"intel-vaapi":{"gpu":"3.56 %","mem":"- %"}},"living_room":{"camera_fps":5.1,"capture_pid":443,"detection_enabled":1,"detection_fps":2.9,"ffmpeg_pid":7212,"pid":431,"process_fps":5.1,"skipped_fps":0.0},"service":{"last_updated":1690119596,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":250.8,"mount_type":"tmpfs","total":268.4,"used":17.6},"/media/frigate/clips":{"free":2730780.0,"mount_type":"ext4","total":2952325.1,"used":71498.7},"/media/frigate/recordings":{"free":2730780.0,"mount_type":"ext4","total":2952325.1,"used":71498.7},"/tmp/cache":{"free":1000.0,"mount_type":"tmpfs","total":1000.0,"used":0.0}},"temperatures":{},"uptime":57054,"version":"0.12.1-367d724"}}

Operating system

Proxmox

Install method

Docker Compose

Coral version

CPU (no coral)

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

I notice you are not setting the detect resolutions of any of the cameras which means they are all being resized to 1280x720, probably a good idea set these to avoid doing that work.

When it comes to memory issues like this, more info is going to be needed. Specifically, we need to know what process(es) or files are using the memory. High memory usage can be caused by a number of different issues.

The logs indicate that garage_side is sending bad data, what type of camera is this?

peveleigh commented 1 year ago

garage_side is a Digoo DG-W01f garage_inside and living_room are xiafang's flashed with dafang hacks driveway and front_walkway are empiretech/dahua IPC-T2431T-AS's

It may be worth noting that I'm running an identical config (except with record enabled) on another machine running bare metal debian.

That one did this without suffering a memory spike:

frigate    | 2023-07-23 07:13:16.348445645  [2023-07-23 07:13:16] watchdog.garage_inside         INFO    : garage_inside exceeded fps limit. Exiting ffmpeg...
frigate    | 2023-07-23 07:13:16.349120691  [2023-07-23 07:13:16] watchdog.garage_inside         INFO    : Waiting for ffmpeg to exit gracefully...
frigate    | 2023-07-23 07:13:22.202258464  192.168.2.10 - - [23/Jul/2023:07:13:22 -0230] "GET /api/stats HTTP/1.1" 200 3378 "-" "HomeAssistant/2023.6.2 aiohttp/3.8.4 Python/3.11" "-"
frigate    | 2023-07-23 07:13:31.230144753  192.168.2.10 - - [23/Jul/2023:07:13:31 -0230] "GET /api/stats HTTP/1.1" 200 3342 "-" "HomeAssistant/2023.6.2 aiohttp/3.8.4 Python/3.11" "-"
frigate    | 2023-07-23 07:13:40.210442889  192.168.2.10 - - [23/Jul/2023:07:13:40 -0230] "GET /api/stats HTTP/1.1" 200 3343 "-" "HomeAssistant/2023.6.2 aiohttp/3.8.4 Python/3.11" "-"
frigate    | 2023-07-23 07:13:46.383362714  [2023-07-23 07:13:46] watchdog.garage_inside         INFO    : FFmpeg did not exit. Force killing...
frigate    | 2023-07-23 07:13:46.887474731  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.888218849  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.888347821  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.888630578  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.888879954  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.890881259  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.891179450  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.891182169  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.891183755  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: Unable to read frames from ffmpeg process.
frigate    | 2023-07-23 07:13:46.891185350  [2023-07-23 07:13:46] frigate.video                  ERROR   : garage_inside: ffmpeg process is not running. exiting capture thread...
frigate    | 2023-07-23 07:13:49.233436055  192.168.2.10 - - [23/Jul/2023:07:13:49 -0230] "GET /api/stats HTTP/1.1" 200 3342 "-" "HomeAssistant/2023.6.2 aiohttp/3.8.4 Python/3.11" "-"
frigate    | 2023-07-23 07:13:50.332904068  [2023-07-23 07:13:50] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/garage_inside-20230723071251.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x562f202506c0] moov atom not found\n/tmp/cache/garage_inside-20230723071251.mp4: Invalid data found when processing input\n'
frigate    | 2023-07-23 07:13:50.333088954  [2023-07-23 07:13:50] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/garage_inside-20230723071251.mp4
frigate    | 2023-07-23 07:13:56.893369085  [2023-07-23 07:13:56] watchdog.garage_inside         ERROR   : Ffmpeg process crashed unexpectedly for garage_inside.
frigate    | 2023-07-23 07:13:56.894087959  [2023-07-23 07:13:56] watchdog.garage_inside         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate    | 2023-07-23 07:13:56.894096245  [2023-07-23 07:13:56] ffmpeg.garage_inside.detect    ERROR   : [rtsp @ 0x5597bd31e100] RTP: PT=60: bad cseq d644 expected=0d95
frigate    | 2023-07-23 07:13:56.894101248  [2023-07-23 07:13:56] ffmpeg.garage_inside.detect    ERROR   : [h264 @ 0x5597bd38b100] Increasing reorder buffer to 1
frigate    | 2023-07-23 07:13:56.900519003  [2023-07-23 07:13:56] ffmpeg.garage_inside.detect    ERROR   : [out_1_0 @ 0x5597bd7d7600] 100 buffers queued in out_1_0, something may be wrong.
frigate    | 2023-07-23 07:13:56.900865772  [2023-07-23 07:13:56] ffmpeg.garage_inside.detect    ERROR   : [out_1_0 @ 0x5597bd7d7600] 1000 buffers queued in out_1_0, something may be wrong.
NickM-27 commented 1 year ago

Right, definitely seems that the cameras are sending bad data. Although it looks like you are using a proxy for the cameras, what type of proxy is it?

peveleigh commented 1 year ago

go2rtc in a separate docker container on the bare metal debian machine.

NickM-27 commented 1 year ago

What version of go2rtc is running?

peveleigh commented 1 year ago

1.5.0

NickM-27 commented 1 year ago

Maybe try 1.6.2 and see if it is any better. Either way though, the memory usage is likely due to a difference in the kernel or drivers on the host, updating those may help (and has helped many other users that were experiencing similar)

peveleigh commented 1 year ago

Ok, thanks. I'll update the detect resolution, update the proxmox host, update go2rtc and go from there.

peveleigh commented 1 year ago

I seem to have resolved (or at least masked) this issue by setting a memory limit for the frigate docker container.

docker-compose.yml

services:
  frigate:
    deploy:
      resources:
        limits:
          memory: 6G

Seems to be a memory management issue whenever a stream sends malformed data. Maybe the proxmox host and lxc can't decide who's supposed to be the oom-killer? Regardless, passing the responsibility on to Docker seems to do the trick.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.