search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.31k stars 1.6k forks source link

Running out of memory? #1413

Closed drinfernoo closed 2 years ago

drinfernoo commented 3 years ago

Describe the bug Almost every day I'm seeing a "cannot allocate memory" error, followed by Frigate stopping working (still running, but nothing happens) until I restart the container.

Version of frigate 0.8.4-5043040

Config file Include your full config file wrapped in triple back ticks.

detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu
  # coral:
  #   type: edgetpu
  #   device: usb
mqtt:
  host: 192.168.1.253
cameras:
  front_porch:
    snapshots:
      enabled: True
      bounding_box: True
      retain:
        default: 5
    clips:
      enabled: True
      objects:
        - person
        - cat
    record:
      enabled: False
    rtmp:
      enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://front:front@192.168.1.202/live
          roles:
            - detect
            - rtmp
            - clips
            # - record
    width: 1920
    height: 1080
    fps: 5
  back_porch:
    snapshots:
      enabled: True
      bounding_box: True
      retain:
        default: 5
    clips:
      enabled: True
      objects:
        - person
        - cat
    record:
      enabled: False
    rtmp:
      enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://back:back@192.168.1.203/live
          roles:
            - detect
            - rtmp
            - clips
            # - record
    width: 1920
    height: 1080
    fps: 5
objects:
  track:
    - person
    # - car
    # - truck
    - cat
    # - dog
    # - bicycle
  # filters:
  # person:
  # threshold: 0.75
# detect:
#   max_disappeared: 25
clips:
  retain:
    default: 5
ffmpeg:
  output_args:
    clips: "-f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac"
    rtmp: "-c copy -an -f flv"
  # record: "-f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac"
# logger:
#   default: info
#   logs:
#     frigate.mqtt: error

Frigate container logs

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/frigate/frigate/video.py", line 185, in run
    self.start_ffmpeg_detect()
  File "/opt/frigate/frigate/video.py", line 209, in start_ffmpeg_detect
    self.ffmpeg_detect_process = start_or_restart_ffmpeg(ffmpeg_cmd, self.logger, self.logpipe, self.frame_size)
  File "/opt/frigate/frigate/video.py", line 98, in start_or_restart_ffmpeg
    process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, stderr=logpipe, stdin = sp.DEVNULL, bufsize=frame_size*10, start_new_session=True)
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1637, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory

Frigate stats

{
  "back_porch": {
    "camera_fps": 5.0, 
    "capture_pid": 46, 
    "detection_fps": 0.0, 
    "pid": 44, 
    "process_fps": 5.0, 
    "skipped_fps": 0.0
  }, 
  "detection_fps": 0.0, 
  "detectors": {
    "cpu1": {
      "detection_start": 0.0, 
      "inference_speed": 94.97, 
      "pid": 38
    }, 
    "cpu2": {
      "detection_start": 0.0, 
      "inference_speed": 96.29, 
      "pid": 40
    }
  }, 
  "front_porch": {
    "camera_fps": 0.0, 
    "capture_pid": 45, 
    "detection_fps": 0.0, 
    "pid": 43, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 63.3, 
        "mount_type": "tmpfs", 
        "total": 67.1, 
        "used": 3.8
      }, 
      "/media/frigate/clips": {
        "free": 14252.8, 
        "mount_type": "ext4", 
        "total": 30509.4, 
        "used": 16202.2
      }, 
      "/media/frigate/recordings": {
        "free": 14252.8, 
        "mount_type": "ext4", 
        "total": 30509.4, 
        "used": 16202.2
      }, 
      "/tmp/cache": {
        "free": 14252.8, 
        "mount_type": "overlay", 
        "total": 30509.4, 
        "used": 16202.2
      }
    }, 
    "uptime": 145, 
    "version": "0.8.4-5043040"
  }
}

FFprobe from your camera

root@ERROL:~# ffprobe rtsp://front:front@192.168.1.202/live
pts item:[000000:00000001:00000001]
ffprobe version 0.8, Copyright (c) 2007-2011 the FFmpeg developers
  built on May 14 2020 05:54:30 with gcc 6.3.1 20170109
  configuration: --prefix=/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/us                                                                                                                                                             r --shlibdir=/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/usr/lib/libplay                                                                                                                                                             er --cross-prefix=/srv/output/amffmpeg/host/bin/aarch64-linux-gnu- --arch=aarch6                                                                                                                                                             4 --extra-ldflags='-L/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/usr/lib                                                                                                                                                             / -L/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/usr/lib/libplayer -lamav                                                                                                                                                             utils -ldl' --disable-yasm --enable-debug --disable-ffplay --disable-ffmpeg --en                                                                                                                                                             able-cross-compile --target-os=linux --disable-librtmp --disable-static --enable                                                                                                                                                             -shared --disable-ffserver --disable-doc --cpu=cortex-a53
  libavutil    51.  9. 1 / 51.  9. 1
  libavcodec   53.  7. 0 / 53.  7. 0
  libavformat  53.  4. 0 / 53.  4. 0
  libavdevice  53.  1. 1 / 53.  1. 1
  libavfilter   2. 23. 0 /  2. 23. 0
  libswscale    2.  0. 0 /  2.  0. 0
[init-input] iformat namertsp return
[rtsp_read_header:363]
[tcp @ 0x20e4e910] tcp will get address from dns!
[tcp @ 0x20e4e910] resolved 192.168.1.202's  ipaddress
Segmentation fault

Computer Hardware

Camera Info:

blakeblackshear commented 3 years ago

Can you post your docker run command or compose file?

drinfernoo commented 3 years ago

Sure, relevant bit from my docker-compose.yaml here:

  frigate: # For interfacing with Frigate
    container_name: frigate
    image: "blakeblackshear/frigate:0.8.4-aarch64"
    ports:
      - "5000:5000"
      - "1935:1935"
    privileged: true
    restart: unless-stopped
    volumes:
      - "/dev/bus/usb:/dev/bus/usb"
      - "/etc/localtime:/etc/localtime:ro"
      - "/etc/timezone:/etc/timezone:ro"
      - "/storage/docker/frigate/config:/config"
      - "/storage/docker/frigate/clips:/media/frigate/clips"
      - "/storage/docker/frigate/recordings:/media/frigate/recordings"
      - "/storage/docker/frigate/config/coco_labels.txt:/labelmap.txt"
blakeblackshear commented 3 years ago

Can you post the logs leading up to the out of memory error? Is the ffmpeg process constantly restarting?

drinfernoo commented 3 years ago

Yeah, something like this:

ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 805906, current: 805782; changing to 805907. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 805907, current: 805573; changing to 805908. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 805908, current: 805403; changing to 805909. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 805909, current: 805532; changing to 805910. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55708b22c0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 820224, current: 820132; changing to 820225. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 820225, current: 820081; changing to 820226. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55708b22c0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 829251, current: 829089; changing to 829252. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55708b22c0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 849044, current: 848850; changing to 849045. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55708b22c0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 850495, current: 850261; changing to 850496. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 850496, current: 850127; changing to 850497. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 850497, current: 850365; changing to 850498. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x55707fa990] Non-monotonous DTS in output stream 0:1; previous: 850498, current: 850390; changing to 850499. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [h264 @ 0x5570889a20] error while decoding MB 16 12, bytestream -7
ffmpeg.back_porch.detect       ERROR   : rtsp://back:back@192.168.1.203/live: corrupt decoded frame in stream 0
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x55708b2700] Failed to update header with correct duration.
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x55708b2700] Failed to update header with correct filesize.
watchdog.back_porch            INFO    : No frames received from back_porch in 20 seconds. Exiting ffmpeg...
watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : back_porch: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_porch: ffmpeg sent a broken frame. read of closed file
frigate.video                  INFO    : back_porch: ffmpeg process is not running. exiting capture thread...
ffmpeg.back_porch.detect       ERROR   : Guessed Channel Layout for Input Stream #0.1 : mono
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Too many bits 8832.000000 > 6144 per frame requested, clamping to max
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 2 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 17572, current: 17491; changing to 17573. This may result in incorrect timestamps in the output file.
Exception in thread Thread-4:
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 17573, current: 17265; changing to 17574. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 17574, current: 17073; changing to 17575. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 23365, current: 23193; changing to 23366. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 175898, current: 175716; changing to 175899. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 179275, current: 179118; changing to 179276. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 184617, current: 184417; changing to 184618. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [rtsp @ 0x556bd05a50] CSeq 10 expected, 9 received.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x556bda6570] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 196386, current: 196309; changing to 196387. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x556bd28990] Non-monotonous DTS in output stream 0:1; previous: 196387, current: 196092; changing to 196388. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x556bda6e00] Failed to update header with correct duration.
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x556bda6e00] Failed to update header with correct filesize.
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/frigate/frigate/video.py", line 185, in run
    self.start_ffmpeg_detect()
  File "/opt/frigate/frigate/video.py", line 209, in start_ffmpeg_detect
    self.ffmpeg_detect_process = start_or_restart_ffmpeg(ffmpeg_cmd, self.logger, self.logpipe, self.frame_size)
  File "/opt/frigate/frigate/video.py", line 98, in start_or_restart_ffmpeg
    process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, stderr=logpipe, stdin = sp.DEVNULL, bufsize=frame_size*10, start_new_session=True)
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1637, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory

I'm not sure if that means ffmpeg is restarting, but there's definitely something going on 🤔

blakeblackshear commented 3 years ago

Looks like it is. Can you run ps -aux inside the container after it has been running for a while, but before it runs out of memory? There must be some lingering processes.

drinfernoo commented 3 years ago

Sure can 😉 

root@c25e28c0c111:/opt/frigate# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  3.4  3.6 1729328 138996 ?      Ssl  15:47   2:15 python3 -u -m frigate
root          19  0.0  0.1  15868  5252 ?        Ss   15:47   0:00 nginx: master process /usr/sbin/nginx
nobody        20  0.7  0.2  16684  9104 ?        S    15:47   0:29 nginx: worker process
root          33  0.0  1.8 488524 69024 ?        S    15:47   0:00 frigate.logger
root          37  0.3  0.2  13468 10092 ?        S    15:47   0:14 /usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(29)
root          38 56.0  2.3 703788 89624 ?        Sl   15:47  36:51 frigate.detector.cpu1
root          40 55.9  2.3 703796 89632 ?        Sl   15:47  36:45 frigate.detector.cpu2
root          43  6.2  2.7 1008604 106288 ?      Sl   15:47   4:05 frigate.process:front_porch
root          44  6.3  2.7 1008600 106148 ?      Sl   15:47   4:08 frigate.process:back_porch
root          45  4.4  2.4 796248 91580 ?        Sl   15:47   2:57 python3 -u -m frigate
root          48  4.5  2.3 839604 90732 ?        Sl   15:47   2:57 python3 -u -m frigate
root          49 31.2  2.3 681312 88688 ?        Ssl  15:47  20:33 ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i
root         873 34.0  2.2 681288 84416 ?        Ssl  16:09  15:01 ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i
root        1975  0.3  0.0   3876  2980 pts/0    Ss   16:53   0:00 bash
root        1986  0.0  0.0   5464  2236 pts/0    R+   16:53   0:00 ps -aux
blakeblackshear commented 3 years ago

I don't see anything unexpected here, and nothing is using much memory. Can you also run ls -lah /dev/shm as well? I don't think that should be maxing out memory since you haven't modified the shm_size, but worth checking if there are lingering frames.

drinfernoo commented 3 years ago

From inside the container, I'm guessing:

root@c25e28c0c111:/opt/frigate# ls -lah /dev/shm
total 16M
drwxrwxrwt  2 root root  660 Jul 23 06:08 .
drwxr-xr-x 10 root root 4.3K Jul 22 15:47 ..
-rw-------  1 root root 300K Jul 22 15:49 back_porch
-rw-------  1 root root 3.0M Jul 22 17:39 back_porch1627000745.411959
-rw-------  1 root root 3.0M Jul 22 18:16 back_porch1627002996.842759
-rw-------  1 root root 3.0M Jul 22 18:38 back_porch1627004302.397539
-rw-------  1 root root 3.0M Jul 22 18:42 back_porch1627004551.668893
-rw-------  1 root root 3.0M Jul 22 18:58 back_porch1627005516.725191
-rw-------  1 root root 3.0M Jul 22 19:56 back_porch1627008977.94357
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009305.439206
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768038
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768511
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768745
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768967
-rw-------  1 root root 3.0M Jul 22 22:44 back_porch1627019046.411611
-rw-------  1 root root 3.0M Jul 23 06:00 back_porch1627045207.41895
-rw-------  1 root root 3.0M Jul 23 06:08 back_porch1627045693.268784
-rw-------  1 root root 3.0M Jul 23 06:08 back_porch1627045693.461923
-rw-------  1 root root 300K Jul 22 15:48 front_porch
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000261.894882
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.335087
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.336026
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.336829
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.337832
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.338705
-rw-------  1 root root 3.0M Jul 22 18:37 front_porch1627004251.949842
-rw-------  1 root root 3.0M Jul 23 06:08 front_porch1627045690.598364
-rw-------  1 root root 3.0M Jul 23 06:08 front_porch1627045692.00582
-rw-------  1 root root 3.0M Jul 23 06:08 front_porch1627045693.010797
-rw-------  1 root root 3.0M Jul 23 06:08 front_porch1627045693.39429
-rw-------  1 root root 3.0M Jul 23 06:08 front_porch1627045693.610831
-rw-------  1 root root  480 Jul 22 15:49 out-back_porch
-rw-------  1 root root  480 Jul 22 15:48 out-front_porch
blakeblackshear commented 3 years ago

Can you check again in a few hours? Some of those frames do look old.

drinfernoo commented 3 years ago

Here's one after a few more hours:

root@c25e28c0c111:/opt/frigate# ls -lah /dev/shm
total 16M
drwxrwxrwt  2 root root 1.2K Jul 23 09:13 .
drwxr-xr-x 10 root root 4.3K Jul 22 15:47 ..
-rw-------  1 root root 300K Jul 22 15:49 back_porch
-rw-------  1 root root 3.0M Jul 22 17:39 back_porch1627000745.411959
-rw-------  1 root root 3.0M Jul 22 18:16 back_porch1627002996.842759
-rw-------  1 root root 3.0M Jul 22 18:38 back_porch1627004302.397539
-rw-------  1 root root 3.0M Jul 22 18:42 back_porch1627004551.668893
-rw-------  1 root root 3.0M Jul 22 18:58 back_porch1627005516.725191
-rw-------  1 root root 3.0M Jul 22 19:56 back_porch1627008977.94357
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009305.439206
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768038
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768511
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768745
-rw-------  1 root root 3.0M Jul 22 20:01 back_porch1627009315.768967
-rw-------  1 root root 3.0M Jul 22 22:44 back_porch1627019046.411611
-rw-------  1 root root 3.0M Jul 23 06:00 back_porch1627045207.41895
-rw-------  1 root root 3.0M Jul 23 06:21 back_porch1627046515.105665
-rw-------  1 root root 3.0M Jul 23 06:22 back_porch1627046535.58308
-rw-------  1 root root 3.0M Jul 23 06:22 back_porch1627046535.585228
-rw-------  1 root root 3.0M Jul 23 06:22 back_porch1627046535.591606
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056241.465402
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056244.526403
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056244.52748
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056244.529919
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056244.530951
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056244.532986
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056244.534893
-rw-------  1 root root 3.0M Jul 23 09:04 back_porch1627056271.566617
-rw-------  1 root root 3.0M Jul 23 09:06 back_porch1627056371.947323
-rw-------  1 root root 3.0M Jul 23 09:06 back_porch1627056373.007196
-rw-------  1 root root 3.0M Jul 23 09:06 back_porch1627056373.00975
-rw-------  1 root root 3.0M Jul 23 09:06 back_porch1627056381.983937
-rw-------  1 root root 3.0M Jul 23 09:06 back_porch1627056412.105337
-rw-------  1 root root 3.0M Jul 23 09:07 back_porch1627056422.143932
-rw-------  1 root root 3.0M Jul 23 09:13 back_porch1627056784.608111
-rw-------  1 root root 3.0M Jul 23 09:13 back_porch1627056784.77138
-rw-------  1 root root 300K Jul 22 15:48 front_porch
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000261.894882
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.335087
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.336026
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.336829
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.337832
-rw-------  1 root root 3.0M Jul 22 17:31 front_porch1627000288.338705
-rw-------  1 root root 3.0M Jul 22 18:37 front_porch1627004251.949842
-rw-------  1 root root 3.0M Jul 23 08:24 front_porch1627053890.627164
-rw-------  1 root root 3.0M Jul 23 08:24 front_porch1627053895.779469
-rw-------  1 root root 3.0M Jul 23 09:00 front_porch1627056003.506461
-rw-------  1 root root 3.0M Jul 23 09:03 front_porch1627056216.540914
-rw-------  1 root root 3.0M Jul 23 09:04 front_porch1627056276.775787
-rw-------  1 root root 3.0M Jul 23 09:04 front_porch1627056286.811937
-rw-------  1 root root 3.0M Jul 23 09:05 front_porch1627056357.083666
-rw-------  1 root root 3.0M Jul 23 09:05 front_porch1627056358.129189
-rw-------  1 root root 3.0M Jul 23 09:05 front_porch1627056358.132797
-rw-------  1 root root 3.0M Jul 23 09:06 front_porch1627056367.109903
-rw-------  1 root root 3.0M Jul 23 09:13 front_porch1627056782.05597
-rw-------  1 root root 3.0M Jul 23 09:13 front_porch1627056782.380104
-rw-------  1 root root 3.0M Jul 23 09:13 front_porch1627056783.005743
-rw-------  1 root root 3.0M Jul 23 09:13 front_porch1627056783.30133
-rw-------  1 root root 3.0M Jul 23 09:13 front_porch1627056784.725366
-rw-------  1 root root  480 Jul 22 15:49 out-back_porch
-rw-------  1 root root  480 Jul 22 15:48 out-front_porch

And my logs to go with it for some time previously:

root@ERROL:~# docker logs frigate -fn 50
[h264 @ 0x5573e42140] error while decoding MB 1 56, bytestream -5
ffmpeg.front_porch.detect      ERROR   : Guessed Channel Layout for Input Stream #0.1 : mono
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Too many bits 8832.000000 > 6144 per frame requested, clamping to max
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   :     Last message repeated 1 times
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 15339, current: 15194; changing to 15340. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 15340, current: 14950; changing to 15341. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 104010, current: 103981; changing to 104011. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 204949, current: 204903; changing to 204950. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 260788, current: 260675; changing to 260789. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 476819, current: 476718; changing to 476820. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 672448, current: 672427; changing to 672449. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 841279, current: 841181; changing to 841280. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1113870, current: 1113856; changing to 1113871. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1226670, current: 1226505; changing to 1226671. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1308622, current: 1308587; changing to 1308623. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   :     Last message repeated 1 times
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1832774, current: 1832724; changing to 1832775. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1832775, current: 1832673; changing to 1832776. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1848994, current: 1848825; changing to 1848995. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1848995, current: 1848898; changing to 1848996. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1864106, current: 1863963; changing to 1864107. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1870829, current: 1870790; changing to 1870830. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1897295, current: 1897162; changing to 1897296. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1969363, current: 1969297; changing to 1969364. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1985415, current: 1985198; changing to 1985416. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [aac @ 0x556e0ae0c0] Queue input is backward in time
ffmpeg.front_porch.detect      ERROR   :     Last message repeated 1 times
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1987016, current: 1986804; changing to 1987017. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [segment @ 0x556e096990] Non-monotonous DTS in output stream 0:1; previous: 1987017, current: 1986675; changing to 1987018. This may result in incorrect timestamps in the output file.
ffmpeg.front_porch.detect      ERROR   : [h264 @ 0x556e126440] error while decoding MB 1 56, bytestream -5
ffmpeg.front_porch.detect      ERROR   : rtsp://front:front@192.168.1.202/live: corrupt decoded frame in stream 0
ffmpeg.front_porch.detect      ERROR   : [flv @ 0x556e114220] Failed to update header with correct duration.
ffmpeg.front_porch.detect      ERROR   : [flv @ 0x556e114220] Failed to update header with correct filesize.
blakeblackshear commented 3 years ago

Definitely seems to be growing. Let's check once more in a few hours again.

drinfernoo commented 3 years ago

So I realized I wasn't getting notifications from my cameras (from a Home Assistant automation), and decided to check the logs:

root@ERROL:~# docker logs frigate -n 100
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x5565aca7d9, 0x16)
        /usr/local/go/src/runtime/panic.go:774 +0x54 fp=0x7ff727de80 sp=0x7ff727de50 pc=0x556480b1f4
runtime.sysMap(0x4000000000, 0x4000000, 0x55675e1a18)
        /usr/local/go/src/runtime/mem_linux.go:169 +0xbc fp=0x7ff727dec0 sp=0x7ff727de80 pc=0x55647f4d6c
runtime.(*mheap).sysAlloc(0x55675c8860, 0x2000, 0x0, 0x0)
        /usr/local/go/src/runtime/malloc.go:701 +0x168 fp=0x7ff727df70 sp=0x7ff727dec0 pc=0x55647e8a18
runtime.(*mheap).grow(0x55675c8860, 0x1, 0xffffffff)
        /usr/local/go/src/runtime/mheap.go:1255 +0x9c fp=0x7ff727dfc0 sp=0x7ff727df70 pc=0x5564802dbc
runtime.(*mheap).allocSpanLocked(0x55675c8860, 0x1, 0x55675e1a28, 0x0)
        /usr/local/go/src/runtime/mheap.go:1170 +0x220 fp=0x7ff727e040 sp=0x7ff727dfc0 pc=0x5564802c60
runtime.(*mheap).alloc_m(0x55675c8860, 0x1, 0x2a, 0x0)
        /usr/local/go/src/runtime/mheap.go:1022 +0xb0 fp=0x7ff727e090 sp=0x7ff727e040 pc=0x5564802530
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:1093 +0x48 fp=0x7ff727e0d0 sp=0x7ff727e090 pc=0x5564832838
runtime.(*mheap).alloc(0x55675c8860, 0x1, 0x7ff701002a, 0x55647f3fa4)
        /usr/local/go/src/runtime/mheap.go:1092 +0x6c fp=0x7ff727e120 sp=0x7ff727e0d0 pc=0x556480283c
runtime.(*mcentral).grow(0x55675c9698, 0x0)
        /usr/local/go/src/runtime/mcentral.go:255 +0x78 fp=0x7ff727e170 sp=0x7ff727e120 pc=0x55647f47d8
runtime.(*mcentral).cacheSpan(0x55675c9698, 0x55647ea338)
        /usr/local/go/src/runtime/mcentral.go:106 +0x35c fp=0x7ff727e1e0 sp=0x7ff727e170 pc=0x55647f42ac
runtime.(*mcache).refill(0x7f813a0008, 0x2a)
        /usr/local/go/src/runtime/mcache.go:138 +0xa4 fp=0x7ff727e210 sp=0x7ff727e1e0 pc=0x55647f3cd4
runtime.(*mcache).nextFree(0x7f813a0008, 0x55647ea12a, 0x7ff727e2d8, 0x55648312a0, 0x55675b7780)
        /usr/local/go/src/runtime/malloc.go:854 +0x8c fp=0x7ff727e260 sp=0x7ff727e210 pc=0x55647e90dc
runtime.mallocgc(0x180, 0x556651d260, 0x7f813a4001, 0x5564831260)
        /usr/local/go/src/runtime/malloc.go:1022 +0x6a8 fp=0x7ff727e320 sp=0x7ff727e260 pc=0x55647e9918
runtime.newobject(0x556651d260, 0x7ff727e338)
        /usr/local/go/src/runtime/malloc.go:1151 +0x38 fp=0x7ff727e350 sp=0x7ff727e320 pc=0x55647e9cc8
runtime.malg(0x424e00008000, 0x556480dafc)
        /usr/local/go/src/runtime/proc.go:3238 +0x2c fp=0x7ff727e3a0 sp=0x7ff727e350 pc=0x55648148fc
runtime.mpreinit(...)
        /usr/local/go/src/runtime/os_linux.go:324
runtime.mcommoninit(0x55675c2b20)
        /usr/local/go/src/runtime/proc.go:626 +0xc8 fp=0x7ff727e3e0 sp=0x7ff727e3a0 pc=0x556480dee8
runtime.schedinit()
        /usr/local/go/src/runtime/proc.go:543 +0x7c fp=0x7ff727e450 sp=0x7ff727e3e0 pc=0x556480db0c
runtime.rt0_go(0x55647de144, 0x0, 0x0, 0x55647de110, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/runtime/asm_arm64.s:70 +0xb8 fp=0x7ff727e480 sp=0x7ff727e450 pc=0x5564834be8

I'm also no longer able to terminal into the container, unsurprisingly 😅 In fact, I was also unable to docker restart frigate... So I rebooted my box to get more logs.

After rebooting, I got this log snippet:

root@ERROL:~# docker logs frigate -fn 100
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 813557, current: 813500; changing to 813558. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 840597, current: 840395; changing to 840598. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 840598, current: 840255; changing to 840599. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 841038, current: 840891; changing to 841039. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 841915, current: 841759; changing to 841916. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 841916, current: 841650; changing to 841917. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 841917, current: 841665; changing to 841918. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 841918, current: 841469; changing to 841919. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 842493, current: 842266; changing to 842494. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 842494, current: 842245; changing to 842495. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 842495, current: 842025; changing to 842496. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 842496, current: 841902; changing to 842497. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 842926, current: 842800; changing to 842927. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 868868, current: 868802; changing to 868869. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 895927, current: 895921; changing to 895928. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 2 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 897066, current: 896987; changing to 897067. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 897067, current: 896847; changing to 897068. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 897068, current: 896722; changing to 897069. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 962779, current: 962654; changing to 962780. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 962780, current: 962570; changing to 962781. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 967717, current: 967489; changing to 967718. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 967915, current: 967694; changing to 967916. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 967916, current: 967468; changing to 967917. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968492, current: 968310; changing to 968493. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968493, current: 968314; changing to 968494. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968494, current: 968129; changing to 968495. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968495, current: 967918; changing to 968496. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968942, current: 968731; changing to 968943. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968943, current: 968530; changing to 968944. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968944, current: 968377; changing to 968945. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 968945, current: 968321; changing to 968946. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1100032, current: 1100004; changing to 1100033. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1102144, current: 1102063; changing to 1102145. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1137382, current: 1137227; changing to 1137383. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1137585, current: 1137478; changing to 1137586. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1428931, current: 1428831; changing to 1428932. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1512259, current: 1512256; changing to 1512260. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1982529, current: 1982497; changing to 1982530. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1983823, current: 1983603; changing to 1983824. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x558c937030] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1984207, current: 1984013; changing to 1984208. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x558c880990] Non-monotonous DTS in output stream 0:1; previous: 1985037, current: 1984824; changing to 1985038. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [rtsp @ 0x558c85da50] CSeq 56 expected, 48 received.
ffmpeg.back_porch.detect       ERROR   : [h264 @ 0x558ca04910] error while decoding MB 6 28, bytestream -5
ffmpeg.back_porch.detect       ERROR   : rtsp://back:back@192.168.1.203/live: corrupt decoded frame in stream 0
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x558c937470] Failed to update header with correct duration.
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x558c937470] Failed to update header with correct filesize.
frigate.app                    INFO    : Stopping...
frigate.object_processing      INFO    : Exiting object processor...
frigate.record                 INFO    : Exiting recording maintenance...
frigate.events                 INFO    : Exiting event cleanup...
 * Starting nginx nginx
   ...done.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
frigate.app                    INFO    : Camera processor started for front_porch: 45
frigate.app                    INFO    : Camera processor started for back_porch: 46
frigate.app                    INFO    : Capture process started for front_porch: 47
frigate.app                    INFO    : Capture process started for back_porch: 50
detector.cpu2                  INFO    : Starting detection process: 41
detector.cpu1                  INFO    : Starting detection process: 40
watchdog.back_porch            INFO    : No frames received from back_porch in 20 seconds. Exiting ffmpeg...
watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...

And this stuff from inside the container:

root@c25e28c0c111:/opt/frigate# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  8.1  2.7 1280684 104264 ?      Ssl  11:50   0:11 python3 -u -m frigate
root          20  0.0  0.1  15868  5252 ?        Ss   11:50   0:00 nginx: master process /usr/sbin/nginx
nobody        21  0.5  0.2  16664  9020 ?        S    11:50   0:00 nginx: worker process
root          35  0.0  1.7 488524 65316 ?        S    11:50   0:00 frigate.logger
root          39  0.4  0.2  13468 10100 ?        R    11:50   0:00 /usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(29)
root          40 95.1  2.2 703784 86040 ?        Rl   11:50   2:02 frigate.detector.cpu1
root          41 95.0  2.2 703796 86196 ?        Sl   11:50   2:02 frigate.detector.cpu2
root          45  9.2  2.7 1011648 102640 ?      Sl   11:50   0:11 frigate.process:front_porch
root          46  2.5  2.1 986636 80120 ?        Sl   11:50   0:03 frigate.process:back_porch
root          47  5.6  2.2 796248 84592 ?        Sl   11:50   0:07 python3 -u -m frigate
root          50  4.2  2.2 839604 84920 ?        Sl   11:50   0:05 python3 -u -m frigate
root          51 35.1  1.9 681136 75672 ?        Ssl  11:50   0:45 ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i
root         527 36.6  1.9 681212 73788 ?        Ssl  11:51   0:27 ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i
root        1120  1.0  0.0   3872  3044 pts/0    Ss   11:52   0:00 bash
root        1158  0.0  0.0   2056   492 ?        S    11:52   0:00 /bin/sh -c ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 /tmp/cache/back_porch-20210723110538.mp4
root        1159  0.0  0.8  65908 31904 ?        R    11:52   0:00 ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 /tmp/cache/back_porch-20210723110538.mp4
root        1160  0.0  0.0   5464  2348 pts/0    R+   11:52   0:00 ps -aux
root@c25e28c0c111:/opt/frigate# ls -lah /dev/shm
total 13M
drwxrwxrwt  2 root root  240 Jul 23 11:52 .
drwxr-xr-x 10 root root 4.3K Jul 23 11:50 ..
-rw-------  1 root root 300K Jul 23 11:51 back_porch
-rw-------  1 root root 3.0M Jul 23 11:52 back_porch1627066371.944827
-rw-------  1 root root 3.0M Jul 23 11:52 back_porch1627066372.283044
-rw-------  1 root root 300K Jul 23 11:50 front_porch
-rw-------  1 root root 3.0M Jul 23 11:52 front_porch1627066371.736839
-rw-------  1 root root 3.0M Jul 23 11:52 front_porch1627066371.817634
-rw-------  1 root root 3.0M Jul 23 11:52 front_porch1627066372.084095
-rw-------  1 root root 3.0M Jul 23 11:52 front_porch1627066372.327728
-rw-------  1 root root  480 Jul 23 11:51 out-back_porch
-rw-------  1 root root  480 Jul 23 11:50 out-front_porch
drinfernoo commented 3 years ago

Is my device just running out of memory for Frigate? This seems to happen "faster" if I use my Coral device... which is odd because I've had it installed for a few months without really any issue 🤔 I'm going to attach it again, change my detectors, and see if my issues present themselves again the same way.

drinfernoo commented 3 years ago

Decided to check today after realizing I hadn't gotten anything from Frigate for awhile...

root@ERROL:~# docker logs -n 50 frigate
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 3844094, current: 3843951; changing to 3844095. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 3844095, current: 3843758; changing to 3844096. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 3848028, current: 3847860; changing to 3848029. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 6719992, current: 6719910; changing to 6719993. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 6885304, current: 6885080; changing to 6885305. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 8447370, current: 8447258; changing to 8447371. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 9516160, current: 9516105; changing to 9516161. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 9648577, current: 9648531; changing to 9648578. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 10061253, current: 10061121; changing to 10061254. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 11711806, current: 11711667; changing to 11711807. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 13583034, current: 13582919; changing to 13583035. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 13614703, current: 13614642; changing to 13614704. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 14126338, current: 14126230; changing to 14126339. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 14243289, current: 14243135; changing to 14243290. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 14821685, current: 14821475; changing to 14821686. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [h264 @ 0x557950d5a0] Increasing reorder buffer to 1
ffmpeg.back_porch.detect       ERROR   : [aac @ 0x55794ff8f0] Queue input is backward in time
ffmpeg.back_porch.detect       ERROR   : [segment @ 0x5579481990] Non-monotonous DTS in output stream 0:1; previous: 17902612, current: 17902581; changing to 17902613. This may result in incorrect timestamps in the output file.
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x55794ffd30] Failed to update header with correct duration.
ffmpeg.back_porch.detect       ERROR   : [flv @ 0x55794ffd30] Failed to update header with correct filesize.
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/frigate/frigate/video.py", line 185, in run
    self.start_ffmpeg_detect()
  File "/opt/frigate/frigate/video.py", line 209, in start_ffmpeg_detect
    self.ffmpeg_detect_process = start_or_restart_ffmpeg(ffmpeg_cmd, self.logger, self.logpipe, self.frame_size)
  File "/opt/frigate/frigate/video.py", line 98, in start_or_restart_ffmpeg
    process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, stderr=logpipe, stdin = sp.DEVNULL, bufsize=frame_size*10, start_new_session=True)
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1637, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
root@c25e28c0c111:/opt/frigate# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  2.9  3.1 1729784 121520 ?      Ssl  Jul23  47:07 python3 -u -m frigate
root          21  0.0  0.1  15868  5240 ?        Ss   Jul23   0:00 nginx: master process /usr/sbin/nginx
nobody        22  0.5  0.1  16828  6968 ?        S    Jul23   9:33 nginx: worker process
root          35  0.0  1.6 488524 61312 ?        S    Jul23   0:27 frigate.logger
root          39  0.2  0.1  13468  5008 ?        S    Jul23   4:43 /usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(29)
root          43  7.3  2.4 1008928 91644 ?       Sl   Jul23 118:26 frigate.process:front_porch
root          44  3.2  2.4 1008620 91432 ?       Sl   Jul23  51:28 frigate.process:back_porch
root          45  3.3  3.0 836552 115848 ?       Sl   Jul23  54:23 python3 -u -m frigate
root          46  3.6  2.6 836552 102340 ?       Sl   Jul23  57:43 python3 -u -m frigate
root       32482  0.0  0.0      0     0 ?        Z    11:03   0:00 [sh] <defunct>
root       32483  0.5  3.0 1646560 114936 ?      Sl   11:03   1:05 frigate.detector.coral
root       32898  1.0  0.0   3872  2692 pts/0    Ss   14:37   0:00 bash
root       32907  0.0  0.0   5464  2152 pts/0    R+   14:37   0:00 ps -aux
root@c25e28c0c111:/opt/frigate# ls -lah /dev/shm
total 6.6M
drwxrwxrwt  2 root root  880 Jul 24 13:20 .
drwxr-xr-x 10 root root 4.3K Jul 23 11:55 ..
-rw-------  1 root root 300K Jul 23 11:56 back_porch
-rw-------  1 root root 3.0M Jul 23 12:22 back_porch1627068159.760846
-rw-------  1 root root 3.0M Jul 23 15:00 back_porch1627077651.925241
-rw-------  1 root root 3.0M Jul 23 15:09 back_porch1627078160.592537
-rw-------  1 root root 3.0M Jul 23 15:13 back_porch1627078413.552338
-rw-------  1 root root 3.0M Jul 23 15:41 back_porch1627080086.001767
-rw-------  1 root root 3.0M Jul 23 15:48 back_porch1627080510.607102
-rw-------  1 root root 3.0M Jul 23 16:31 back_porch1627083111.7108
-rw-------  1 root root 3.0M Jul 23 20:30 back_porch1627097433.733909
-rw-------  1 root root 3.0M Jul 24 00:18 back_porch1627111123.359138
-rw-------  1 root root 3.0M Jul 24 07:03 back_porch1627135412.68179
-rw-------  1 root root 3.0M Jul 24 07:04 back_porch1627135453.779106
-rw-------  1 root root 3.0M Jul 24 07:39 back_porch1627137574.473766
-rw-------  1 root root 3.0M Jul 24 10:09 back_porch1627146595.946418
-rw-------  1 root root 3.0M Jul 24 11:07 back_porch1627150020.033788
-rw-------  1 root root 3.0M Jul 24 13:18 back_porch1627157928.71699
-rw-------  1 root root 300K Jul 23 11:56 front_porch
-rw-------  1 root root 3.0M Jul 23 12:23 front_porch1627068185.197091
-rw-------  1 root root 3.0M Jul 23 12:59 front_porch1627070364.559324
-rw-------  1 root root 3.0M Jul 23 14:59 front_porch1627077571.708737
-rw-------  1 root root 3.0M Jul 23 15:13 front_porch1627078418.613594
-rw-------  1 root root 3.0M Jul 23 15:13 front_porch1627078423.744951
-rw-------  1 root root 3.0M Jul 23 15:13 front_porch1627078423.746422
-rw-------  1 root root 3.0M Jul 23 15:13 front_porch1627078423.747681
-rw-------  1 root root 3.0M Jul 23 15:13 front_porch1627078423.748918
-rw-------  1 root root 3.0M Jul 23 15:13 front_porch1627078423.756283
-rw-------  1 root root 3.0M Jul 23 15:47 front_porch1627080458.434526
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080860.79402
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080881.252355
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080881.253644
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080881.255245
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080881.256518
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080881.258009
-rw-------  1 root root 3.0M Jul 23 15:54 front_porch1627080881.269378
-rw-------  1 root root 3.0M Jul 23 16:18 front_porch1627082282.206602
-rw-------  1 root root 3.0M Jul 23 21:36 front_porch1627101385.479861
-rw-------  1 root root 3.0M Jul 24 06:00 front_porch1627131636.70735
-rw-------  1 root root 3.0M Jul 24 07:39 front_porch1627137574.473766
-rw-------  1 root root 3.0M Jul 24 09:29 front_porch1627144141.737098
-rw-------  1 root root 3.0M Jul 24 11:12 front_porch1627150353.146263
-rw-------  1 root root  480 Jul 24 11:03 out-back_porch
-rw-------  1 root root  480 Jul 24 11:03 out-front_porch

That's with my Coral USB attached and configured as detector.

blakeblackshear commented 3 years ago

I don't think this has anything to do with which detector is configured. I definitely see old frames in shm, but not that many.

What does memory usage look like on the host with something like htop?

drinfernoo commented 3 years ago

I'm guessing this would be considered high usage? 😅 image

blakeblackshear commented 3 years ago

Can you sort that by memory usage? I don't see any processes with high memory usage.

drinfernoo commented 3 years ago

Ah, sorry about that... Never used htop before 😅 image

drinfernoo commented 3 years ago

I did restart Frigate after posting my comment earlier when it had run out of memory, if that makes any difference.

blakeblackshear commented 3 years ago

Next time it happens, stop the frigate container and see if your memory usage is still so high. It may not be a frigate issue if memory usage is that high immediately after a restart of the container. It looks like home assistant is using almost all of the system memory.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

drinfernoo commented 2 years ago

I've since moved my whole operation to a different computer with better specs in every regard, and haven't run into this at all anymore.