[Support]: 1 hour lost recording and high memory and CPU usage when Daylight Saving Time (DST) occurred.

deviant77 commented 1 year ago

Describe the problem you are having

At 02:00 this morning the clock went back to 01:00 for the end of Daylight Saving Time.

For one hour, Frigate's log went crazy (6,512 entries) and all recordings were lost, and the memory and CPU usage starting increasing dramatically.

In the recordings folder, the "01" folder is missing, and the "00" folder was created at 01:00:

$ ll frigate/recordings/2023-10-29/
drwxrwxrwx+  8 root root 4096 Oct 29 01:00 00
drwxrwxrwx+  8 root root 4096 Oct 29 02:00 02
drwxrwxrwx+  8 root root 4096 Oct 29 03:00 03

When I checked on Frigate at around lunchtime, the memory usage of frigate.recording_manager was 44.1% of 8 GB (usually ~1.3%), and the CPU usage was 44.3% (usually ~5.9%). I had to restart Frigate to return to normal memory and CPU usage.

Version

0.13.0-cd64399 (beta 3)

Frigate config file

# yaml-language-server: $schema=http://192.168.0.10:8001/api/config/schema.json

mqtt:
  host: 192.168.0.10
  user: "{FRIGATE_MQTT_USER}"
  password: "{FRIGATE_MQTT_PASSWORD}"

detectors:
  coral:
    type: edgetpu
    device: usb

database:
  path: /database/frigate.db

logger:
  logs:
    frigate.comms.dispatcher: warning

birdseye:
  enabled: true
  width: 1920
  height: 1080
  quality: 8
  mode: continuous

ffmpeg:
  hwaccel_args: preset-vaapi
  input_args: preset-rtsp-restream-low-latency
  output_args:
    record: preset-record-generic-audio-copy

detect:
  width: 640
  height: 360
  fps: 4
  annotation_offset: -1400

motion:
  improve_contrast: True

record:
  enabled: True
  expire_interval: 720
  retain:
    days: 60
    mode: all

ui:
  live_mode: mse

model:
  labelmap:
    2: vehicle
    3: vehicle
    7: vehicle
    15: animal
    16: animal
    17: animal

go2rtc:
  streams:
    front:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_FRONT}@192.168.0.11:554/H264/ch1/main/av_stream
      - ffmpeg:front#audio=opus
    front_sub:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_FRONT}@192.168.0.11:554/H264/ch1/sub/av_stream
    side:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_SIDE}@192.168.0.12:554/H264/ch1/main/av_stream
      - ffmpeg:side#audio=opus
    side_sub:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_SIDE}@192.168.0.12:554/H264/ch1/sub/av_stream
    crossroad:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_CROSSROAD}@192.168.0.13:554/H264/ch1/main/av_stream
      - ffmpeg:crossroad#audio=opus
    crossroad_sub:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_CROSSROAD}@192.168.0.13:554/H264/ch1/sub/av_stream
    garage:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_GARAGE}@192.168.0.14:554/H264/ch1/main/av_stream
      - ffmpeg:garage#audio=opus
    garage_sub:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_GARAGE}@192.168.0.14:554/H264/ch1/sub/av_stream
    garden:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_GARDEN}@192.168.0.15:554/H264/ch1/main/av_stream
      - ffmpeg:garden#audio=opus
    garden_sub:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_GARDEN}@192.168.0.15:554/H264/ch1/sub/av_stream
    pond:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_POND}@192.168.0.16:554/H264/ch1/main/av_stream
      - ffmpeg:pond#audio=opus
    pond_sub:
      - rtsp://frigate:{FRIGATE_RTSP_PASSWORD_POND}@192.168.0.16:554/H264/ch1/sub/av_stream

cameras:

  front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front?video=copy&audio=aac
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_sub
          roles:
            - detect
    objects:
      track:
        - animal
        - person
      mask:
        - 58,335,205,328,191,221,62,245
        - 311,98,269,100,278,148,319,147
        - 495,165,487,289,354,268,392,128
        - 0,301,105,333,293,336,352,355,496,318,545,360,0,360
    record:
      events:
        required_zones:
          - front_garden
    motion:
      mask:
        - 0,298,73,325,257,327,364,360,504,301,557,360,0,360
        - 456,205,511,165,531,107,484,54,424,65,382,106,410,196
        - 175,255,210,156,173,98,100,106,44,139,63,220,137,274
        - 0,0,640,0,640,41,594,30,427,15,323,12,181,33,52,68,0,92
    zones:
      front_garden:
        coordinates: 640,360,640,360,640,315,577,211,516,150,442,119,335,109,281,121,210,132,62,157,0,177,0,360
    ui:
      order: 0
    birdseye:
      order: 0

  side:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/side?video=copy&audio=aac
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/side_sub
          roles:
            - detect
    objects:
      track:
        - person
      filters:
        person:
          min_area: 900
    record:
      events:
        required_zones:
          - side_path
    motion:
      mask:
        - 91,201,245,166,383,143,640,105,640,0,0,0,0,360,640,360,640,320,435,338,242,331,80,298
    zones:
      side_path:
        coordinates: 640,360,640,262,543,287,356,302,147,293,143,360
    ui:
      order: 1
    birdseye:
      order: 1

  crossroad:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/crossroad?video=copy&audio=aac
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/crossroad_sub
          roles:
            - detect
    objects:
      track:
        - vehicle
    record:
      events:
        required_zones:
          - driveway
    motion:
      mask:
        - 0,0,640,0,640,360,539,360,575,292,544,235,610,152,458,91,212,130,0,178
    zones:
      driveway:
        coordinates: 447,339,539,252,449,251,245,360
        objects:
          - vehicle
    ui:
      order: 2
    birdseye:
      order: 2

  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage?video=copy&audio=aac
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garage_sub
          roles:
            - detect
    motion:
      mask:
        - 640,360,640,323,431,322,430,350,352,360,213,353,105,331,46,197,88,81,134,0,0,0,0,360
        - 640,83,640,0,593,0
        - 411,64,394,180,225,168,284,24
    objects:
      track:
        - animal
        - person
      filters:
        person:
          mask:
            - 527,258,437,284,432,192,509,171
            - 367,134,364,171,253,170,255,130
            - 215,22,215,45,187,50,188,24
      mask:
        - 105,331,46,197,88,81,134,0,0,0,0,360
        - 411,64,394,180,225,168,284,24
    ui:
      order: 3
    birdseye:
      order: 3

  garden:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garden?video=copy&audio=aac
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garden_sub
          roles:
            - detect
    objects:
      track:
        - animal
        - person
      filters:
        person:
          mask:
            - 461,234,456,201,439,203,438,237
            - 227,117,225,167,180,172,180,118
            - 383,188,382,206,365,205,366,190
            - 246,109,331,112,337,154,252,155
            - 61,261,61,282,37,284,38,257
            - 446,212,475,212,474,237,447,237
            - 35,327,19,328,32,309,20,312
      mask:
        - 0,167,113,84,155,61,184,43,286,40,458,47,640,154,640,0,0,0
    motion:
      mask:
        - 0,167,113,84,155,61,184,43,286,40,470,51,640,145,640,0,0,0
        - 434,342,625,340,624,324,433,326
    zones:
      rear_garden:
        coordinates: 0,360,640,360,640,298,350,60,176,67,0,223
      neighbour:
        coordinates: 570,239,640,232,640,157,464,54,358,61
    ui:
      order: 4
    birdseye:
      order: 4

  pond:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/pond?video=copy&audio=aac
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/pond_sub
          roles:
            - detect
    objects:
      track:
        - animal
        - person
      mask:
        - 286,333,422,332,420,360,289,360
      filters:
        person:
          mask:
            - 295,182,283,152,247,150,248,178
            - 439,81,440,62,420,66,423,83
            - 344,116,364,116,363,138,342,138
            - 426,62,491,64,490,99,418,95
            - 421,205,392,209,389,178,420,179
    motion:
        mask:
          - 210,326,214,344,5,347,8,324
    ui:
      order: 5
    birdseye:
      order: 5

telemetry:
  stats:
    amd_gpu_stats: False
    intel_gpu_stats: False
    network_bandwidth: False

Relevant log output

2023-10-28 19:18:45.636686017  [INFO] Preparing Frigate...
2023-10-28 19:18:45.740409896  [INFO] Starting Frigate...
2023-10-28 19:18:49.068741026  [2023-10-28 19:18:49] frigate.app                    INFO    : Starting Frigate (0.13.0-cd64399)
2023-10-28 19:18:49.256085970  [2023-10-28 19:18:49] peewee_migrate.logs            INFO    : Starting migrations
2023-10-28 19:18:49.293546422  [2023-10-28 19:18:49] peewee_migrate.logs            INFO    : Migrate "019_create_regions_table"
2023-10-28 19:18:49.293907813  [2023-10-28 19:18:49] peewee_migrate.logs            INFO    : sql ('CREATE TABLE IF NOT EXISTS "regions" ("camera" VARCHAR(20) NOT NULL PRIMARY KEY, "last_update" DATETIME NOT NULL, "grid" JSON)',)
2023-10-28 19:18:49.294448307  [2023-10-28 19:18:49] peewee_migrate.logs            INFO    : Done 019_create_regions_table
2023-10-28 19:20:57.277533025  [2023-10-28 19:20:57] frigate.app                    INFO    : Recording process started: 363
2023-10-28 19:20:57.281596754  [2023-10-28 19:20:57] frigate.app                    INFO    : go2rtc process pid: 85
2023-10-28 19:20:57.330550894  [2023-10-28 19:20:57] detector.coral                 INFO    : Starting detection process: 373
2023-10-28 19:20:57.336260778  [2023-10-28 19:20:57] frigate.app                    INFO    : Output process started: 375
2023-10-28 19:20:57.457326332  [2023-10-28 19:20:57] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-10-28 19:21:00.079734792  [2023-10-28 19:20:57] frigate.app                    INFO    : Camera processor started for front: 397
2023-10-28 19:21:00.079742275  [2023-10-28 19:20:57] frigate.app                    INFO    : Camera processor started for side: 398
2023-10-28 19:21:00.081064662  [2023-10-28 19:20:57] frigate.app                    INFO    : Camera processor started for crossroad: 399
2023-10-28 19:21:00.081072284  [2023-10-28 19:20:57] frigate.app                    INFO    : Camera processor started for garage: 402
2023-10-28 19:21:00.081087348  [2023-10-28 19:20:57] frigate.app                    INFO    : Camera processor started for garden: 403
2023-10-28 19:21:00.081117609  [2023-10-28 19:20:57] frigate.app                    INFO    : Camera processor started for pond: 404
2023-10-28 19:21:00.081120245  [2023-10-28 19:20:57] frigate.app                    INFO    : Capture process started for front: 407
2023-10-28 19:21:00.081122422  [2023-10-28 19:20:57] frigate.app                    INFO    : Capture process started for side: 408
2023-10-28 19:21:00.081124666  [2023-10-28 19:20:57] frigate.app                    INFO    : Capture process started for crossroad: 409
2023-10-28 19:21:00.081126802  [2023-10-28 19:20:57] frigate.app                    INFO    : Capture process started for garage: 413
2023-10-28 19:21:00.081128868  [2023-10-28 19:20:57] frigate.app                    INFO    : Capture process started for garden: 415
2023-10-28 19:21:00.081131301  [2023-10-28 19:20:57] frigate.app                    INFO    : Capture process started for pond: 416
2023-10-28 19:21:00.098979680  [2023-10-28 19:21:00] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-10-29 01:00:53.647334206  [2023-10-29 01:00:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:00:53.647750245  [2023-10-29 01:00:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:00:53.656176095  [2023-10-29 01:00:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:00:53.656596571  [2023-10-29 01:00:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:03.406140300  [2023-10-29 01:01:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:03.420747494  [2023-10-29 01:01:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:03.422181516  [2023-10-29 01:01:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:03.422978877  [2023-10-29 01:01:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:13.407449926  [2023-10-29 01:01:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:13.416122119  [2023-10-29 01:01:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:13.426276504  [2023-10-29 01:01:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:13.427699175  [2023-10-29 01:01:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:23.412561898  [2023-10-29 01:01:23] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:23.424596626  [2023-10-29 01:01:23] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:23.426746817  [2023-10-29 01:01:23] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:23.437103226  [2023-10-29 01:01:23] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:33.408060668  [2023-10-29 01:01:33] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:33.408336621  [2023-10-29 01:01:33] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:33.424037780  [2023-10-29 01:01:33] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:33.436575446  [2023-10-29 01:01:33] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:43.408936890  [2023-10-29 01:01:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:43.413538523  [2023-10-29 01:01:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:43.414221975  [2023-10-29 01:01:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:43.415063109  [2023-10-29 01:01:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:53.409351860  [2023-10-29 01:01:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:53.409553152  [2023-10-29 01:01:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:53.426538082  [2023-10-29 01:01:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:53.427979412  [2023-10-29 01:01:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:01:59.232674787  [2023-10-29 01:01:59] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:01:59.244651247  [2023-10-29 01:01:59] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:01:59.256489031  [2023-10-29 01:01:59] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:01:59.256896085  [2023-10-29 01:01:59] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:01:59.257004216  [2023-10-29 01:01:59] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:01:59.257113946  [2023-10-29 01:01:59] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:01:59.257758571  [2023-10-29 01:01:59] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:01:59.257879569  [2023-10-29 01:01:59] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:01:59.258212107  [2023-10-29 01:01:59] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:01:59.276230040  [2023-10-29 01:01:59] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:01:59.284434546  [2023-10-29 01:01:59] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:01:59.294813371  [2023-10-29 01:01:59] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:01:59.294943830  [2023-10-29 01:01:59] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:01:59.298766634  [2023-10-29 01:01:59] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:01:59.299283093  [2023-10-29 01:01:59] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:01:59.412482866  [2023-10-29 01:01:59] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:01:59.412597898  [2023-10-29 01:01:59] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:01:59.412731681  [2023-10-29 01:01:59] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:03.409130812  [2023-10-29 01:02:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:03.409996216  [2023-10-29 01:02:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:03.420227552  [2023-10-29 01:02:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:03.427826813  [2023-10-29 01:02:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:09.288820104  [2023-10-29 01:02:09] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:09.289315356  [2023-10-29 01:02:09] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:09.289320792  [2023-10-29 01:02:09] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:09.289323168  [2023-10-29 01:02:09] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:09.289325372  [2023-10-29 01:02:09] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:09.289327450  [2023-10-29 01:02:09] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:09.306266854  [2023-10-29 01:02:09] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:09.306752721  [2023-10-29 01:02:09] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:09.306758727  [2023-10-29 01:02:09] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:09.320614095  [2023-10-29 01:02:09] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:09.322969045  [2023-10-29 01:02:09] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:09.322976145  [2023-10-29 01:02:09] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:09.448770949  [2023-10-29 01:02:09] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:09.448878516  [2023-10-29 01:02:09] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:09.449435446  [2023-10-29 01:02:09] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:09.451628692  [2023-10-29 01:02:09] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:09.451635723  [2023-10-29 01:02:09] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:09.451638075  [2023-10-29 01:02:09] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:13.408983152  [2023-10-29 01:02:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:13.424595527  [2023-10-29 01:02:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:13.426501096  [2023-10-29 01:02:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:13.430596710  [2023-10-29 01:02:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:19.316268501  [2023-10-29 01:02:19] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:19.316373822  [2023-10-29 01:02:19] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:19.316454960  [2023-10-29 01:02:19] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:19.321527375  [2023-10-29 01:02:19] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:19.321637795  [2023-10-29 01:02:19] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:19.321730753  [2023-10-29 01:02:19] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:19.347636299  [2023-10-29 01:02:19] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:19.347858547  [2023-10-29 01:02:19] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:19.347862469  [2023-10-29 01:02:19] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:19.348586123  [2023-10-29 01:02:19] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:19.348706123  [2023-10-29 01:02:19] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:19.348796258  [2023-10-29 01:02:19] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:19.468067706  [2023-10-29 01:02:19] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:19.468327849  [2023-10-29 01:02:19] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:19.468332094  [2023-10-29 01:02:19] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:19.495751798  [2023-10-29 01:02:19] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:19.496082582  [2023-10-29 01:02:19] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:19.496086691  [2023-10-29 01:02:19] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:23.410745505  [2023-10-29 01:02:23] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:29.364801081  [2023-10-29 01:02:29] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:29.364895514  [2023-10-29 01:02:29] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:29.364994189  [2023-10-29 01:02:29] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:29.367596444  [2023-10-29 01:02:29] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:29.367722505  [2023-10-29 01:02:29] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:29.367826804  [2023-10-29 01:02:29] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:29.391608560  [2023-10-29 01:02:29] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:29.391720018  [2023-10-29 01:02:29] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:29.391823881  [2023-10-29 01:02:29] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:29.391924991  [2023-10-29 01:02:29] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:29.392025853  [2023-10-29 01:02:29] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:29.392118361  [2023-10-29 01:02:29] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:29.486201504  [2023-10-29 01:02:29] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:29.486402127  [2023-10-29 01:02:29] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:29.486416584  [2023-10-29 01:02:29] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:29.538438499  [2023-10-29 01:02:29] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:29.538539585  [2023-10-29 01:02:29] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:29.538663421  [2023-10-29 01:02:29] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:33.410360208  [2023-10-29 01:02:33] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:33.411022015  [2023-10-29 01:02:33] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:39.410358176  [2023-10-29 01:02:39] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:39.410462567  [2023-10-29 01:02:39] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:39.410554800  [2023-10-29 01:02:39] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:39.423699604  [2023-10-29 01:02:39] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:39.423806928  [2023-10-29 01:02:39] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:39.423906770  [2023-10-29 01:02:39] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:39.439997427  [2023-10-29 01:02:39] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:39.440092856  [2023-10-29 01:02:39] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:39.440179266  [2023-10-29 01:02:39] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:39.441843545  [2023-10-29 01:02:39] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:39.441954927  [2023-10-29 01:02:39] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:39.442026855  [2023-10-29 01:02:39] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:39.504463166  [2023-10-29 01:02:39] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:39.504561503  [2023-10-29 01:02:39] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:39.504699248  [2023-10-29 01:02:39] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:39.581170238  [2023-10-29 01:02:39] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:39.581279255  [2023-10-29 01:02:39] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:39.581366563  [2023-10-29 01:02:39] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:43.412417754  [2023-10-29 01:02:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:43.420060355  [2023-10-29 01:02:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:49.488192544  [2023-10-29 01:02:49] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:49.489071142  [2023-10-29 01:02:49] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:49.489077533  [2023-10-29 01:02:49] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:49.489080553  [2023-10-29 01:02:49] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:49.489082819  [2023-10-29 01:02:49] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:49.489085062  [2023-10-29 01:02:49] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:49.489087770  [2023-10-29 01:02:49] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:49.489089915  [2023-10-29 01:02:49] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:49.489106983  [2023-10-29 01:02:49] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:49.500464614  [2023-10-29 01:02:49] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:49.500472266  [2023-10-29 01:02:49] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:49.500474769  [2023-10-29 01:02:49] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:49.531792994  [2023-10-29 01:02:49] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:49.531800417  [2023-10-29 01:02:49] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:49.531802841  [2023-10-29 01:02:49] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:49.657734021  [2023-10-29 01:02:49] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:49.657994883  [2023-10-29 01:02:49] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:49.658103237  [2023-10-29 01:02:49] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:53.454337509  [2023-10-29 01:02:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:02:59.508649380  [2023-10-29 01:02:59] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:59.508658078  [2023-10-29 01:02:59] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:59.509823109  [2023-10-29 01:02:59] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:59.517144736  [2023-10-29 01:02:59] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:59.517521254  [2023-10-29 01:02:59] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:59.517526701  [2023-10-29 01:02:59] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:59.534093527  [2023-10-29 01:02:59] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:59.534202297  [2023-10-29 01:02:59] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:59.534312811  [2023-10-29 01:02:59] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:59.538338446  [2023-10-29 01:02:59] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:59.538738538  [2023-10-29 01:02:59] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:59.538743283  [2023-10-29 01:02:59] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:59.548788379  [2023-10-29 01:02:59] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:59.548796388  [2023-10-29 01:02:59] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:59.548798791  [2023-10-29 01:02:59] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:02:59.676821697  [2023-10-29 01:02:59] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:02:59.676920054  [2023-10-29 01:02:59] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:02:59.677012804  [2023-10-29 01:02:59] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...

...

2023-10-29 01:59:05.189372876  [2023-10-29 01:59:05] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:05.189464957  [2023-10-29 01:59:05] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:05.189560808  [2023-10-29 01:59:05] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:05.278403674  [2023-10-29 01:59:05] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:05.278540829  [2023-10-29 01:59:05] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:05.278653681  [2023-10-29 01:59:05] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:05.597408572  [2023-10-29 01:59:05] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:05.597867391  [2023-10-29 01:59:05] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:05.597872728  [2023-10-29 01:59:05] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:06.791959620  [2023-10-29 01:59:06] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:06.792066324  [2023-10-29 01:59:06] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:06.792156424  [2023-10-29 01:59:06] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:07.037207931  [2023-10-29 01:59:07] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:07.037314023  [2023-10-29 01:59:07] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:07.037405684  [2023-10-29 01:59:07] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:07.680265968  [2023-10-29 01:59:07] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:07.680364271  [2023-10-29 01:59:07] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:07.680455625  [2023-10-29 01:59:07] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:15.215910977  [2023-10-29 01:59:15] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:15.216132249  [2023-10-29 01:59:15] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:15.216136436  [2023-10-29 01:59:15] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:15.322530090  [2023-10-29 01:59:15] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:15.322715801  [2023-10-29 01:59:15] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:15.322842154  [2023-10-29 01:59:15] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:15.640478327  [2023-10-29 01:59:15] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:15.640568980  [2023-10-29 01:59:15] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:15.640673037  [2023-10-29 01:59:15] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:16.809952572  [2023-10-29 01:59:16] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:16.810046193  [2023-10-29 01:59:16] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:16.810146965  [2023-10-29 01:59:16] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:17.068833904  [2023-10-29 01:59:17] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:17.068950046  [2023-10-29 01:59:17] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:17.069045845  [2023-10-29 01:59:17] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:17.726008840  [2023-10-29 01:59:17] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:17.726115334  [2023-10-29 01:59:17] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:17.726209395  [2023-10-29 01:59:17] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:25.235710159  [2023-10-29 01:59:25] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:25.235804251  [2023-10-29 01:59:25] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:25.235902134  [2023-10-29 01:59:25] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:25.397707784  [2023-10-29 01:59:25] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:25.397888734  [2023-10-29 01:59:25] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:25.397892646  [2023-10-29 01:59:25] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:25.684964492  [2023-10-29 01:59:25] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:25.685400953  [2023-10-29 01:59:25] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:25.685404974  [2023-10-29 01:59:25] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:26.885026890  [2023-10-29 01:59:26] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:26.885034442  [2023-10-29 01:59:26] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:26.885060241  [2023-10-29 01:59:26] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:27.111961854  [2023-10-29 01:59:27] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:27.112160312  [2023-10-29 01:59:27] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:27.112164041  [2023-10-29 01:59:27] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:27.769130148  [2023-10-29 01:59:27] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:27.769314409  [2023-10-29 01:59:27] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:27.769433906  [2023-10-29 01:59:27] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:35.263149006  [2023-10-29 01:59:35] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:35.263908643  [2023-10-29 01:59:35] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:35.263915443  [2023-10-29 01:59:35] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:35.440318969  [2023-10-29 01:59:35] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:35.440447137  [2023-10-29 01:59:35] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:35.440594750  [2023-10-29 01:59:35] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:35.764688366  [2023-10-29 01:59:35] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:35.764885407  [2023-10-29 01:59:35] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:35.764959845  [2023-10-29 01:59:35] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:36.911911883  [2023-10-29 01:59:36] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:36.912047575  [2023-10-29 01:59:36] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:36.913098957  [2023-10-29 01:59:36] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:37.138505548  [2023-10-29 01:59:37] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:37.138625468  [2023-10-29 01:59:37] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:37.138700495  [2023-10-29 01:59:37] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:37.844011276  [2023-10-29 01:59:37] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:37.844103819  [2023-10-29 01:59:37] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:37.844195100  [2023-10-29 01:59:37] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:45.289470573  [2023-10-29 01:59:45] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:45.289576271  [2023-10-29 01:59:45] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:45.289700641  [2023-10-29 01:59:45] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:45.483533186  [2023-10-29 01:59:45] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:45.483647214  [2023-10-29 01:59:45] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:45.483757396  [2023-10-29 01:59:45] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:45.791467259  [2023-10-29 01:59:45] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:45.791637486  [2023-10-29 01:59:45] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:45.791797836  [2023-10-29 01:59:45] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:46.939016401  [2023-10-29 01:59:46] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:46.939223121  [2023-10-29 01:59:46] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:46.939227159  [2023-10-29 01:59:46] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:47.181197395  [2023-10-29 01:59:47] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:47.181305340  [2023-10-29 01:59:47] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:47.181381733  [2023-10-29 01:59:47] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:47.886558475  [2023-10-29 01:59:47] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:47.886681592  [2023-10-29 01:59:47] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:47.886779175  [2023-10-29 01:59:47] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:55.415826230  [2023-10-29 01:59:55] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:55.416491660  [2023-10-29 01:59:55] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:55.416497052  [2023-10-29 01:59:55] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:55.511116426  [2023-10-29 01:59:55] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:55.511243766  [2023-10-29 01:59:55] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:55.511542775  [2023-10-29 01:59:55] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:55.834369720  [2023-10-29 01:59:55] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:55.834461019  [2023-10-29 01:59:55] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:55.834556539  [2023-10-29 01:59:55] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:56.965639421  [2023-10-29 01:59:56] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:56.965760388  [2023-10-29 01:59:56] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:56.965865737  [2023-10-29 01:59:56] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:57.224440032  [2023-10-29 01:59:57] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:57.224566280  [2023-10-29 01:59:57] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:57.224664650  [2023-10-29 01:59:57] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:59:57.932287043  [2023-10-29 01:59:57] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:59:57.932380951  [2023-10-29 01:59:57] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:59:57.932473620  [2023-10-29 01:59:57] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:00:05.434180384  [2023-10-29 02:00:05] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 02:00:05.435039872  [2023-10-29 02:00:05] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 02:00:05.435046650  [2023-10-29 02:00:05] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:00:05.585851821  [2023-10-29 02:00:05] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 02:00:05.586082617  [2023-10-29 02:00:05] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 02:00:05.586086778  [2023-10-29 02:00:05] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:00:05.877500313  [2023-10-29 02:00:05] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 02:00:05.877691697  [2023-10-29 02:00:05] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 02:00:05.877846302  [2023-10-29 02:00:05] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:00:07.042655869  [2023-10-29 02:00:07] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 02:00:07.042975376  [2023-10-29 02:00:07] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 02:00:07.043654476  [2023-10-29 02:00:07] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:00:07.271105453  [2023-10-29 02:00:07] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 02:00:07.274544284  [2023-10-29 02:00:07] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 02:00:07.274678372  [2023-10-29 02:00:07] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:00:07.960927872  [2023-10-29 02:00:07] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 02:00:07.960934935  [2023-10-29 02:00:07] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 02:00:07.960937317  [2023-10-29 02:00:07] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 02:30:26.361845668  [h264 @ 0x7fb9bc0dbd00] Invalid NAL unit size (0 > 887).
2023-10-29 02:30:26.361852781  [h264 @ 0x7fb9bc0dbd00] Error splitting the input into NAL units.
2023-10-29 02:57:30.387918367  [2023-10-29 02:57:30] frigate.http                   WARNING : Recording clip is missing or empty: /media/frigate/recordings/2023-10-29/00/front/59.51.mp4
2023-10-29 02:57:30.388130499  [2023-10-29 02:57:30] frigate.http                   WARNING : Recording clip is missing or empty: /media/frigate/recordings/2023-10-29/00/front/59.57.mp4

FFprobe output from your camera

N/A

Frigate stats

No response

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Annke C500

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

Unfortunately these logs don't provide any info other than that recordings were having some sort of issue being saved. Frigate saves recordings in the recordings folder based in UTC time so it is not affected by daylight savings time.

I'm wondering if this may be related to go2rtc, but that's only a guess based on the fact that recordings stopped being created for some reason

ERROR : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process..

deviant77 commented 1 year ago

Unfortunately these logs don't provide any info other than that recordings were having some sort of issue being saved. Frigate saves recordings in the recordings folder based in UTC time so it is not affected by daylight savings time.

I'm wondering if this may be related to go2rtc, but that's only a guess based on the fact that recordings stopped being created for some reason

ERROR : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process..

I remember the previous issues with DST and the move to UTC, so I was surprised at this occurrence!

About ERROR : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process..... Looking through the whole log, this repeats during the whole hour:

2023-10-29 01:38:43.503402136  [2023-10-29 01:38:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:38:43.504031402  [2023-10-29 01:38:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:38:43.514826237  [2023-10-29 01:38:43] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:38:49.400909130  [2023-10-29 01:38:49] watchdog.side                  ERROR   : No new recording segments were created for side in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:38:49.402645797  [2023-10-29 01:38:49] watchdog.side                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:38:49.403148443  [2023-10-29 01:38:49] watchdog.side                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:38:49.471721306  [2023-10-29 01:38:49] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:38:49.472145826  [2023-10-29 01:38:49] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:38:49.472402558  [2023-10-29 01:38:49] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:38:49.500645196  [2023-10-29 01:38:49] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:38:49.500885589  [2023-10-29 01:38:49] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:38:49.500891142  [2023-10-29 01:38:49] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:38:50.637479380  [2023-10-29 01:38:50] watchdog.crossroad             ERROR   : No new recording segments were created for crossroad in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:38:50.637809587  [2023-10-29 01:38:50] watchdog.crossroad             INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:38:50.637813462  [2023-10-29 01:38:50] watchdog.crossroad             INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:38:50.975201366  [2023-10-29 01:38:50] watchdog.pond                  ERROR   : No new recording segments were created for pond in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:38:50.975209671  [2023-10-29 01:38:50] watchdog.pond                  INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:38:50.975212614  [2023-10-29 01:38:50] watchdog.pond                  INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:38:51.897284185  [2023-10-29 01:38:51] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-10-29 01:38:51.897391135  [2023-10-29 01:38:51] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-10-29 01:38:51.897513595  [2023-10-29 01:38:51] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-10-29 01:38:53.510391890  [2023-10-29 01:38:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:38:53.511396327  [2023-10-29 01:38:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for pond. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-10-29 01:38:53.511686296  [2023-10-29 01:38:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for crossroad. Keeping the 5 most recent segments out of 6 and discarding the rest...

Frigate says there's too many recordings segments in cache for a particular camera (which would indicate recording segments are being put in to cache?), and then 6 seconds later says that no new recording segments were created for that camera in the last 120 seconds, and then 4 seconds later says there's too many segments again!?

deviant77 commented 1 year ago

I forgot to add that the same thing occurred on another installation with a very similar configuration (including go2rtc) at the same time.

NickM-27 commented 1 year ago

It seems like it's some sort of error in the logic that's checking to make sure recordings are being written correctly. However, they all use .timestamp() which shouldn't care about daylight savings so that doesn't quite add up

MikePaer commented 7 months ago

It's daylight savings time here this morning and I just experienced this same issue. Had to restart Frigate to get recordings started again. Version 0.13.2-6476F8A.

NickM-27 commented 7 months ago

weird, no issues here

blakeblackshear / frigate