[Config Support]: Unable to keep up with recording segments in cache for (Camera)

Describe the problem you are having

It seems like either the API gets hosed by a custom card in Home Assistant and/or Frigate is struggling with my storage. The Frigate instance lives in a Debian 12 VM under Proxmox. My local drive for this is a Samsung 870 EVO.

This recently started happening around the time I used ~1TB of my recording storage (TrueNAS NFS share off a pair of mirrored Skyhawks).

Version

0.12.1-367d724

Frigate config file

mqtt:
  enabled: True
  host: 192.168.x.x
  port: 1883
  topic_prefix: frigate
  user: user
  password: password

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac

objects:
  track:
    - person
    - dog
    - cat
   # - car
    - bus
    - motorcycle
    - bicycle

rtmp:
  enabled: false

detect:
  fps: 7
  enabled: true

record:
  enabled: True
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 28
      mode: active_objects

snapshots:
  enabled: true
  clean_copy: true
  retain:
    default:  14

detectors:
  coral:
    type: edgetpu
    device: pci

#Camera Configuration 
go2rtc:
  streams:

    back_porch: 
      - http://192.168.X.X/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=X&password=X
      - "ffmpeg:garage#audio=opus"
    back_porch_sub: 
      - http://192.168.X.X/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=X&password=X

[garden,front,garage all have same config]    

cameras:

  back_porch:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/back_porch_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect

[garden,front,garage all have same config]   

logger:
  logs:
    frigate.record: debug

Relevant log output

2023-07-17 19:32:56.888268108  [INFO] Starting Frigate...
2023-07-17 19:32:58.360263364  [2023-07-17 19:32:58] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-17 19:32:58.401973583  [2023-07-17 19:32:58] peewee_migrate                 INFO    : Starting migrations
2023-07-17 19:32:58.923417688  [2023-07-17 19:32:58] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-17 19:32:58.949201939  [2023-07-17 19:32:58] detector.coral                 INFO    : Starting detection process: 289
2023-07-17 19:32:58.954664080  [2023-07-17 19:32:58] frigate.app                    INFO    : Output process started: 291
2023-07-17 19:32:58.957027987  [2023-07-17 19:32:58] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2023-07-17 19:32:58.964160853  [2023-07-17 19:32:58] frigate.app                    INFO    : Camera processor started for garage: 297
2023-07-17 19:32:58.970756812  [2023-07-17 19:32:58] frigate.app                    INFO    : Camera processor started for back_porch: 298
2023-07-17 19:32:58.977450370  [2023-07-17 19:32:58] frigate.app                    INFO    : Camera processor started for garden: 299
2023-07-17 19:32:58.984029476  [2023-07-17 19:32:58] frigate.app                    INFO    : Camera processor started for front: 300
2023-07-17 19:32:58.990319704  [2023-07-17 19:32:58] frigate.app                    INFO    : Capture process started for garage: 301
2023-07-17 19:32:58.995477063  [2023-07-17 19:32:58] frigate.app                    INFO    : Capture process started for back_porch: 304
2023-07-17 19:32:59.002014577  [2023-07-17 19:32:59] frigate.app                    INFO    : Capture process started for garden: 308
2023-07-17 19:32:59.008904539  [2023-07-17 19:32:59] frigate.app                    INFO    : Capture process started for front: 312
2023-07-17 19:32:59.078715703  [2023-07-17 19:32:59] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-07-17 19:33:31.390434358  [2023-07-17 19:33:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/33.11.mp4 in 6.998477220535278 seconds.
2023-07-17 19:33:40.207717380  [2023-07-17 19:33:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/33.10.mp4 in 8.08652114868164 seconds.
2023-07-17 19:33:45.817393323  [2023-07-17 19:33:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/33.22.mp4 in 5.140644073486328 seconds.
2023-07-17 19:33:51.458002150  [2023-07-17 19:33:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/33.22.mp4 in 5.309648036956787 seconds.
2023-07-17 19:33:59.223776741  [2023-07-17 19:33:59] frigate.record                 DEBUG   : Start expire recordings (new).
2023-07-17 19:33:59.223971951  [2023-07-17 19:33:59] frigate.record                 DEBUG   : Start deleted cameras.
2023-07-17 19:34:00.058883279  [2023-07-17 19:34:00] frigate.record                 DEBUG   : Expiring 0 recordings
2023-07-17 19:34:00.069911593  [2023-07-17 19:34:00] frigate.record                 DEBUG   : End deleted cameras.
2023-07-17 19:34:00.070051621  [2023-07-17 19:34:00] frigate.record                 DEBUG   : Start all cameras.
2023-07-17 19:34:00.070206241  [2023-07-17 19:34:00] frigate.record                 DEBUG   : Start camera: garage.
2023-07-17 19:34:00.213581209  [2023-07-17 19:34:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/33.22.mp4 in 8.360814094543457 seconds.
2023-07-17 19:34:09.144599135  [2023-07-17 19:34:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/33.31.mp4 in 8.173317909240723 seconds.
2023-07-17 19:34:20.364726916  [2023-07-17 19:34:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/33.32.mp4 in 10.415712118148804 seconds.
2023-07-17 19:34:30.553836891  [2023-07-17 19:34:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/33.42.mp4 in 9.832204818725586 seconds.
2023-07-17 19:34:41.213239026  [2023-07-17 19:34:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/33.53.mp4 in 10.05471396446228 seconds.
2023-07-17 19:34:41.769672818  [2023-07-17 19:34:41] frigate.record                 DEBUG   : Expiring 18 recordings
2023-07-17 19:34:44.490132804  [2023-07-17 19:34:44] frigate.record                 DEBUG   : End camera: garage.
2023-07-17 19:34:44.490345848  [2023-07-17 19:34:44] frigate.record                 DEBUG   : Start camera: back_porch.
2023-07-17 19:34:50.447924864  [2023-07-17 19:34:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/34.03.mp4 in 8.166709899902344 seconds.
2023-07-17 19:34:59.883874557  [2023-07-17 19:34:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/34.13.mp4 in 8.846074104309082 seconds.
2023-07-17 19:35:07.268690582  [2023-07-17 19:35:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/34.02.mp4 in 6.656141996383667 seconds.
2023-07-17 19:35:17.490499338  [2023-07-17 19:35:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/34.03.mp4 in 9.133104085922241 seconds.
2023-07-17 19:35:18.140065831  [2023-07-17 19:35:18] frigate.record                 DEBUG   : Expiring 38 recordings
2023-07-17 19:35:19.585496901  [2023-07-17 19:35:19] frigate.record                 DEBUG   : End camera: back_porch.
2023-07-17 19:35:19.585671283  [2023-07-17 19:35:19] frigate.record                 DEBUG   : Start camera: garden.
2023-07-17 19:35:39.921692133  [2023-07-17 19:35:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/34.42.mp4 in 17.389102935791016 seconds.
2023-07-17 19:35:48.544993232  [2023-07-17 19:35:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/34.52.mp4 in 8.260787010192871 seconds.
2023-07-17 19:35:58.003970784  [2023-07-17 19:35:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/35.02.mp4 in 9.042999029159546 seconds.
2023-07-17 19:35:59.085187673  [2023-07-17 19:35:59] frigate.record                 DEBUG   : Expiring 75 recordings
2023-07-17 19:36:00.997205199  [2023-07-17 19:36:00] frigate.record                 DEBUG   : End camera: garden.
2023-07-17 19:36:00.997346736  [2023-07-17 19:36:00] frigate.record                 DEBUG   : Start camera: front.
2023-07-17 19:36:01.140124740  [2023-07-17 19:36:01] frigate.record                 DEBUG   : Expiring 0 recordings
2023-07-17 19:36:01.140410995  [2023-07-17 19:36:01] frigate.record                 DEBUG   : End camera: front.
2023-07-17 19:36:01.140554102  [2023-07-17 19:36:01] frigate.record                 DEBUG   : End all cameras.
2023-07-17 19:36:01.140795424  [2023-07-17 19:36:01] frigate.record                 DEBUG   : End expire recordings (new).
2023-07-17 19:36:01.140798094  [2023-07-17 19:36:01] frigate.record                 DEBUG   : Start expire files (legacy).
2023-07-17 19:36:05.304029716  [2023-07-17 19:36:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/35.11.mp4 in 6.230081081390381 seconds.
2023-07-17 19:36:15.808122353  [2023-07-17 19:36:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/35.12.mp4 in 8.293522834777832 seconds.
2023-07-17 19:36:22.170008631  [2023-07-17 19:36:22] frigate.record                 DEBUG   : Oldest recording in the db: 1687284491.415331
2023-07-17 19:36:29.530260325  [2023-07-17 19:36:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/35.22.mp4 in 13.402080059051514 seconds.
2023-07-17 19:36:44.938688155  [2023-07-17 19:36:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/35.32.mp4 in 14.803064823150635 seconds.
2023-07-17 19:36:53.285511550  [2023-07-17 19:36:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/35.42.mp4 in 7.996834993362427 seconds.
2023-07-17 19:36:53.786141468  [2023-07-17 19:36:53] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-17 19:36:53.787705231  [2023-07-17 19:36:53] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-17 19:36:53.789384357  [2023-07-17 19:36:53] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-17 19:36:53.790740395  [2023-07-17 19:36:53] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-17 19:37:03.744546537  [2023-07-17 19:37:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/36.02.mp4 in 7.506227970123291 seconds.
2023-07-17 19:37:11.746054847  [2023-07-17 19:37:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/36.12.mp4 in 7.661715030670166 seconds.
2023-07-17 19:37:19.205411300  [2023-07-17 19:37:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/36.22.mp4 in 7.03543496131897 seconds.
2023-07-17 19:37:27.036058031  [2023-07-17 19:37:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/36.32.mp4 in 7.137976884841919 seconds.
2023-07-17 19:37:34.191525272  [2023-07-17 19:37:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/36.52.mp4 in 5.704174041748047 seconds.
2023-07-17 19:37:40.415161474  [2023-07-17 19:37:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/37.02.mp4 in 5.90873384475708 seconds.
2023-07-17 19:37:48.425061587  [2023-07-17 19:37:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/36.52.mp4 in 7.447642803192139 seconds.
2023-07-17 19:37:56.760824049  [2023-07-17 19:37:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/37.02.mp4 in 7.903470039367676 seconds.
2023-07-17 19:38:08.413102714  [2023-07-17 19:38:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/37.32.mp4 in 9.592751026153564 seconds.
2023-07-17 19:38:19.447118146  [2023-07-17 19:38:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/37.42.mp4 in 10.487174987792969 seconds.
2023-07-17 19:38:27.685807656  [2023-07-17 19:38:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/37.52.mp4 in 7.151366949081421 seconds.
2023-07-17 19:38:35.173476611  [2023-07-17 19:38:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/38.02.mp4 in 6.738548040390015 seconds.
2023-07-17 19:38:53.653416522  [2023-07-17 19:38:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/37.52.mp4 in 9.782251119613647 seconds.
2023-07-17 19:39:04.284612138  [2023-07-17 19:39:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/38.02.mp4 in 10.101324796676636 seconds.
2023-07-17 19:39:11.991702563  [2023-07-17 19:39:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/38.33.mp4 in 6.9688379764556885 seconds.
2023-07-17 19:39:26.715151897  [2023-07-17 19:39:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/38.43.mp4 in 6.367475986480713 seconds.
2023-07-17 19:39:34.880938612  [2023-07-17 19:39:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/38.31.mp4 in 7.232635974884033 seconds.
2023-07-17 19:39:42.757433205  [2023-07-17 19:39:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/38.41.mp4 in 7.072658061981201 seconds.
2023-07-17 19:39:54.816954646  [2023-07-17 19:39:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/38.22.mp4 in 6.094346046447754 seconds.
2023-07-17 19:40:00.510586098  [2023-07-17 19:40:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/38.32.mp4 in 5.167553901672363 seconds.
2023-07-17 19:40:06.675956567  [2023-07-17 19:40:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/38.53.mp4 in 5.68609094619751 seconds.
2023-07-17 19:40:15.990455237  [2023-07-17 19:40:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/38.12.mp4 in 6.765107870101929 seconds.
2023-07-17 19:40:23.063474209  [2023-07-17 19:40:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/38.22.mp4 in 6.677067995071411 seconds.
2023-07-17 19:40:30.141172783  [2023-07-17 19:40:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/38.32.mp4 in 6.590408086776733 seconds.
2023-07-17 19:40:42.923989379  [2023-07-17 19:40:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/38.42.mp4 in 7.077895164489746 seconds.
2023-07-17 19:40:49.703200131  [2023-07-17 19:40:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/38.52.mp4 in 6.1166839599609375 seconds.
2023-07-17 19:40:50.272829185  [2023-07-17 19:40:50] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-17 19:40:50.280021187  [2023-07-17 19:40:50] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-17 19:40:50.287515051  [2023-07-17 19:40:50] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-17 19:40:50.294524457  [2023-07-17 19:40:50] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-17 19:40:55.882123956  [2023-07-17 19:40:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/39.51.mp4 in 4.631936073303223 seconds.
2023-07-17 19:41:00.989330323  [2023-07-17 19:41:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/front/40.01.mp4 in 4.766309022903442 seconds.
2023-07-17 19:41:06.557200260  [2023-07-17 19:41:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garage/39.52.mp4 in 4.662719011306763 seconds.
2023-07-17 19:41:13.397861443  [2023-07-17 19:41:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/39.52.mp4 in 5.975752830505371 seconds.
2023-07-17 19:41:19.843218383  [2023-07-17 19:41:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/40.02.mp4 in 6.080513954162598 seconds.
2023-07-17 19:41:26.329854042  [2023-07-17 19:41:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/40.12.mp4 in 6.0680530071258545 seconds.
2023-07-17 19:41:32.746296340  [2023-07-17 19:41:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/40.22.mp4 in 6.044753074645996 seconds.
2023-07-17 19:41:38.970109600  [2023-07-17 19:41:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/garden/40.32.mp4 in 5.837963819503784 seconds.
2023-07-17 19:41:41.572837912  [2023-07-17 19:41:41] frigate.record                 DEBUG   : End expire files (legacy).
2023-07-17 19:41:44.422811037  [2023-07-17 19:41:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/40.52.mp4 in 4.620270013809204 seconds.
2023-07-17 19:41:49.301053671  [2023-07-17 19:41:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/41.02.mp4 in 4.6027891635894775 seconds.
2023-07-17 19:41:54.050431474  [2023-07-17 19:41:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/00/back_porch/41.12.mp4 in 4.474018096923828 seconds.

Frigate stats

{
    "back_porch": {
        "camera_fps": 7.1,
        "capture_pid": 304,
        "detection_enabled": 1,
        "detection_fps": 0.1,
        "ffmpeg_pid": 309,
        "pid": 298,
        "process_fps": 7.1,
        "skipped_fps": 0.0
    },
    "cpu_usages": {
        "%Cpu(s):": {
            "cpu": "id,",
            "mem": "12.5"
        },
        "1": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "104": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "110": {
            "cpu": "2.0",
            "mem": "3.1"
        },
        "117": {
            "cpu": "0.0",
            "mem": "0.2"
        },
        "134": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "135": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "136": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "137": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "15": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "17": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "2011": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "2016": {
            "cpu": "1.0",
            "mem": "0.4"
        },
        "2019": {
            "cpu": "0.3",
            "mem": "0.0"
        },
        "2021": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "2022": {
            "cpu": "3.3",
            "mem": "0.0"
        },
        "24": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "25": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "26": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "27": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "28": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "282": {
            "cpu": "0.0",
            "mem": "1.0"
        },
        "288": {
            "cpu": "0.3",
            "mem": "0.1"
        },
        "289": {
            "cpu": "1.3",
            "mem": "1.4"
        },
        "29": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "291": {
            "cpu": "1.7",
            "mem": "1.1"
        },
        "297": {
            "cpu": "1.0",
            "mem": "1.2"
        },
        "298": {
            "cpu": "1.0",
            "mem": "1.2"
        },
        "299": {
            "cpu": "4.0",
            "mem": "1.2"
        },
        "30": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "300": {
            "cpu": "0.7",
            "mem": "1.2"
        },
        "301": {
            "cpu": "2.0",
            "mem": "1.1"
        },
        "304": {
            "cpu": "2.0",
            "mem": "1.1"
        },
        "307": {
            "cpu": "4.0",
            "mem": "0.8"
        },
        "308": {
            "cpu": "1.7",
            "mem": "1.1"
        },
        "309": {
            "cpu": "3.7",
            "mem": "0.8"
        },
        "31": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "312": {
            "cpu": "2.0",
            "mem": "1.1"
        },
        "314": {
            "cpu": "3.7",
            "mem": "0.8"
        },
        "317": {
            "cpu": "3.3",
            "mem": "0.8"
        },
        "32": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "320": {
            "cpu": "4.3",
            "mem": "0.4"
        },
        "325": {
            "cpu": "0.0",
            "mem": "0.2"
        },
        "338": {
            "cpu": "1.3",
            "mem": "0.5"
        },
        "339": {
            "cpu": "4.3",
            "mem": "0.4"
        },
        "340": {
            "cpu": "4.7",
            "mem": "0.4"
        },
        "341": {
            "cpu": "0.0",
            "mem": "0.3"
        },
        "342": {
            "cpu": "0.0",
            "mem": "0.2"
        },
        "343": {
            "cpu": "0.0",
            "mem": "0.3"
        },
        "344": {
            "cpu": "0.0",
            "mem": "0.2"
        },
        "41": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "42": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "79": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "81": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "82": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "89": {
            "cpu": "22.3",
            "mem": "0.3"
        },
        "MiB": {
            "cpu": "5542.2",
            "mem": "avail"
        },
        "PID": {
            "cpu": "%CPU",
            "mem": "%MEM"
        },
        "Tasks:": {
            "cpu": "stopped,",
            "mem": "0"
        },
        "top": {
            "cpu": "average:",
            "mem": "2.05,"
        }
    },
    "detection_fps": 0.8,
    "detectors": {
        "coral": {
            "detection_start": 0.0,
            "inference_speed": 7.06,
            "pid": 289
        }
    },
    "front": {
        "camera_fps": 7.0,
        "capture_pid": 312,
        "detection_enabled": 1,
        "detection_fps": 0.0,
        "ffmpeg_pid": 317,
        "pid": 300,
        "process_fps": 7.0,
        "skipped_fps": 0.0
    },
    "garage": {
        "camera_fps": 7.0,
        "capture_pid": 301,
        "detection_enabled": 1,
        "detection_fps": 0.0,
        "ffmpeg_pid": 307,
        "pid": 297,
        "process_fps": 7.0,
        "skipped_fps": 0.0
    },
    "garden": {
        "camera_fps": 7.1,
        "capture_pid": 308,
        "detection_enabled": 1,
        "detection_fps": 0.7,
        "ffmpeg_pid": 314,
        "pid": 299,
        "process_fps": 7.1,
        "skipped_fps": 0.0
    },
    "gpu_usages": {
        "intel-vaapi": {
            "gpu": "3.62 %",
            "mem": "- %"
        }
    },
    "service": {
        "last_updated": 1689641563,
        "latest_version": "0.12.1",
        "storage": {
            "/dev/shm": {
                "free": 395.1,
                "mount_type": "tmpfs",
                "total": 402.7,
                "used": 7.6
            },
            "/media/frigate/clips": {
                "free": 2786668.7,
                "mount_type": "nfs",
                "total": 3861620.1,
                "used": 1074951.4
            },
            "/media/frigate/recordings": {
                "free": 2786668.7,
                "mount_type": "nfs",
                "total": 3861620.1,
                "used": 1074951.4
            },
            "/tmp/cache": {
                "free": 1795.9,
                "mount_type": "tmpfs",
                "total": 2000.0,
                "used": 204.1
            }
        },
        "temperatures": {
            "apex_0": 52.3
        },
        "uptime": 1184,
        "version": "0.12.1-367d724"
    }
}

Operating system

Proxmox

Install method

Docker Compose

Coral version

PCIe

Any other information that may be helpful

CPU(s)

4 x Intel(R) Core(TM) i5-7260U CPU @ 2.20GHz (1 Socket) (Intel NUC)

32GB RAM

870 EVO for storage

Additionally, RAM seems to always very quickly just almost max out, then stay there. Should I be giving it more memory than 8GB?

I don't think it's going to be due to the custom card, it is heavily used including by myself and I've not seen this issue for myself or often.

One thing that is unclear though is where your database is stored. If it is on the NAS that could be the source of the problem.

For the RAM usage you'll want to check what process is actually using the RAM to see why. 8GB seems excessive.

I don't think it's going to be due to the custom card, it is heavily used including by myself and I've not seen this issue for myself or often.

I didn't think it was the card's fault either, but I wanted to mention it as it does seem like it throws quite a few GET requests when trying to load my cameras.

One thing that is unclear though is where your database is stored. If it is on the NAS that could be the source of the problem.

`version: "3.9" services: frigate: container_name: frigate privileged: true # this may not be necessary for all setups restart: unless-stopped image: ghcr.io/blakeblackshear/frigate:stable

shm_size: "384mb" # update for your cameras based on calculation above
devices:
  - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
  - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
  - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
  - /dev/apex_1:/dev/apex_1 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /mnt/NFSshareForProxmox/proxmox/docker/docker_data/frigate/config.yml:/config/config.yml:ro
  - /mnt/cameras/Cameras/frigate:/media/frigate
  - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
    target: /tmp/cache
    tmpfs:
      size: 2000000000
ports:
  - "5000:5000"
  - "1935:1935" # RTMP feeds
  - "8554:8554" # RTSP feeds
  - "8555:8555/tcp" # WebRTC over tcp
  - "8555:8555/udp" # WebRTC over udp 

environment:
  FRIGATE_RTSP_PASSWORD: "password"

D'oh. Looks like it is going to my NAS, which is unfortunately a pile of spins. That makes a bit more sense.

While not a hard requirement for my use-case, but rather a prefer, I am planning on (hopefully soon) replacing that array with SSDs, along with a 10GB network upgrade down the line. Would I see this issue go away with either of these improvements?

For the RAM usage you'll want to check what process is actually using the RAM to see why. 8GB seems excessive.

Regarding RAM, here is the output of top:

In 0.13 you'll need to move your db off the media directory anyway. It is highly recommended to have it locally and not on a network share.

In 0.13 you'll need to move your db off the media directory anyway. It is highly recommended to have it locally and not on a network share.

Ugh, so that means I have to be a normal person and just make backups instead of being lazy storing my data on a network share and backing that up? /sarcasm

Understood though, I'll go ahead and make that change. If I understand correct, the change in 0.13 will not affect the NFS share I've created for recordings, just the db?

Yes, with the new HA storage feature we are having users move the db to a new location other than media which is usually done on a network share or spinning storage.

While everything responds MUCH faster now, I'm still experiencing this issue. The only thing I've changed is the path that was originally pointed towards my NAS is now in /home/appdata_folder/that_i_created

Was I mistaken where the DB lives and that I didn't move it correctly? Please let me know what other information I can provide.

so now you are not using the NAS for recordings? You should either update the docker compose to bind mount the entire /config and update the frigate config to save the db there (/config/frigate.db) or create a new /db/ directory on local storage

so now you are not using the NAS for recordings? You should either update the docker compose to bind mount the entire /config and update the frigate config to save the db there (/config/frigate.db) or create a new /db/ directory on local storage

Sorry, my comment was a bit confusing. My recordings still go to my NAS, but that is now the only thing that lives on there. My /config path is on my local storage (SSD). The bind mount in my docker compose is /home/appdata_folder/that_i_created:/config. I'll copy the exact line out of my docker compose when I'm back at my computer.

What I need to see if your frigate config itself

I will copy it here as soon as I can 👍

What I need to see if your frigate config itself

version: "3.9" services: frigate: container_name: frigate privileged: true # this may not be necessary for all setups restart: unless-stopped image: ghcr.io/blakeblackshear/frigate:stable

shm_size: "384mb" # update for your cameras based on calculation above
devices:
  - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
  - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
  - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
  - /dev/apex_1:/dev/apex_1 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /home/907/appdata/frigate/config.yml:/config/config.yml:ro
  - /mnt/NFSshare/frigate:/media/frigate
  - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
    target: /tmp/cache
    tmpfs:
      size: 2000000000
ports:
  - "5000:5000"
  - "1935:1935" # RTMP feeds
  - "8554:8554" # RTSP feeds
  - "8555:8555/tcp" # WebRTC over tcp
  - "8555:8555/udp" # WebRTC over udp 

environment:
  FRIGATE_RTSP_PASSWORD: "password"

That is the docker compose, I'm looking to see the frigate config

My apologies!!

mqtt:
  enabled: True
  host: 192.168.x.x
  port: 1883
  topic_prefix: frigate
  user: frigatemqtt
  password: frigatemqttpw

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac

objects:
  track:
    - person
    - dog
    - cat
   # - car
    - bus
    - motorcycle
    - bicycle

rtmp:
  enabled: false

detect:
  fps: 7
  enabled: true

record:
  enabled: True
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 28
      mode: active_objects

snapshots:
  enabled: true
  clean_copy: true
  retain:
    default:  14

detectors:
  coral:
    type: edgetpu
    device: pci

#Camera Configuration 
go2rtc:
  streams:
    garage: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:garage#audio=opus"
    garage_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw

    back_porch: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:back_porch#audio=opus"
    back_porch_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw

    garden: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:garden#audio=opus"
    garden_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw

    front: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:front#audio=opus"
    front_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw 

cameras:
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garage_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  back_porch:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/back_porch_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  garden:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garden?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garden_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect

logger:
  logs:
    frigate.record: debug

You haven't changed the location of the database, it is still running on your NAS

I guess I'm confused, how would I find and move it's location? I apologize

https://docs.frigate.video/configuration/advanced#database

The default location is /media/frigate/frigate.db

Unfortunately, even with that change, it results in the same issues.

New config:

database:
  path: /home/appdata/frigate/db/frigate.db

mqtt:
  enabled: True
  host: 192.168.x.x
  port: 1883
  topic_prefix: frigate
  user: frigatemqtt
  password: frigatemqttpw

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac

objects:
  track:
    - person
    - dog
    - cat
   # - car
    - bus
    - motorcycle
    - bicycle

rtmp:
  enabled: false

detect:
  fps: 7
  enabled: true

record:
  enabled: True
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 28
      mode: active_objects

snapshots:
  enabled: true
  clean_copy: true
  retain:
    default:  14

detectors:
  coral:
    type: edgetpu
    device: pci

#Camera Configuration 
go2rtc:
  streams:
    garage: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:garage#audio=opus"
    garage_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw

    back_porch: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:back_porch#audio=opus"
    back_porch_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw

    garden: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:garden#audio=opus"
    garden_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw

    front: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pw
      - "ffmpeg:front#audio=opus"
    front_sub: 
      - http://192.168.x.x/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pw 

cameras:
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garage_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  back_porch:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/back_porch_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  garden:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garden?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garden_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect

logger:
  logs:
    frigate.record: debug

Logs:

2023-07-18 18:42:07.917666884  2023/07/18 18:42:07 [warn] 134#134: *48 an upstream response is buffered to a temporary file /usr/local/nginx/proxy_temp/6/00/0000000006 while reading upstream, client: 192.168.1.222, server: , request: "GET /api/garage/latest.jpg HTTP/1.1", upstream: "http://127.0.0.1:5001garage/latest.jpg", host: "192.168.69.69:5000"

2023-07-18 18:42:07.966674829  192.168.1.222 - - [18/Jul/2023:18:42:07 -0500] "GET /api/garage/latest.jpg HTTP/1.1" 200 156380 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:08.200113322  192.168.1.222 - - [18/Jul/2023:18:42:08 -0500] "GET /api/garage/latest.jpg HTTP/1.1" 200 156327 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:10.732065648  192.168.1.222 - - [18/Jul/2023:18:42:10 -0500] "GET /api/stats HTTP/1.1" 200 3172 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:12.130433441  192.168.1.222 - - [18/Jul/2023:18:42:12 -0500] "GET /api/events?cameras=back_porch&limit=50&include_thumbnails=0 HTTP/1.1" 499 0 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:15.536076812  192.168.1.222 - - [18/Jul/2023:18:42:15 -0500] "GET /api/events?cameras=garage&limit=50&include_thumbnails=0 HTTP/1.1" 499 0 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:19.995056948  192.168.1.222 - - [18/Jul/2023:18:42:19 -0500] "GET /api/stats HTTP/1.1" 200 3136 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:25.341477384  [2023-07-18 18:42:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/23/garage/40.26.mp4 in 21.341169118881226 seconds.

2023-07-18 18:42:31.050305759  192.168.1.222 - - [18/Jul/2023:18:42:31 -0500] "GET /api/stats HTTP/1.1" 200 3201 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:40.504784145  192.168.1.222 - - [18/Jul/2023:18:42:40 -0500] "GET /api/stats HTTP/1.1" 200 3138 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:45.984853051  [2023-07-18 18:42:45] frigate.record                 DEBUG   : Expiring 7 recordings

2023-07-18 18:42:47.924542548  [2023-07-18 18:42:47] frigate.record                 DEBUG   : End camera: garage.

2023-07-18 18:42:47.924670948  [2023-07-18 18:42:47] frigate.record                 DEBUG   : Start camera: back_porch.

2023-07-18 18:42:50.298723453  [2023-07-18 18:42:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/23/garden/40.16.mp4 in 22.5413658618927 seconds.

2023-07-18 18:42:50.978160843  192.168.1.222 - - [18/Jul/2023:18:42:50 -0500] "GET /api/stats HTTP/1.1" 200 3138 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:42:59.811024340  192.168.1.222 - - [18/Jul/2023:18:42:59 -0500] "GET /api/stats HTTP/1.1" 200 3139 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:43:08.854230993  192.168.1.222 - - [18/Jul/2023:18:43:08 -0500] "GET /api/stats HTTP/1.1" 200 3138 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:43:10.887423695  [2023-07-18 18:43:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/23/garden/40.26.mp4 in 19.656504154205322 seconds.

2023-07-18 18:43:17.464135851  192.168.1.222 - - [18/Jul/2023:18:43:17 -0500] "GET /api/stats HTTP/1.1" 200 3171 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:43:26.468096799  192.168.1.222 - - [18/Jul/2023:18:43:26 -0500] "GET /api/stats HTTP/1.1" 200 3138 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:43:28.693701876  [2023-07-18 18:43:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-18/23/garden/40.36.mp4 in 16.663961172103882 seconds.

2023-07-18 18:43:30.486709089  [2023-07-18 18:43:30] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 16 and discarding the rest...

2023-07-18 18:43:30.500268992  [2023-07-18 18:43:30] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 16 and discarding the rest...

2023-07-18 18:43:30.516430590  [2023-07-18 18:43:30] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 16 and discarding the rest...

2023-07-18 18:43:30.534189795  [2023-07-18 18:43:30] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 16 and discarding the rest...

2023-07-18 18:43:35.362402216  192.168.1.222 - - [18/Jul/2023:18:43:35 -0500] "GET /api/stats HTTP/1.1" 200 3153 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:43:39.623068721  [2023-07-18 18:43:39] frigate.record                 DEBUG   : Expiring 29 recordings

2023-07-18 18:43:40.939069663  [2023-07-18 18:43:40] frigate.record                 DEBUG   : End camera: back_porch.

2023-07-18 18:43:40.939074556  [2023-07-18 18:43:40] frigate.record                 DEBUG   : Start camera: garden.

2023-07-18 18:43:44.697698648  192.168.1.222 - - [18/Jul/2023:18:43:44 -0500] "GET /api/stats HTTP/1.1" 200 3139 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

2023-07-18 18:43:53.399417991  192.168.1.222 - - [18/Jul/2023:18:43:53 -0500] "GET /api/stats HTTP/1.1" 200 3138 "-" "HomeAssistant/2023.7.0 aiohttp/3.8.4 Python/3.11" "-"

Your DB path is using the path on your host but that is wrong, this configuration will wipe your DB when you update the docker container. You need to use a path that is inside the container and mounted inside the docker compose file.

as far as the actual issue goes, 19 seconds is an incredible amount of time to transfer a 10 second segment and points to issues somewhere in the setup on the NAS or potentially the connection to the NAS.

Your DB path is using the path on your host but that is wrong, this configuration will wipe your DB when you update the docker container. You need to use a path that is inside the container and mounted inside the docker compose file.

Fixing now. I appreciate your patience with me on this.

as far as the actual issue goes, 19 seconds is an incredible amount of time to transfer a 10 second segment and points to issues somewhere in the setup on the NAS or potentially the connection to the NAS.

I realize it's a bit out of the realm of support for this, but I'm not sure where I would start troubleshooting with this. Neither the physical cables, nor my switch seem to be the culprit (GS752TP).

Here is the new compose, the path inside the config file is /db/frigate.db to match. Is this correct or am I not understanding.

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable

    shm_size: "384mb" # update for your cameras based on calculation above
    devices:
      - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/apex_1:/dev/apex_1 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home//appdata/frigate/config.yml:/config/config.yml:ro

      - /home/appdata/frigate/db:/db

      - /mnt/Cameras/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 2000000000

Yes that looks good. As far as the recordings issue, what type of connection are you using? (SMB, NFS, etc?)

Yes that looks good. As far as the recordings issue, what type of connection are you using? (SMB, NFS, etc?)

NFS. Using truenas as the OS, and the hardware is admittedly extreme overkill for just running truenas.

So you could either remove the NAS for now and do recordings to a local ssd to see what the timings are looking like or look online for other troubleshooting steps related to truenas and slow writes

I would try the SSD if I had one big enough, my current one would fill very very quickly. I'll dig into the slow writes to TrueNAS and see what I come up with.

Just updating this - I am still experiencing this, but I seem to be slowly making headway. Tweaking my NFS share has resulted in faster speed but far from optimal for the hardware, only hitting 50MByte transfer speed testing with iperf.

Current logs show much better copy speeds than before:


2023-07-19 13:45:11.074554142  [2023-07-19 13:45:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/44.06.mp4 in 5.218786954879761 seconds.
2023-07-19 13:45:16.903136530  [2023-07-19 13:45:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/44.15.mp4 in 5.530848979949951 seconds.
2023-07-19 13:45:22.392843681  [2023-07-19 13:45:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/44.25.mp4 in 5.237979888916016 seconds.
2023-07-19 13:45:28.303432878  [2023-07-19 13:45:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/44.35.mp4 in 5.5755650997161865 seconds.
2023-07-19 13:45:33.986721797  [2023-07-19 13:45:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/44.45.mp4 in 5.42293906211853 seconds.
2023-07-19 13:45:41.259378836  [2023-07-19 13:45:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/front/44.12.mp4 in 6.22213888168335 seconds.
2023-07-19 13:45:47.444455639  [2023-07-19 13:45:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/front/44.42.mp4 in 5.65068793296814 seconds.
2023-07-19 13:45:54.181605343  [2023-07-19 13:45:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/44.34.mp4 in 5.930906057357788 seconds.
2023-07-19 13:46:00.970279759  [2023-07-19 13:46:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/44.44.mp4 in 6.476713180541992 seconds.
2023-07-19 13:46:07.244665104  [2023-07-19 13:46:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/44.53.mp4 in 5.990568161010742 seconds.
2023-07-19 13:46:16.152216619  [2023-07-19 13:46:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garden/44.11.mp4 in 8.237550973892212 seconds.
2023-07-19 13:46:25.479634312  [2023-07-19 13:46:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garden/44.21.mp4 in 8.67781686782837 seconds.
2023-07-19 13:46:33.220409844  [2023-07-19 13:46:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garden/44.31.mp4 in 7.334014177322388 seconds.
2023-07-19 13:46:47.040376530  [2023-07-19 13:46:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garden/44.41.mp4 in 7.011584997177124 seconds.
2023-07-19 13:46:54.567062412  [2023-07-19 13:46:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garden/44.51.mp4 in 7.212277889251709 seconds.
2023-07-19 13:46:55.092776185  [2023-07-19 13:46:55] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-07-19 13:46:55.100904466  [2023-07-19 13:46:55] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-07-19 13:46:55.108483781  [2023-07-19 13:46:55] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-07-19 13:46:55.117114070  [2023-07-19 13:46:55] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-07-19 13:47:00.032769569  [2023-07-19 13:47:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/45.56.mp4 in 4.762727975845337 seconds.
2023-07-19 13:47:05.415998134  [2023-07-19 13:47:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/46.05.mp4 in 5.084378004074097 seconds.
2023-07-19 13:47:11.210142712  [2023-07-19 13:47:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/46.15.mp4 in 4.826885938644409 seconds.
2023-07-19 13:47:16.147183511  [2023-07-19 13:47:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/46.25.mp4 in 4.629177808761597 seconds.
2023-07-19 13:47:21.102951318  [2023-07-19 13:47:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/back_porch/46.35.mp4 in 4.655766010284424 seconds.
2023-07-19 13:47:27.073968325  [2023-07-19 13:47:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/front/46.42.mp4 in 5.1911301612854 seconds.
2023-07-19 13:47:32.372395769  [2023-07-19 13:47:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/46.03.mp4 in 4.985778093338013 seconds.
2023-07-19 13:47:37.570951106  [2023-07-19 13:47:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/46.13.mp4 in 4.835997104644775 seconds.
2023-07-19 13:47:42.791365738  [2023-07-19 13:47:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/46.23.mp4 in 4.812151193618774 seconds.
2023-07-19 13:47:47.880578963  [2023-07-19 13:47:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-19/18/garage/46.33.mp4 in 4.789103984832764 seconds.

Due to this, my issues with API are still likely linked to it:

I apologize for the gravedigging of my own issue, I did manage to track down the awful iperf results to a bad keystone, one of the wire "slots" seemed to be just loose enough to allow it to slip out, limiting me to 100Mbp/s. Terminated it into a new one and speeds have been MUCH better and stable. However, I still seem to be running into the same issue, it just seems to be happening less. I am still missing detections when this happens. Here are my current logs:

2023-08-10 17:29:40.829453940  [2023-08-10 17:29:40] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-10 17:29:51.854673165  [2023-08-10 17:29:51] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-08-10 17:29:51.854800406  [2023-08-10 17:29:51] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-08-10 17:29:51.854909408  [2023-08-10 17:29:51] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-10 17:30:53.813956367  [2023-08-10 17:30:53] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:31:46.577995445  [2023-08-10 17:31:46] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:31:46.579012184  [2023-08-10 17:31:46] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:31:46.580698747  [2023-08-10 17:31:46] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:31:46.582973422  [2023-08-10 17:31:46] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:32:44.955049686  [2023-08-10 17:32:44] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:32:44.956233155  [2023-08-10 17:32:44] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:32:44.957343817  [2023-08-10 17:32:44] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-10 17:50:11.361389670  [2023-08-10 17:50:11] watchdog.back_porch            INFO    : No frames received from back_porch in 20 seconds. Exiting ffmpeg...
2023-08-10 17:50:11.361394342  [2023-08-10 17:50:11] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-10 17:50:11.402352552  [2023-08-10 17:50:11] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-08-10 17:50:11.402727100  [2023-08-10 17:50:11] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-08-10 17:50:21.405905633  [2023-08-10 17:50:21] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-08-10 17:50:21.406095348  [2023-08-10 17:50:21] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-10 17:51:41.420052851  [2023-08-10 17:51:41] watchdog.back_porch            ERROR   : No new recording segments were created for back_porch in the last 120s. restarting the ffmpeg record process...
2023-08-10 17:51:41.420288880  [2023-08-10 17:51:41] watchdog.back_porch            INFO    : Terminating the existing ffmpeg process...
2023-08-10 17:51:41.420346705  [2023-08-10 17:51:41] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...

I do not believe it to be my NAS, but I could be wrong, my TrueNAS shows 73GB of 96GB of RAM used, using ~1-2% CPU. Just not sure where to look trying to chase down this issue.

Need to see the copy times like before

Whoops, sorry! Forgot I commented out the debug line in my config. Let me change that.

From: Nicolas Mowen @.> Sent: Thursday, August 10, 2023 6:18:14 PM To: blakeblackshear/frigate @.> Cc: Thomas Young @.>; Author @.> Subject: Re: [blakeblackshear/frigate] [Config Support]: Unable to keep up with recording segments in cache for (Camera) (Issue #7196)

Need to see the copy times like before

— Reply to this email directly, view it on GitHubhttps://github.com/blakeblackshear/frigate/issues/7196#issuecomment-1674042290, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZSP7AQCX3RKRDZ2IIFS6VDXUVT3NANCNFSM6AAAAAA2NVFM5E. You are receiving this because you authored the thread.Message ID: @.***>

As of now, here are my logs:

2023-08-10 18:43:17.081809196  [2023-08-10 18:43:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/42.12.mp4 in 5.0465850830078125 seconds.
2023-08-10 18:43:22.447016317  [2023-08-10 18:43:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/42.32.mp4 in 5.065753936767578 seconds.
2023-08-10 18:43:27.744426522  [2023-08-10 18:43:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/42.42.mp4 in 5.16062593460083 seconds.
2023-08-10 18:43:33.141956843  [2023-08-10 18:43:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/front/42.21.mp4 in 5.127915859222412 seconds.
2023-08-10 18:43:38.619619816  [2023-08-10 18:43:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/front/42.51.mp4 in 4.977240085601807 seconds.
2023-08-10 18:43:43.737074959  [2023-08-10 18:43:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/42.12.mp4 in 4.825901031494141 seconds.
2023-08-10 18:43:48.945892923  [2023-08-10 18:43:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/42.22.mp4 in 5.084611892700195 seconds.
2023-08-10 18:43:54.659373634  [2023-08-10 18:43:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/42.52.mp4 in 5.2997519969940186 seconds.
2023-08-10 18:44:01.470889222  [2023-08-10 18:44:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/42.21.mp4 in 6.649992942810059 seconds.
2023-08-10 18:44:08.314222330  [2023-08-10 18:44:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/42.31.mp4 in 6.657732009887695 seconds.
2023-08-10 18:44:15.318464841  [2023-08-10 18:44:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/42.41.mp4 in 6.812041997909546 seconds.
2023-08-10 18:44:22.284445758  [2023-08-10 18:44:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/42.51.mp4 in 6.791501998901367 seconds.
2023-08-10 18:44:29.038574109  [2023-08-10 18:44:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/43.01.mp4 in 6.578855991363525 seconds.
2023-08-10 18:44:29.246360269  [2023-08-10 18:44:29] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-08-10 18:44:29.250081539  [2023-08-10 18:44:29] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 7 and discarding the rest...
2023-08-10 18:44:29.252784499  [2023-08-10 18:44:29] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-08-10 18:44:29.257421874  [2023-08-10 18:44:29] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 7 and discarding the rest...
2023-08-10 18:44:38.480834988  [2023-08-10 18:44:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/43.31.mp4 in 6.901102066040039 seconds.
2023-08-10 18:44:45.202935171  [2023-08-10 18:44:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/43.41.mp4 in 6.5728600025177 seconds.
2023-08-10 18:44:51.917430354  [2023-08-10 18:44:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/43.51.mp4 in 6.53606104850769 seconds.
2023-08-10 18:44:59.734218873  [2023-08-10 18:44:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/44.01.mp4 in 7.621011972427368 seconds.
2023-08-10 18:45:05.260823913  [2023-08-10 18:45:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/44.32.mp4 in 5.037322998046875 seconds.
2023-08-10 18:45:10.198138745  [2023-08-10 18:45:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/44.42.mp4 in 4.769363880157471 seconds.
2023-08-10 18:45:16.074839839  [2023-08-10 18:45:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/44.42.mp4 in 4.982651948928833 seconds.
2023-08-10 18:45:21.856112579  [2023-08-10 18:45:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/44.52.mp4 in 5.0826051235198975 seconds.
2023-08-10 18:45:27.093080314  [2023-08-10 18:45:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/45.02.mp4 in 5.0885210037231445 seconds.
2023-08-10 18:45:32.568381783  [2023-08-10 18:45:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/44.51.mp4 in 5.052844047546387 seconds.
2023-08-10 18:45:37.905775654  [2023-08-10 18:45:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/45.02.mp4 in 5.177648067474365 seconds.
2023-08-10 18:45:45.010416278  [2023-08-10 18:45:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/44.51.mp4 in 6.897123098373413 seconds.
2023-08-10 18:45:52.104511028  [2023-08-10 18:45:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/45.01.mp4 in 6.937830924987793 seconds.
2023-08-10 18:45:57.426870309  [2023-08-10 18:45:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/45.12.mp4 in 5.112721920013428 seconds.
2023-08-10 18:46:03.008602257  [2023-08-10 18:46:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/45.22.mp4 in 5.449790954589844 seconds.
2023-08-10 18:46:08.496524040  [2023-08-10 18:46:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/45.32.mp4 in 5.317131996154785 seconds.
2023-08-10 18:46:13.602401153  [2023-08-10 18:46:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/45.42.mp4 in 4.966424942016602 seconds.
2023-08-10 18:46:19.576579686  [2023-08-10 18:46:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/45.11.mp4 in 5.208902835845947 seconds.
2023-08-10 18:46:27.177997701  [2023-08-10 18:46:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/45.11.mp4 in 7.0248329639434814 seconds.
2023-08-10 18:46:33.005768749  [2023-08-10 18:46:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/45.51.mp4 in 5.051142930984497 seconds.
2023-08-10 18:46:38.147219276  [2023-08-10 18:46:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/46.02.mp4 in 5.000365972518921 seconds.
2023-08-10 18:46:43.358132625  [2023-08-10 18:46:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/46.12.mp4 in 5.06746506690979 seconds.
2023-08-10 18:46:49.251740520  [2023-08-10 18:46:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garage/46.01.mp4 in 5.179362773895264 seconds.
2023-08-10 18:46:56.216998250  [2023-08-10 18:46:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/45.51.mp4 in 6.618769884109497 seconds.
2023-08-10 18:47:03.296534771  [2023-08-10 18:47:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/garden/46.01.mp4 in 6.9130518436431885 seconds.
2023-08-10 18:47:09.008007720  [2023-08-10 18:47:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/46.21.mp4 in 5.307416915893555 seconds.
2023-08-10 18:47:14.154292299  [2023-08-10 18:47:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/46.31.mp4 in 5.008370876312256 seconds.
2023-08-10 18:47:19.249484789  [2023-08-10 18:47:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/46.41.mp4 in 4.95521092414856 seconds.
2023-08-10 18:47:24.486268010  [2023-08-10 18:47:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/back_porch/46.51.mp4 in 5.097209215164185 seconds.
2023-08-10 18:47:30.003408431  [2023-08-10 18:47:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-10/23/front/46.30.mp4 in 5.20337986946106 seconds.

These times are still too slow, for 4 cameras (which each create a 10 second segment every 10 seconds) it takes ~ 6 seconds.

In other words, it takes 24 seconds to move 10 seconds of recording footage for all cameras which is way too slow. It may not be the NAS itself but the way it's mounted.

Here is my /etc/fstab

I know they're private IPs but it's a habit

My apologies for not updating this sooner, life caught up with me. If this helps whittle down the cause at all (probably not), restarting it causes the issue to go away for a few days (longest has been a week so far), but then it starts rearing it's ugly head again. The rampant RAM issue has never gone away though.

Really unsure on where to go with this - seeing ffmpeg crashes, seems like Frigate is restarting on its own as I'm missing logs for the majority of the day, random write spikes...for the most part, it seems to sit around 1-2.5s for copy

Logs were too long, uploaded to pastebin: https://pastebin.com/8u1bYmjg

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "400mb"
    #cap_add: 
    #  - CAP_PERFMON
    devices:
      - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri/card1 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      #- /dev/apex_1:/dev/apex_1 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
    volumes:
      - /etc/localtime:/etc/localtime:ro
      -/path/to/config.yml:/config/config.yml:ro
      - /path/to/appdata/frigate/db:/db
      - /path/to/Cameras/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

Config as of time of posting:

database:
  path: /db/frigate.db

mqtt:
  enabled: True
  host: XXXX
  port: XXXX
  topic_prefix: frigate
  user: frigatemqtt
  password: XXXX

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac

objects:
  track:
    - person
    - dog
    - cat
   # - car
    - bus
    - motorcycle
    - bicycle

rtmp:
  enabled: false

detect:
  fps: 7
  enabled: true

record:
  enabled: True
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 28
      mode: active_objects

snapshots:
  enabled: true
  clean_copy: true
  retain:
    default:  14

detectors:
  coral:
    type: edgetpu
    device: pci

#Camera Configuration 
go2rtc:
  streams:
    garage: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=XXXX&password=XXXX
      - "ffmpeg:garage#audio=opus"
    garage_sub: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=XXXX&password=XXXX

    back_porch: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=XXXX&password=XXXX
      - "ffmpeg:back_porch#audio=opus"
    back_porch_sub: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=XXXX&password=XXXX

    garden: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=XXXX&password=XXXX
      - "ffmpeg:garden#audio=opus"
    garden_sub: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=XXXX&password=XXXX

    front: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=XXXX&password=XXXX
      - "ffmpeg:front#audio=opus"
    front_sub: 
      - http://192.168.XXX.XXX/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=XXXX&password=XXXX

cameras:
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garage_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  back_porch:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/back_porch_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  garden:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garden?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/garden_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
  front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    zones:  
      front_porch_area:
        coordinates: 0,720,276,720,268,426,0,420        

# Zones

logger:
  logs:
    frigate.record: debug

iperf3 results:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  1019 MBytes   855 Mbits/sec  2079   79.2 KBytes
[  5]  10.00-20.00  sec  1014 MBytes   851 Mbits/sec  2065   69.3 KBytes
[  5]  20.00-30.00  sec  1.00 GBytes   860 Mbits/sec  1827    192 KBytes
[  5]  30.00-40.00  sec  1.00 GBytes   859 Mbits/sec  1466    201 KBytes
[  5]  40.00-50.00  sec  1.01 GBytes   865 Mbits/sec  1632    201 KBytes
[  5]  50.00-60.00  sec  1012 MBytes   849 Mbits/sec  2198    206 KBytes

When I write to my Unraid NAS via NFS a 10 second segment it takes ~ 600 milliseconds. I don't know what directly is normal but even 1.6 seconds seems somewhat slow. The higher values are definitely concerning. This seems like a question for true nas, why a few megabytes files are taking relatively long to write.

When I write to my Unraid NAS via NFS a 10 second segment it takes ~ 600 milliseconds. I don't know what directly is normal but even 1.6 seconds seems somewhat slow. The higher values are definitely concerning. This seems like a question for true nas, why a few megabytes files are taking relatively long to write.

Understood, I'll reach out online and see what people are saying about the slow speeds with TrueNAS.... definitely starting to feel like a needle in a haystack as poking around truenas, everything seems /okay/. Regardless, I sincerely appreciate the help in guiding me along with this. I will update this if tweaking truenas fixes anything. Thanks!

I'd check the default nfs mounting params to see what it is using when mounting, could very well be something small in there. A truenas forum that is more familiar with the defaults and normal behavior would be able to answer though

I'd check the default nfs mounting params to see what it is using when mounting, could very well be something small in there. A truenas forum that is more familiar with the defaults and normal behavior would be able to answer though

Speed issue ended up being due to sync vs async - TrueNAS defaults to sync on datasets, changing the recording dataset to async showed an immediate improvement that can be seen in the logs below. I still do seem to be running into lingering issues though, seems like some streams crash and restart pretty frequently.

2023-09-04 10:15:02.095678225  [INFO] Starting Frigate...
2023-09-04 10:15:03.951857712  [2023-09-04 10:15:03] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-04 10:15:03.993308716  [2023-09-04 10:15:03] peewee_migrate                 INFO    : Starting migrations
2023-09-04 10:15:04.021045035  [2023-09-04 10:15:04] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-04 10:15:04.044993791  [2023-09-04 10:15:04] frigate.app                    INFO    : Output process started: 290
2023-09-04 10:15:04.051361532  [2023-09-04 10:15:04] detector.coral                 INFO    : Starting detection process: 289
2023-09-04 10:15:04.051366054  [2023-09-04 10:15:04] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2023-09-04 10:15:04.055053018  [2023-09-04 10:15:04] frigate.app                    INFO    : Camera processor started for garage: 294
2023-09-04 10:15:04.067439715  [2023-09-04 10:15:04] frigate.app                    INFO    : Camera processor started for back_porch: 298
2023-09-04 10:15:04.072848778  [2023-09-04 10:15:04] frigate.app                    INFO    : Camera processor started for garden: 299
2023-09-04 10:15:04.077967987  [2023-09-04 10:15:04] frigate.app                    INFO    : Camera processor started for front: 300
2023-09-04 10:15:04.081965508  [2023-09-04 10:15:04] frigate.app                    INFO    : Capture process started for garage: 301
2023-09-04 10:15:04.085938251  [2023-09-04 10:15:04] frigate.app                    INFO    : Capture process started for back_porch: 302
2023-09-04 10:15:04.089626283  [2023-09-04 10:15:04] frigate.app                    INFO    : Capture process started for garden: 304
2023-09-04 10:15:04.094096601  [2023-09-04 10:15:04] frigate.app                    INFO    : Capture process started for front: 308
2023-09-04 10:15:04.205569471  [2023-09-04 10:15:04] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-04 10:15:09.392689863  [2023-09-04 10:15:09] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:15:09.392860868  [2023-09-04 10:15:09] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:15:24.195645076  [2023-09-04 10:15:24] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-04 10:15:24.195910059  [2023-09-04 10:15:24] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:15:24.196216246  [2023-09-04 10:15:24] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Invalid data found when processing input
2023-09-04 10:15:30.382595579  [2023-09-04 10:15:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/15.15.mp4 in 0.5895159244537354 seconds.
2023-09-04 10:15:34.952805009  [2023-09-04 10:15:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/15.19.mp4 in 0.4722428321838379 seconds.
2023-09-04 10:15:39.854262613  [2023-09-04 10:15:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/15.27.mp4 in 0.3969099521636963 seconds.
2023-09-04 10:15:40.493910520  [2023-09-04 10:15:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/15.27.mp4 in 0.4936330318450928 seconds.
2023-09-04 10:15:44.845814198  [2023-09-04 10:15:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/15.31.mp4 in 0.3722379207611084 seconds.
2023-09-04 10:15:45.359441231  [2023-09-04 10:15:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/15.29.mp4 in 0.39125895500183105 seconds.
2023-09-04 10:15:50.078954034  [2023-09-04 10:15:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/15.36.mp4 in 0.4699089527130127 seconds.
2023-09-04 10:15:54.856582921  [2023-09-04 10:15:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/15.44.mp4 in 0.38998889923095703 seconds.
2023-09-04 10:15:55.369471023  [2023-09-04 10:15:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/15.39.mp4 in 0.38740110397338867 seconds.
2023-09-04 10:16:00.093937545  [2023-09-04 10:16:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/15.46.mp4 in 0.49404215812683105 seconds.
2023-09-04 10:16:04.311401672  [2023-09-04 10:16:04] frigate.record                 DEBUG   : Start expire recordings (new).
2023-09-04 10:16:04.311597868  [2023-09-04 10:16:04] frigate.record                 DEBUG   : Start deleted cameras.
2023-09-04 10:16:04.364223186  [2023-09-04 10:16:04] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-04 10:16:04.365583392  [2023-09-04 10:16:04] frigate.record                 DEBUG   : End deleted cameras.
2023-09-04 10:16:04.365784248  [2023-09-04 10:16:04] frigate.record                 DEBUG   : Start all cameras.
2023-09-04 10:16:04.366034405  [2023-09-04 10:16:04] frigate.record                 DEBUG   : Start camera: garage.
2023-09-04 10:16:04.903958348  [2023-09-04 10:16:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/15.52.mp4 in 0.42452406883239746 seconds.
2023-09-04 10:16:06.648160240  [2023-09-04 10:16:06] frigate.record                 DEBUG   : Expiring 17 recordings
2023-09-04 10:16:06.655824369  [2023-09-04 10:16:06] frigate.record                 DEBUG   : End camera: garage.
2023-09-04 10:16:06.656008566  [2023-09-04 10:16:06] frigate.record                 DEBUG   : Start camera: back_porch.
2023-09-04 10:16:07.439866745  [2023-09-04 10:16:07] frigate.record                 DEBUG   : Expiring 46 recordings
2023-09-04 10:16:07.450658020  [2023-09-04 10:16:07] frigate.record                 DEBUG   : End camera: back_porch.
2023-09-04 10:16:07.450782829  [2023-09-04 10:16:07] frigate.record                 DEBUG   : Start camera: garden.
2023-09-04 10:16:08.197474465  [2023-09-04 10:16:08] frigate.record                 DEBUG   : Expiring 45 recordings
2023-09-04 10:16:08.204426329  [2023-09-04 10:16:08] frigate.record                 DEBUG   : End camera: garden.
2023-09-04 10:16:08.204619763  [2023-09-04 10:16:08] frigate.record                 DEBUG   : Start camera: front.
2023-09-04 10:16:08.899470506  [2023-09-04 10:16:08] frigate.record                 DEBUG   : Expiring 14 recordings
2023-09-04 10:16:08.906357637  [2023-09-04 10:16:08] frigate.record                 DEBUG   : End camera: front.
2023-09-04 10:16:08.906562717  [2023-09-04 10:16:08] frigate.record                 DEBUG   : End all cameras.
2023-09-04 10:16:08.906798262  [2023-09-04 10:16:08] frigate.record                 DEBUG   : End expire recordings (new).
2023-09-04 10:16:08.906949811  [2023-09-04 10:16:08] frigate.record                 DEBUG   : Start expire files (legacy).
2023-09-04 10:16:08.958822249  [2023-09-04 10:16:08] frigate.record                 DEBUG   : Oldest recording in the db: 1691421724.7690797
2023-09-04 10:16:09.909925719  [2023-09-04 10:16:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/15.57.mp4 in 0.422421932220459 seconds.
2023-09-04 10:16:10.681941988  [2023-09-04 10:16:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/15.56.mp4 in 0.5090038776397705 seconds.
2023-09-04 10:16:14.960635290  [2023-09-04 10:16:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/16.01.mp4 in 0.452908992767334 seconds.
2023-09-04 10:16:18.364013778  [2023-09-04 10:16:18] frigate.record                 DEBUG   : End expire files (legacy).
2023-09-04 10:16:19.926965398  [2023-09-04 10:16:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/16.07.mp4 in 0.4322810173034668 seconds.
2023-09-04 10:16:20.673057667  [2023-09-04 10:16:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/16.07.mp4 in 0.4873688220977783 seconds.
2023-09-04 10:16:24.845802002  [2023-09-04 10:16:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/16.11.mp4 in 0.39127302169799805 seconds.
2023-09-04 10:16:30.210613703  [2023-09-04 10:16:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/16.16.mp4 in 0.47377514839172363 seconds.
2023-09-04 10:16:34.875464627  [2023-09-04 10:16:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/16.21.mp4 in 0.4052128791809082 seconds.
2023-09-04 10:16:40.217027288  [2023-09-04 10:16:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/16.27.mp4 in 0.499100923538208 seconds.
2023-09-04 10:16:44.842528268  [2023-09-04 10:16:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/16.31.mp4 in 0.4024372100830078 seconds.
2023-09-04 10:16:49.973375811  [2023-09-04 10:16:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/16.39.mp4 in 0.37520599365234375 seconds.
2023-09-04 10:16:50.620567647  [2023-09-04 10:16:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/16.36.mp4 in 0.4948899745941162 seconds.
2023-09-04 10:16:54.835816132  [2023-09-04 10:16:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/16.41.mp4 in 0.3823659420013428 seconds.
2023-09-04 10:17:00.193611742  [2023-09-04 10:17:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/16.46.mp4 in 0.4801931381225586 seconds.
2023-09-04 10:17:04.849992994  [2023-09-04 10:17:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/16.51.mp4 in 0.39429807662963867 seconds.
2023-09-04 10:17:09.892167308  [2023-09-04 10:17:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/16.57.mp4 in 0.41464686393737793 seconds.
2023-09-04 10:17:10.408701059  [2023-09-04 10:17:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/16.59.mp4 in 0.386991024017334 seconds.
2023-09-04 10:17:14.891081668  [2023-09-04 10:17:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/17.01.mp4 in 0.42399001121520996 seconds.
2023-09-04 10:17:19.962940236  [2023-09-04 10:17:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/17.09.mp4 in 0.3833169937133789 seconds.
2023-09-04 10:17:20.584206818  [2023-09-04 10:17:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/17.06.mp4 in 0.47808003425598145 seconds.
2023-09-04 10:17:24.891233649  [2023-09-04 10:17:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/17.11.mp4 in 0.4261300563812256 seconds.
2023-09-04 10:17:30.022704845  [2023-09-04 10:17:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/17.19.mp4 in 0.4129018783569336 seconds.
2023-09-04 10:17:30.637201745  [2023-09-04 10:17:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/17.16.mp4 in 0.4799950122833252 seconds.
2023-09-04 10:17:34.854222789  [2023-09-04 10:17:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/17.22.mp4 in 0.39424896240234375 seconds.
2023-09-04 10:17:39.961427584  [2023-09-04 10:17:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/17.29.mp4 in 0.37264513969421387 seconds.
2023-09-04 10:17:40.615959131  [2023-09-04 10:17:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/17.26.mp4 in 0.5032961368560791 seconds.
2023-09-04 10:17:49.851928413  [2023-09-04 10:17:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/17.31.mp4 in 0.3874979019165039 seconds.
2023-09-04 10:17:50.407479172  [2023-09-04 10:17:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/17.37.mp4 in 0.41117000579833984 seconds.
2023-09-04 10:17:51.044254221  [2023-09-04 10:17:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/17.36.mp4 in 0.5100250244140625 seconds.
2023-09-04 10:17:59.869984200  [2023-09-04 10:17:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/17.44.mp4 in 0.4101431369781494 seconds.
2023-09-04 10:18:00.417341694  [2023-09-04 10:18:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/17.47.mp4 in 0.3950839042663574 seconds.
2023-09-04 10:18:01.028523518  [2023-09-04 10:18:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/17.46.mp4 in 0.48421311378479004 seconds.
2023-09-04 10:18:04.825441578  [2023-09-04 10:18:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/17.56.mp4 in 0.3833630084991455 seconds.
2023-09-04 10:18:05.311287670  [2023-09-04 10:18:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/17.49.mp4 in 0.37472009658813477 seconds.
2023-09-04 10:18:09.908730219  [2023-09-04 10:18:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/17.57.mp4 in 0.42136383056640625 seconds.
2023-09-04 10:18:10.405165179  [2023-09-04 10:18:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/17.59.mp4 in 0.3757960796356201 seconds.
2023-09-04 10:18:11.025392284  [2023-09-04 10:18:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/17.56.mp4 in 0.4807760715484619 seconds.
2023-09-04 10:18:14.839497135  [2023-09-04 10:18:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/18.03.mp4 in 0.3963298797607422 seconds.
2023-09-04 10:18:19.954614187  [2023-09-04 10:18:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/18.09.mp4 in 0.3787388801574707 seconds.
2023-09-04 10:18:20.614180645  [2023-09-04 10:18:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/18.06.mp4 in 0.49323296546936035 seconds.
2023-09-04 10:18:24.811465870  [2023-09-04 10:18:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/18.11.mp4 in 0.37044191360473633 seconds.
2023-09-04 10:18:29.981606442  [2023-09-04 10:18:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/18.19.mp4 in 0.3700220584869385 seconds.
2023-09-04 10:18:30.858417071  [2023-09-04 10:18:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/18.16.mp4 in 0.7451310157775879 seconds.
2023-09-04 10:18:34.881943541  [2023-09-04 10:18:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/18.21.mp4 in 0.41901206970214844 seconds.
2023-09-04 10:18:39.954723632  [2023-09-04 10:18:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/18.29.mp4 in 0.3884410858154297 seconds.
2023-09-04 10:18:40.630214876  [2023-09-04 10:18:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/18.26.mp4 in 0.5071079730987549 seconds.
2023-09-04 10:18:44.868450005  [2023-09-04 10:18:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/18.32.mp4 in 0.4128890037536621 seconds.
2023-09-04 10:18:50.019966454  [2023-09-04 10:18:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/18.39.mp4 in 0.39463114738464355 seconds.
2023-09-04 10:18:50.649923410  [2023-09-04 10:18:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/18.36.mp4 in 0.4944009780883789 seconds.
2023-09-04 10:18:54.839743048  [2023-09-04 10:18:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/18.42.mp4 in 0.38460707664489746 seconds.
2023-09-04 10:18:59.995828892  [2023-09-04 10:18:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/18.49.mp4 in 0.38834095001220703 seconds.
2023-09-04 10:19:00.588235143  [2023-09-04 10:19:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/18.46.mp4 in 0.46636199951171875 seconds.
2023-09-04 10:19:04.820700140  [2023-09-04 10:19:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/18.51.mp4 in 0.38601207733154297 seconds.
2023-09-04 10:19:09.994429018  [2023-09-04 10:19:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/18.59.mp4 in 0.3958170413970947 seconds.
2023-09-04 10:19:10.560488925  [2023-09-04 10:19:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/18.56.mp4 in 0.4576120376586914 seconds.
2023-09-04 10:19:14.843030315  [2023-09-04 10:19:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/19.01.mp4 in 0.39045095443725586 seconds.
2023-09-04 10:19:19.995185829  [2023-09-04 10:19:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/19.06.mp4 in 0.44513797760009766 seconds.
2023-09-04 10:19:24.859716923  [2023-09-04 10:19:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/19.13.mp4 in 0.41448497772216797 seconds.
2023-09-04 10:19:25.359177915  [2023-09-04 10:19:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/19.08.mp4 in 0.38289713859558105 seconds.
2023-09-04 10:19:30.082199320  [2023-09-04 10:19:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/19.16.mp4 in 0.4893159866333008 seconds.
2023-09-04 10:19:31.244223361  [2023-09-04 10:19:31] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:19:31.244430437  [2023-09-04 10:19:31] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:19:34.221168202  [2023-09-04 10:19:34] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-04 10:19:34.221395727  [2023-09-04 10:19:34] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:19:34.221399364  [2023-09-04 10:19:34] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Connection timed out
2023-09-04 10:19:34.221534665  [2023-09-04 10:19:34] ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
2023-09-04 10:19:34.971974185  [2023-09-04 10:19:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/19.23.mp4 in 0.3746349811553955 seconds.
2023-09-04 10:19:39.338330172  [2023-09-04 10:19:39] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:19:39.338335456  [2023-09-04 10:19:39] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:19:39.961712521  [2023-09-04 10:19:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/19.31.mp4 in 0.3799450397491455 seconds.
2023-09-04 10:19:40.590491089  [2023-09-04 10:19:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/19.26.mp4 in 0.4848790168762207 seconds.
2023-09-04 10:19:44.228949654  [2023-09-04 10:19:44] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-04 10:19:44.229134643  [2023-09-04 10:19:44] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:19:44.229282131  [2023-09-04 10:19:44] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Invalid data found when processing input
2023-09-04 10:19:49.346216473  [2023-09-04 10:19:49] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:19:49.346433167  [2023-09-04 10:19:49] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:19:50.209887704  [2023-09-04 10:19:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/19.36.mp4 in 0.4726521968841553 seconds.
2023-09-04 10:19:54.233330997  [2023-09-04 10:19:54] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-04 10:19:54.233624481  [2023-09-04 10:19:54] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:19:54.233627910  [2023-09-04 10:19:54] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Invalid data found when processing input
2023-09-04 10:19:59.358496007  [2023-09-04 10:19:59] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:19:59.358700965  [2023-09-04 10:19:59] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:20:00.188981266  [2023-09-04 10:20:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/19.46.mp4 in 0.48575401306152344 seconds.
2023-09-04 10:20:04.242723127  [2023-09-04 10:20:04] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-04 10:20:04.242857708  [2023-09-04 10:20:04] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:20:04.243059648  [2023-09-04 10:20:04] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Invalid data found when processing input
2023-09-04 10:20:09.899309974  [2023-09-04 10:20:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/19.57.mp4 in 0.421314001083374 seconds.
2023-09-04 10:20:10.413329071  [2023-09-04 10:20:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/19.58.mp4 in 0.37975001335144043 seconds.
2023-09-04 10:20:11.089884283  [2023-09-04 10:20:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/19.56.mp4 in 0.5164940357208252 seconds.
2023-09-04 10:20:14.883680734  [2023-09-04 10:20:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/20.01.mp4 in 0.4203770160675049 seconds.
2023-09-04 10:20:20.196367742  [2023-09-04 10:20:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/20.06.mp4 in 0.4896512031555176 seconds.
2023-09-04 10:20:24.877391001  [2023-09-04 10:20:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/20.10.mp4 in 0.40445613861083984 seconds.
2023-09-04 10:20:30.272396302  [2023-09-04 10:20:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/20.16.mp4 in 0.5109100341796875 seconds.
2023-09-04 10:20:34.953073771  [2023-09-04 10:20:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/20.20.mp4 in 0.4649529457092285 seconds.
2023-09-04 10:20:40.259046722  [2023-09-04 10:20:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/20.26.mp4 in 0.5045630931854248 seconds.
2023-09-04 10:20:44.910177416  [2023-09-04 10:20:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/20.30.mp4 in 0.437147855758667 seconds.
2023-09-04 10:20:50.230382401  [2023-09-04 10:20:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/20.36.mp4 in 0.4922349452972412 seconds.
2023-09-04 10:20:54.918101191  [2023-09-04 10:20:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/20.40.mp4 in 0.4279952049255371 seconds.
2023-09-04 10:21:10.217663930  [2023-09-04 10:21:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/20.56.mp4 in 0.49664902687072754 seconds.
2023-09-04 10:21:20.200418768  [2023-09-04 10:21:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/21.06.mp4 in 0.48848414421081543 seconds.
2023-09-04 10:21:24.901991426  [2023-09-04 10:21:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/21.10.mp4 in 0.41906094551086426 seconds.
2023-09-04 10:21:34.868608172  [2023-09-04 10:21:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/21.22.mp4 in 0.407944917678833 seconds.
2023-09-04 10:21:44.873239167  [2023-09-04 10:21:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/21.31.mp4 in 0.4031641483306885 seconds.
2023-09-04 10:21:50.269061954  [2023-09-04 10:21:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/21.36.mp4 in 0.5050570964813232 seconds.
2023-09-04 10:21:54.899976436  [2023-09-04 10:21:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/21.44.mp4 in 0.4310750961303711 seconds.
2023-09-04 10:22:00.237946498  [2023-09-04 10:22:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/21.46.mp4 in 0.5200698375701904 seconds.
2023-09-04 10:22:04.856783501  [2023-09-04 10:22:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/21.53.mp4 in 0.39836597442626953 seconds.
2023-09-04 10:22:10.256946351  [2023-09-04 10:22:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/21.56.mp4 in 0.5174000263214111 seconds.
2023-09-04 10:22:15.120300609  [2023-09-04 10:22:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/22.01.mp4 in 0.6456568241119385 seconds.
2023-09-04 10:22:20.161834366  [2023-09-04 10:22:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/22.06.mp4 in 0.4548311233520508 seconds.
2023-09-04 10:22:24.882061240  [2023-09-04 10:22:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/22.13.mp4 in 0.4063870906829834 seconds.
2023-09-04 10:22:30.046222371  [2023-09-04 10:22:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/22.16.mp4 in 0.47011899948120117 seconds.
2023-09-04 10:22:34.918004521  [2023-09-04 10:22:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/22.21.mp4 in 0.4248349666595459 seconds.
2023-09-04 10:22:39.893552373  [2023-09-04 10:22:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/22.27.mp4 in 0.41162800788879395 seconds.
2023-09-04 10:22:40.515148372  [2023-09-04 10:22:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/22.26.mp4 in 0.4880859851837158 seconds.
2023-09-04 10:22:44.913510247  [2023-09-04 10:22:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/22.30.mp4 in 0.4265608787536621 seconds.
2023-09-04 10:22:48.802307027  [2023-09-04 10:22:48] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-04 10:22:48.802545578  [2023-09-04 10:22:48] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:22:49.885839661  [2023-09-04 10:22:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/22.37.mp4 in 0.40192508697509766 seconds.
2023-09-04 10:22:50.691238440  [2023-09-04 10:22:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/22.36.mp4 in 0.6801438331604004 seconds.
2023-09-04 10:22:54.213096151  [2023-09-04 10:22:54] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-04 10:22:54.213317452  [2023-09-04 10:22:54] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:22:54.213440666  [2023-09-04 10:22:54] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Connection timed out
2023-09-04 10:22:54.213557836  [2023-09-04 10:22:54] ffmpeg.garage.detect           ERROR   :     Last message repeated 1 times
2023-09-04 10:22:54.884718683  [2023-09-04 10:22:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/22.40.mp4 in 0.41924095153808594 seconds.
2023-09-04 10:22:59.334871681  [2023-09-04 10:22:59] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-04 10:22:59.337004882  [2023-09-04 10:22:59] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:22:59.885812599  [2023-09-04 10:22:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/22.46.mp4 in 0.4005608558654785 seconds.
2023-09-04 10:23:00.622302926  [2023-09-04 10:23:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/22.46.mp4 in 0.48885107040405273 seconds.
2023-09-04 10:23:04.222321610  [2023-09-04 10:23:04] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-04 10:23:04.222667217  [2023-09-04 10:23:04] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:23:04.222928159  [2023-09-04 10:23:04] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Invalid data found when processing input
2023-09-04 10:23:04.880415510  [2023-09-04 10:23:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/22.50.mp4 in 0.4092259407043457 seconds.
2023-09-04 10:23:09.346758990  [2023-09-04 10:23:09] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-04 10:23:09.346974856  [2023-09-04 10:23:09] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:23:10.187085019  [2023-09-04 10:23:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/22.56.mp4 in 0.48333096504211426 seconds.
2023-09-04 10:23:14.232346958  [2023-09-04 10:23:14] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-04 10:23:14.232595621  [2023-09-04 10:23:14] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:23:14.232752823  [2023-09-04 10:23:14] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Invalid data found when processing input
2023-09-04 10:23:14.925779414  [2023-09-04 10:23:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/23.00.mp4 in 0.4485890865325928 seconds.
2023-09-04 10:23:20.252642960  [2023-09-04 10:23:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/23.06.mp4 in 0.5045428276062012 seconds.
2023-09-04 10:23:24.940598606  [2023-09-04 10:23:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/23.10.mp4 in 0.45052313804626465 seconds.
2023-09-04 10:23:30.253678267  [2023-09-04 10:23:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/23.16.mp4 in 0.5007500648498535 seconds.
2023-09-04 10:23:34.909500618  [2023-09-04 10:23:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/23.20.mp4 in 0.4181249141693115 seconds.
2023-09-04 10:23:40.052708668  [2023-09-04 10:23:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/23.28.mp4 in 0.4466850757598877 seconds.
2023-09-04 10:23:40.932492508  [2023-09-04 10:23:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/23.26.mp4 in 0.7433187961578369 seconds.
2023-09-04 10:23:50.020078447  [2023-09-04 10:23:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/23.30.mp4 in 0.4461989402770996 seconds.
2023-09-04 10:23:50.703455093  [2023-09-04 10:23:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/23.38.mp4 in 0.43014001846313477 seconds.
2023-09-04 10:23:51.371079205  [2023-09-04 10:23:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/23.36.mp4 in 0.5443949699401855 seconds.
2023-09-04 10:23:54.879557999  [2023-09-04 10:23:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/23.46.mp4 in 0.4034240245819092 seconds.
2023-09-04 10:24:00.023804229  [2023-09-04 10:24:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/23.48.mp4 in 0.4363420009613037 seconds.
2023-09-04 10:24:00.622013052  [2023-09-04 10:24:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/23.46.mp4 in 0.49317407608032227 seconds.
2023-09-04 10:24:04.900884348  [2023-09-04 10:24:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/23.52.mp4 in 0.43970179557800293 seconds.
2023-09-04 10:24:10.157549439  [2023-09-04 10:24:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/23.56.mp4 in 0.45802807807922363 seconds.
2023-09-04 10:24:14.875071242  [2023-09-04 10:24:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/24.00.mp4 in 0.40620994567871094 seconds.
2023-09-04 10:24:20.237414360  [2023-09-04 10:24:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/24.06.mp4 in 0.4938669204711914 seconds.
2023-09-04 10:24:24.897786329  [2023-09-04 10:24:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/24.11.mp4 in 0.43105506896972656 seconds.
2023-09-04 10:24:30.181805840  [2023-09-04 10:24:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/24.16.mp4 in 0.4672231674194336 seconds.
2023-09-04 10:24:34.927826238  [2023-09-04 10:24:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/24.20.mp4 in 0.4118950366973877 seconds.
2023-09-04 10:24:40.239737263  [2023-09-04 10:24:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/24.26.mp4 in 0.4903731346130371 seconds.
2023-09-04 10:24:44.935876153  [2023-09-04 10:24:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/24.30.mp4 in 0.41783690452575684 seconds.
2023-09-04 10:24:50.193414959  [2023-09-04 10:24:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/24.36.mp4 in 0.4835810661315918 seconds.
2023-09-04 10:24:54.911735232  [2023-09-04 10:24:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/24.41.mp4 in 0.4218418598175049 seconds.
2023-09-04 10:25:00.252255052  [2023-09-04 10:25:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/24.46.mp4 in 0.502108097076416 seconds.
2023-09-04 10:25:04.918614578  [2023-09-04 10:25:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/24.50.mp4 in 0.41270995140075684 seconds.
2023-09-04 10:25:10.207721316  [2023-09-04 10:25:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/24.56.mp4 in 0.4940488338470459 seconds.
2023-09-04 10:25:19.867233731  [2023-09-04 10:25:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/25.03.mp4 in 0.3896639347076416 seconds.
2023-09-04 10:25:20.718413642  [2023-09-04 10:25:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/25.06.mp4 in 0.47098517417907715 seconds.
2023-09-04 10:25:30.209949278  [2023-09-04 10:25:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/25.16.mp4 in 0.49646782875061035 seconds.
2023-09-04 10:25:34.892104096  [2023-09-04 10:25:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/25.15.mp4 in 0.41216301918029785 seconds.
2023-09-04 10:25:39.891292615  [2023-09-04 10:25:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/25.29.mp4 in 0.3915371894836426 seconds.
2023-09-04 10:25:40.730599092  [2023-09-04 10:25:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/25.26.mp4 in 0.47292089462280273 seconds.
2023-09-04 10:25:49.884899171  [2023-09-04 10:25:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/25.37.mp4 in 0.4049341678619385 seconds.
2023-09-04 10:25:50.436349613  [2023-09-04 10:25:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/25.36.mp4 in 0.415848970413208 seconds.
2023-09-04 10:25:51.184991397  [2023-09-04 10:25:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/25.36.mp4 in 0.49258899688720703 seconds.
2023-09-04 10:26:04.266869980  [2023-09-04 10:26:04] watchdog.back_porch            INFO    : No frames received from back_porch in 20 seconds. Exiting ffmpeg...
2023-09-04 10:26:04.267012845  [2023-09-04 10:26:04] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-04 10:26:04.900621334  [2023-09-04 10:26:04] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:26:04.901639683  [2023-09-04 10:26:04] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-04 10:26:04.901643512  [2023-09-04 10:26:04] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-04 10:26:10.404338506  [2023-09-04 10:26:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/25.56.mp4 in 0.7020370960235596 seconds.
2023-09-04 10:26:14.901644413  [2023-09-04 10:26:14] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-04 10:26:14.901877663  [2023-09-04 10:26:14] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 10:26:14.901881343  [2023-09-04 10:26:14] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Connection timed out
2023-09-04 10:26:20.524807794  [2023-09-04 10:26:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/26.05.mp4 in 0.7764101028442383 seconds.
2023-09-04 10:26:24.959326377  [2023-09-04 10:26:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/26.10.mp4 in 0.461076021194458 seconds.
2023-09-04 10:26:30.460809602  [2023-09-04 10:26:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/26.16.mp4 in 0.6156048774719238 seconds.
2023-09-04 10:26:35.030974639  [2023-09-04 10:26:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/26.20.mp4 in 0.5111241340637207 seconds.
2023-09-04 10:26:40.061398746  [2023-09-04 10:26:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/26.28.mp4 in 0.4278230667114258 seconds.
2023-09-04 10:26:40.787947008  [2023-09-04 10:26:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/26.25.mp4 in 0.5401110649108887 seconds.
2023-09-04 10:26:44.916321679  [2023-09-04 10:26:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/26.32.mp4 in 0.4362170696258545 seconds.
2023-09-04 10:26:50.024570149  [2023-09-04 10:26:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/26.38.mp4 in 0.4413449764251709 seconds.
2023-09-04 10:26:50.909579327  [2023-09-04 10:26:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/26.35.mp4 in 0.7565250396728516 seconds.
2023-09-04 10:26:55.116674040  [2023-09-04 10:26:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/26.41.mp4 in 0.6299550533294678 seconds.
2023-09-04 10:27:00.024487227  [2023-09-04 10:27:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/26.48.mp4 in 0.43021202087402344 seconds.
2023-09-04 10:27:01.089603741  [2023-09-04 10:27:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/26.45.mp4 in 0.9484140872955322 seconds.
2023-09-04 10:27:04.936261077  [2023-09-04 10:27:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/back_porch/26.50.mp4 in 0.46771693229675293 seconds.
2023-09-04 10:27:09.909630052  [2023-09-04 10:27:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/front/26.56.mp4 in 0.41812610626220703 seconds.
2023-09-04 10:27:10.705316146  [2023-09-04 10:27:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garage/26.58.mp4 in 0.6719338893890381 seconds.
2023-09-04 10:27:11.389705460  [2023-09-04 10:27:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-04/15/garden/26.55.mp4 in 0.533473014831543 seconds.

go2rtc logs:

2023-09-04 10:15:02.094515865  [INFO] Preparing go2rtc config...
2023-09-04 10:15:02.725527633  [INFO] Starting go2rtc...
2023-09-04 10:15:02.819627382  10:15:02.819 INF go2rtc version 1.2.0 linux/amd64
2023-09-04 10:15:02.819962296  10:15:02.819 INF [api] listen addr=:1984
2023-09-04 10:15:02.820384252  10:15:02.820 INF [rtsp] listen addr=:8554
2023-09-04 10:15:02.820878482  10:15:02.820 INF [srtp] listen addr=:8443
2023-09-04 10:15:02.821148813  10:15:02.821 INF [webrtc] listen addr=:8555
2023-09-04 10:15:10.230429872  10:15:10.230 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:221 > error=EOF
2023-09-04 10:15:12.100859270  [INFO] Starting go2rtc healthcheck service...
2023-09-04 10:20:01.669706868  10:20:01.669 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:221 > error=EOF
2023-09-04 10:20:01.669712104  10:20:01.669 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:221 > error=EOF
2023-09-04 10:20:01.669740889  10:20:01.669 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:221 > error=EOF
2023-09-04 10:23:09.955324547  10:23:09.955 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:221 > error=EOF
2023-09-04 10:23:09.955351698  10:23:09.955 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:221 > error=EOF

Fixed that error, but created another.

Initial error showed up again, with aditional python errors,

2023-09-05 17:15:45.209426625  [2023-09-05 17:15:45] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:15:45.209431697  future: <Task finished name='Task-27161' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:15:45.209433645  Traceback (most recent call last):
2023-09-05 17:15:45.209435558    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:15:45.209437258      intel_usage = get_intel_gpu_stats()
2023-09-05 17:15:45.209439126    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:15:45.209440613      p = sp.run(
2023-09-05 17:15:45.209442400    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:15:45.209444167      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:15:45.209446032    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:15:45.209447912      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:15:45.209449761    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:15:45.209451440      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:15:45.209453134  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:15:47.300476116  [2023-09-05 17:15:47] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:15:47.301739757  [2023-09-05 17:15:47] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:15:47.304543268  [2023-09-05 17:15:47] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:15:47.304712381  [2023-09-05 17:15:47] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:15:50.203654167  [2023-09-05 17:15:50] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:15:50.203658843  future: <Task finished name='Task-27165' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:15:50.203660486  Traceback (most recent call last):
2023-09-05 17:15:50.203662172    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:15:50.203663664      cpu_stats = get_cpu_stats()
2023-09-05 17:15:50.203665305    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:15:50.203666846      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:15:50.203668502    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:15:50.203670022      if get_cgroups_version() == "cgroup2":
2023-09-05 17:15:50.203671746    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:15:50.203673014      p = sp.run(
2023-09-05 17:15:50.203674628    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:15:50.203676167      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:15:50.203677792    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:15:50.203679328      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:15:50.203680922    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:15:50.203682347      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:15:50.203683767  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:15:50.203822858  [2023-09-05 17:15:50] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:15:50.203827378  future: <Task finished name='Task-27164' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:15:50.203829388  Traceback (most recent call last):
2023-09-05 17:15:50.203831515    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:15:50.203833446      intel_usage = get_intel_gpu_stats()
2023-09-05 17:15:50.203835662    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:15:50.203837226      p = sp.run(
2023-09-05 17:15:50.203839580    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:15:50.203841070      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:15:50.203842720    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:15:50.203844326      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:15:50.203845776    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:15:50.203847321      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:15:50.203848751  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:15:52.306718786  [2023-09-05 17:15:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:15:52.309603908  [2023-09-05 17:15:52] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:15:52.309744811  [2023-09-05 17:15:52] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:15:55.205167800  [2023-09-05 17:15:55] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:15:55.205172701  future: <Task finished name='Task-27168' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:15:55.205174438  Traceback (most recent call last):
2023-09-05 17:15:55.205176173    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:15:55.205177688      cpu_stats = get_cpu_stats()
2023-09-05 17:15:55.205179390    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:15:55.205180920      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:15:55.205182623    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:15:55.205184202      if get_cgroups_version() == "cgroup2":
2023-09-05 17:15:55.205185882    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:15:55.205187111      p = sp.run(
2023-09-05 17:15:55.205188645    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:15:55.205190069      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:15:55.205191613    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:15:55.205193201      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:15:55.205194759    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:15:55.205196204      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:15:55.205197872  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:15:55.205488864  [2023-09-05 17:15:55] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:15:55.205492126  future: <Task finished name='Task-27167' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:15:55.205493506  Traceback (most recent call last):
2023-09-05 17:15:55.205495168    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:15:55.205496486      intel_usage = get_intel_gpu_stats()
2023-09-05 17:15:55.205498237    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:15:55.205499431      p = sp.run(
2023-09-05 17:15:55.205501064    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:15:55.205502483      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:15:55.205504000    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:15:55.205505659      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:15:55.205507166    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:15:55.205508505      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:15:55.205509987  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:15:57.304439882  [2023-09-05 17:15:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:15:57.305596958  [2023-09-05 17:15:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:15:57.307122079  [2023-09-05 17:15:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:15:57.310245518  [2023-09-05 17:15:57] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:15:57.310526982  [2023-09-05 17:15:57] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:00.203227747  [2023-09-05 17:16:00] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:00.203232468  future: <Task finished name='Task-27171' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:00.203234227  Traceback (most recent call last):
2023-09-05 17:16:00.203235866    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:00.203237360      cpu_stats = get_cpu_stats()
2023-09-05 17:16:00.203239138    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:00.203240696      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:00.203242272    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:00.203243998      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:00.203245980    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:00.203247148      p = sp.run(
2023-09-05 17:16:00.203248639    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:00.203250046      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:00.203251743    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:00.203253249      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:00.203254914    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:00.203256412      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:00.203257883  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:00.203752156  [2023-09-05 17:16:00] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:00.203756659  future: <Task finished name='Task-27170' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:00.203758383  Traceback (most recent call last):
2023-09-05 17:16:00.203760186    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:00.203761721      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:00.203763381    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:00.203764625      p = sp.run(
2023-09-05 17:16:00.203766188    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:00.203767978      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:00.203769804    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:00.203771629      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:00.203773673    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:00.203775430      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:00.203777148  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:02.305127190  [2023-09-05 17:16:02] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:02.306289794  [2023-09-05 17:16:02] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:02.307852454  [2023-09-05 17:16:02] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:02.311819858  [2023-09-05 17:16:02] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:02.311992263  [2023-09-05 17:16:02] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:05.203020758  [2023-09-05 17:16:05] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:05.203025297  future: <Task finished name='Task-27174' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:05.203026891  Traceback (most recent call last):
2023-09-05 17:16:05.203054090    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:05.203056026      cpu_stats = get_cpu_stats()
2023-09-05 17:16:05.203057696    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:05.203059407      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:05.203061081    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:05.203062561      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:05.203064206    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:05.203065739      p = sp.run(
2023-09-05 17:16:05.203067696    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:05.203069629      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:05.203071609    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:05.203073757      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:05.203075338    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:05.203076725      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:05.203078178  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:05.203344850  [2023-09-05 17:16:05] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:05.203348695  future: <Task finished name='Task-27173' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:05.203350432  Traceback (most recent call last):
2023-09-05 17:16:05.203352241    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:05.203353746      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:05.203355471    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:05.203356784      p = sp.run(
2023-09-05 17:16:05.203358413    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:05.203359958      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:05.203361631    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:05.203363175      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:05.203364728    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:05.203366119      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:05.203367536  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:07.304133812  [2023-09-05 17:16:07] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:07.306403597  [2023-09-05 17:16:07] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:07.306522391  [2023-09-05 17:16:07] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:10.202738051  [2023-09-05 17:16:10] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:10.202742443  future: <Task finished name='Task-27177' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:10.202744052  Traceback (most recent call last):
2023-09-05 17:16:10.202745968    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:10.202747848      cpu_stats = get_cpu_stats()
2023-09-05 17:16:10.202749650    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:10.202751173      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:10.202752728    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:10.202754283      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:10.202755936    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:10.202757276      p = sp.run(
2023-09-05 17:16:10.202758768    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:10.202760245      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:10.202761775    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:10.202763430      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:10.202764971    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:10.202766669      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:10.202768168  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:10.202988572  [2023-09-05 17:16:10] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:10.202991575  future: <Task finished name='Task-27176' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:10.202993100  Traceback (most recent call last):
2023-09-05 17:16:10.202994772    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:10.202996166      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:10.202997788    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:10.202999160      p = sp.run(
2023-09-05 17:16:10.203000774    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:10.203002299      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:10.203003892    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:10.203005318      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:10.203006915    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:10.203008407      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:10.203009977  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:12.312015550  [2023-09-05 17:16:12] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:12.313590297  [2023-09-05 17:16:12] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:12.314836489  [2023-09-05 17:16:12] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:12.318130786  [2023-09-05 17:16:12] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:12.318262819  [2023-09-05 17:16:12] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:15.203287485  [2023-09-05 17:16:15] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:15.203291911  future: <Task finished name='Task-27180' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:15.203293560  Traceback (most recent call last):
2023-09-05 17:16:15.203295223    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:15.203296569      cpu_stats = get_cpu_stats()
2023-09-05 17:16:15.203298218    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:15.203299696      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:15.203301343    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:15.203302773      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:15.203304451    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:15.203305768      p = sp.run(
2023-09-05 17:16:15.203307696    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:15.203309499      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:15.203311543    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:15.203313577      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:15.203315544    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:15.203317581      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:15.203318957  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:15.203916197  [2023-09-05 17:16:15] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:15.203920111  future: <Task finished name='Task-27179' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:15.203921656  Traceback (most recent call last):
2023-09-05 17:16:15.203923343    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:15.203924713      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:15.203926339    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:15.203927698      p = sp.run(
2023-09-05 17:16:15.203929223    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:15.203930706      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:15.203932218    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:15.203933655      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:15.203935290    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:15.203936761      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:15.203938240  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:17.307276771  [2023-09-05 17:16:17] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:17.310112076  [2023-09-05 17:16:17] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:17.310262533  [2023-09-05 17:16:17] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:20.208669751  [2023-09-05 17:16:20] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:20.208675039  future: <Task finished name='Task-27183' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:20.208676832  Traceback (most recent call last):
2023-09-05 17:16:20.208678671    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:20.208680039      cpu_stats = get_cpu_stats()
2023-09-05 17:16:20.208681513    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:20.208682924      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:20.208684391    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:20.208685670      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:20.208687163    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:20.208688412      p = sp.run(
2023-09-05 17:16:20.208689852    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:20.208691178      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:20.208692734    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:20.208694082      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:20.208695519    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:20.208696862      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:20.208698179  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:20.208699748  [2023-09-05 17:16:20] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:20.208701887  future: <Task finished name='Task-27182' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:20.208703205  Traceback (most recent call last):
2023-09-05 17:16:20.208704641    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:20.208706023      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:20.208707636    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:20.208708911      p = sp.run(
2023-09-05 17:16:20.208710508    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:20.208711940      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:20.208713602    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:20.208715440      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:20.208717413    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:20.208718821      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:20.208720384  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:22.308830530  [2023-09-05 17:16:22] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:22.309978481  [2023-09-05 17:16:22] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:22.313896631  [2023-09-05 17:16:22] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:22.314014011  [2023-09-05 17:16:22] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:25.203998095  [2023-09-05 17:16:25] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:25.204002466  future: <Task finished name='Task-27186' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:25.204004273  Traceback (most recent call last):
2023-09-05 17:16:25.204006043    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:25.204007649      cpu_stats = get_cpu_stats()
2023-09-05 17:16:25.204009283    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:25.204010894      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:25.204012543    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:25.204014120      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:25.204015714    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:25.204016948      p = sp.run(
2023-09-05 17:16:25.204018488    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:25.204019922      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:25.204021472    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:25.204023059      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:25.204024597    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:25.204026242      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:25.204027870  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:25.204340789  [2023-09-05 17:16:25] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:25.204343967  future: <Task finished name='Task-27185' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:25.204345412  Traceback (most recent call last):
2023-09-05 17:16:25.204347089    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:25.204348596      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:25.204350042    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:25.204351499      p = sp.run(
2023-09-05 17:16:25.204352947    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:25.204354459      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:25.204356073    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:25.204357532      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:25.204359127    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:25.204360494      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:25.204361880  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:27.310558085  [2023-09-05 17:16:27] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:27.311927917  [2023-09-05 17:16:27] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:27.314745921  [2023-09-05 17:16:27] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:27.314854513  [2023-09-05 17:16:27] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:30.202112323  [2023-09-05 17:16:30] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:30.202116851  future: <Task finished name='Task-27189' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:30.202118453  Traceback (most recent call last):
2023-09-05 17:16:30.202120271    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:30.202121682      cpu_stats = get_cpu_stats()
2023-09-05 17:16:30.202123371    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:30.202125092      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:30.202126756    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:30.202128227      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:30.202129827    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:30.202131051      p = sp.run(
2023-09-05 17:16:30.202132557    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:30.202134126      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:30.202135710    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:30.202137321      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:30.202139082    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:30.202140982      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:30.202142803  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:30.202398801  [2023-09-05 17:16:30] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:30.202402549  future: <Task finished name='Task-27188' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:30.202404428  Traceback (most recent call last):
2023-09-05 17:16:30.202406405    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:30.202407925      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:30.202417688    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:30.202419000      p = sp.run(
2023-09-05 17:16:30.202420480    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:30.202421892      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:30.202423430    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:30.202424830      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:30.202426426    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:30.202427916      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:30.202429255  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:32.305482142  [2023-09-05 17:16:32] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:32.306751982  [2023-09-05 17:16:32] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:32.308133001  [2023-09-05 17:16:32] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:32.311211551  [2023-09-05 17:16:32] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:32.311284616  [2023-09-05 17:16:32] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:35.204276840  [2023-09-05 17:16:35] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:35.204281412  future: <Task finished name='Task-27192' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:35.204283193  Traceback (most recent call last):
2023-09-05 17:16:35.204284962    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:35.204286433      cpu_stats = get_cpu_stats()
2023-09-05 17:16:35.204288077    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:35.204289633      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:35.204291301    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:35.204292715      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:35.204294374    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:35.204295599      p = sp.run(
2023-09-05 17:16:35.204297135    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:35.204298623      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:35.204300140    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:35.204301626      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:35.204303170    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:35.204304559      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:35.204305960  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:35.204442023  [2023-09-05 17:16:35] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:35.204667008  future: <Task finished name='Task-27191' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:35.204669490  Traceback (most recent call last):
2023-09-05 17:16:35.204671626    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:35.204673401      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:35.204675148    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:35.204676460      p = sp.run(
2023-09-05 17:16:35.204678052    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:35.204679610      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:35.204681830    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:35.204683344      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:35.204684912    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:35.204686735      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:35.204688570  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:37.305834466  [2023-09-05 17:16:37] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:37.310053070  [2023-09-05 17:16:37] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:37.310275080  [2023-09-05 17:16:37] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:38.453469607  [2023-09-05 17:16:38] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:38.453474391  future: <Task finished name='Task-27195' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:38.453476095  Traceback (most recent call last):
2023-09-05 17:16:38.453477900    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:38.453479348      cpu_stats = get_cpu_stats()
2023-09-05 17:16:38.453480924    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:38.453482512      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:38.453484066    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:38.453485552      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:38.453487313    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:38.453488592      p = sp.run(
2023-09-05 17:16:38.453490225    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:38.453491601      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:38.453493140    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:38.453494593      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:38.453496180    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:38.453497665      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:38.453499121  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:38.453812005  [2023-09-05 17:16:38] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:38.453815466  future: <Task finished name='Task-27194' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:38.453816900  Traceback (most recent call last):
2023-09-05 17:16:38.453818410    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:38.453820002      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:38.453821535    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:38.453822737      p = sp.run(
2023-09-05 17:16:38.453824427    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:38.453825750      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:38.453827253    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:38.453828956      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:38.453830513    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:38.453831920      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:38.453833280  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:40.204274133  [2023-09-05 17:16:40] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:40.204279291  future: <Task finished name='Task-27198' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:40.204281012  Traceback (most recent call last):
2023-09-05 17:16:40.204282767    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:40.204284260      cpu_stats = get_cpu_stats()
2023-09-05 17:16:40.204285777    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:40.204287355      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:40.204289149    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:40.204290615      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:40.204292252    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:40.204293525      p = sp.run(
2023-09-05 17:16:40.204295050    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:40.204296519      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:40.204298152    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:40.204299751      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:40.204301341    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:40.204302794      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:40.204304493  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:40.205176956  [2023-09-05 17:16:40] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:40.205181778  future: <Task finished name='Task-27197' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:40.205183589  Traceback (most recent call last):
2023-09-05 17:16:40.205185356    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:40.205186876      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:40.205188585    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:40.205189985      p = sp.run(
2023-09-05 17:16:40.205191440    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:40.205193171      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:40.205194994    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:40.205196980      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:40.205198534    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:40.205199830      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:40.205201114  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:42.309553039  [2023-09-05 17:16:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:42.310972754  [2023-09-05 17:16:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:42.312323739  [2023-09-05 17:16:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:42.313343583  [2023-09-05 17:16:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:42.316088883  [2023-09-05 17:16:42] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:42.316191691  [2023-09-05 17:16:42] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:45.202978189  [2023-09-05 17:16:45] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:45.202983698  future: <Task finished name='Task-27201' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:45.202985690  Traceback (most recent call last):
2023-09-05 17:16:45.202987945    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:45.202989814      cpu_stats = get_cpu_stats()
2023-09-05 17:16:45.202991807    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:45.202993918      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:45.202996199    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:45.202998033      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:45.203000242    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:45.203001834      p = sp.run(
2023-09-05 17:16:45.203027737    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:45.203029467      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:45.203031076    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:45.203032650      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:45.203034233    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:45.203035771      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:45.203037251  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:45.203859310  [2023-09-05 17:16:45] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:45.203863746  future: <Task finished name='Task-27200' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:45.203865563  Traceback (most recent call last):
2023-09-05 17:16:45.203867218    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:45.203868720      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:45.203870453    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:45.203871700      p = sp.run(
2023-09-05 17:16:45.203873136    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:45.203874687      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:45.203876254    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:45.203877679      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:45.203879071    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:45.203880672      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:45.203882406  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:47.306624146  [2023-09-05 17:16:47] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:47.306686853  [2023-09-05 17:16:47] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:50.205762769  [2023-09-05 17:16:50] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:50.205767431  future: <Task finished name='Task-27204' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:50.205769336  Traceback (most recent call last):
2023-09-05 17:16:50.205771145    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:50.205772649      cpu_stats = get_cpu_stats()
2023-09-05 17:16:50.205774358    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:50.205775981      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:50.205777616    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:50.205779113      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:50.205780879    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:50.205782244      p = sp.run(
2023-09-05 17:16:50.205783828    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:50.205785556      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:50.205787451    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:50.205789918      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:50.205791428    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:50.205792770      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:50.205794210  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:50.206253344  [2023-09-05 17:16:50] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:50.206257630  future: <Task finished name='Task-27203' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:50.206259459  Traceback (most recent call last):
2023-09-05 17:16:50.206261328    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:50.206262901      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:50.206264496    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:50.206265868      p = sp.run(
2023-09-05 17:16:50.206267463    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:50.206268931      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:50.206270514    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:50.206272038      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:50.206273583    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:50.206275091      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:50.206276531  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:52.311159309  [2023-09-05 17:16:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:52.312358610  [2023-09-05 17:16:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:52.313965377  [2023-09-05 17:16:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:52.317507715  [2023-09-05 17:16:52] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:52.317625875  [2023-09-05 17:16:52] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:16:55.206165865  [2023-09-05 17:16:55] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:55.206170408  future: <Task finished name='Task-27207' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:55.206172109  Traceback (most recent call last):
2023-09-05 17:16:55.206173858    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:16:55.206175293      cpu_stats = get_cpu_stats()
2023-09-05 17:16:55.206176964    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:16:55.206178557      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:16:55.206180155    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:16:55.206181644      if get_cgroups_version() == "cgroup2":
2023-09-05 17:16:55.206183174    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:16:55.206184497      p = sp.run(
2023-09-05 17:16:55.206186341    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:55.206188225      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:55.206189723    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:55.206191371      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:55.206192899    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:55.206194399      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:55.206195735  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:55.206505749  [2023-09-05 17:16:55] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:16:55.206508723  future: <Task finished name='Task-27206' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:16:55.206510112  Traceback (most recent call last):
2023-09-05 17:16:55.206511632    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:16:55.206513184      intel_usage = get_intel_gpu_stats()
2023-09-05 17:16:55.206515134    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:16:55.206516497      p = sp.run(
2023-09-05 17:16:55.206518146    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:16:55.206519594      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:16:55.206521247    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:16:55.206523148      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:16:55.206525033    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:16:55.206526552      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:16:55.206528162  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:16:57.310661653  [2023-09-05 17:16:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:16:57.313569665  [2023-09-05 17:16:57] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:16:57.313685680  [2023-09-05 17:16:57] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:00.205909440  [2023-09-05 17:17:00] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:00.205914223  future: <Task finished name='Task-27210' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:00.205915928  Traceback (most recent call last):
2023-09-05 17:17:00.205917512    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:00.205918820      cpu_stats = get_cpu_stats()
2023-09-05 17:17:00.205920271    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:00.205921873      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:00.205923470    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:00.205925004      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:00.205926658    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:00.205927982      p = sp.run(
2023-09-05 17:17:00.205929543    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:00.205931066      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:00.205932700    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:00.205934182      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:00.205935607    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:00.205936960      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:00.205938529  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:00.206308907  [2023-09-05 17:17:00] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:00.206312397  future: <Task finished name='Task-27209' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:00.206313928  Traceback (most recent call last):
2023-09-05 17:17:00.206315407    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:00.206316782      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:00.206318263    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:00.206319585      p = sp.run(
2023-09-05 17:17:00.206321091    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:00.206322645      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:00.206324146    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:00.206325681      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:00.206327228    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:00.206328554      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:00.206329936  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:02.306081925  [2023-09-05 17:17:02] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:02.307274825  [2023-09-05 17:17:02] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:02.308985878  [2023-09-05 17:17:02] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:02.312706240  [2023-09-05 17:17:02] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:02.312821526  [2023-09-05 17:17:02] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:05.204145619  [2023-09-05 17:17:05] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:05.204150334  future: <Task finished name='Task-27213' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:05.204151900  Traceback (most recent call last):
2023-09-05 17:17:05.204153629    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:05.204155099      cpu_stats = get_cpu_stats()
2023-09-05 17:17:05.204156839    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:05.204158294      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:05.204159877    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:05.204161194      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:05.204163029    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:05.204164276      p = sp.run(
2023-09-05 17:17:05.204165836    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:05.204167407      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:05.204169073    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:05.204170609      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:05.204172305    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:05.204173791      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:05.204191853  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:05.204382237  [2023-09-05 17:17:05] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:05.204385818  future: <Task finished name='Task-27212' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:05.204387169  Traceback (most recent call last):
2023-09-05 17:17:05.204388769    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:05.204390299      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:05.204392342    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:05.204393884      p = sp.run(
2023-09-05 17:17:05.204395708    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:05.204397122      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:05.204398643    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:05.204400360      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:05.204402436    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:05.204404356      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:05.204406130  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:07.304755195  [2023-09-05 17:17:07] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:07.307429358  [2023-09-05 17:17:07] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:07.307530977  [2023-09-05 17:17:07] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:10.205520536  [2023-09-05 17:17:10] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:10.205525581  future: <Task finished name='Task-27216' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:10.205527444  Traceback (most recent call last):
2023-09-05 17:17:10.205529400    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:10.205531035      cpu_stats = get_cpu_stats()
2023-09-05 17:17:10.205532969    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:10.205534812      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:10.205536741    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:10.205538464      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:10.205540444    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:10.205541891      p = sp.run(
2023-09-05 17:17:10.205543759    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:10.205545537      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:10.205547351    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:10.205549125      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:10.205550908    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:10.205552572      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:10.205554197  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:10.206079228  [2023-09-05 17:17:10] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:10.206083070  future: <Task finished name='Task-27215' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:10.206084846  Traceback (most recent call last):
2023-09-05 17:17:10.206086736    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:10.206088390      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:10.206090243    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:10.206091733      p = sp.run(
2023-09-05 17:17:10.206093543    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:10.206095239      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:10.206097050    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:10.206098842      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:10.206100656    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:10.206102234      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:10.206103827  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:12.306215799  [2023-09-05 17:17:12] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:12.307378459  [2023-09-05 17:17:12] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:12.310591629  [2023-09-05 17:17:12] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:12.310721804  [2023-09-05 17:17:12] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:15.206342694  [2023-09-05 17:17:15] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:15.206347701  future: <Task finished name='Task-27219' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:15.206349398  Traceback (most recent call last):
2023-09-05 17:17:15.206350984    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:15.206352426      cpu_stats = get_cpu_stats()
2023-09-05 17:17:15.206353970    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:15.206355583      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:15.206357095    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:15.206358583      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:15.206360398    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:15.206361631      p = sp.run(
2023-09-05 17:17:15.206363047    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:15.206364471      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:15.206365958    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:15.206367415      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:15.206368884    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:15.206382986      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:15.206384425  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:15.206386346  [2023-09-05 17:17:15] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:15.206388361  future: <Task finished name='Task-27218' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:15.206389812  Traceback (most recent call last):
2023-09-05 17:17:15.206391335    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:15.206392572      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:15.206394091    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:15.206395359      p = sp.run(
2023-09-05 17:17:15.206397209    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:15.206398698      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:15.206400253    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:15.206401724      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:15.206403225    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:15.206404719      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:15.206406132  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:17.314420297  [2023-09-05 17:17:17] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:17.315743014  [2023-09-05 17:17:17] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:17.318646791  [2023-09-05 17:17:17] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:17.318798602  [2023-09-05 17:17:17] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:20.203294519  [2023-09-05 17:17:20] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:20.203299844  future: <Task finished name='Task-27222' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:20.203301698  Traceback (most recent call last):
2023-09-05 17:17:20.203303447    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:20.203304971      cpu_stats = get_cpu_stats()
2023-09-05 17:17:20.203306645    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:20.203308270      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:20.203309923    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:20.203311321      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:20.203312915    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:20.203314186      p = sp.run(
2023-09-05 17:17:20.203315926    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:20.203317388      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:20.203318955    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:20.203320473      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:20.203322007    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:20.203323606      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:20.203324989  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:20.203894490  [2023-09-05 17:17:20] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:20.203897450  future: <Task finished name='Task-27221' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:20.203898681  Traceback (most recent call last):
2023-09-05 17:17:20.203899947    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:20.203901060      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:20.203902293    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:20.203903276      p = sp.run(
2023-09-05 17:17:20.203904442    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:20.203905585      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:20.203906732    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:20.203907855      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:20.203908999    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:20.203910053      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:20.203911120  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:22.307757321  [2023-09-05 17:17:22] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:22.308957441  [2023-09-05 17:17:22] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:22.310561110  [2023-09-05 17:17:22] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:22.314094427  [2023-09-05 17:17:22] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:22.314245085  [2023-09-05 17:17:22] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:25.205108185  [2023-09-05 17:17:25] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:25.205111752  future: <Task finished name='Task-27225' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:25.205113205  Traceback (most recent call last):
2023-09-05 17:17:25.205114585    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:25.205115702      cpu_stats = get_cpu_stats()
2023-09-05 17:17:25.205116975    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:25.205118151      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:25.205119384    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:25.205120495      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:25.205121747    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:25.205122715      p = sp.run(
2023-09-05 17:17:25.205123902    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:25.205125034      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:25.205126152    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:25.205127247      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:25.205128381    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:25.205129427      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:25.205130474  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:25.205284222  [2023-09-05 17:17:25] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:25.205287952  future: <Task finished name='Task-27224' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:25.205289406  Traceback (most recent call last):
2023-09-05 17:17:25.205291036    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:25.205292442      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:25.205294460    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:25.205295750      p = sp.run(
2023-09-05 17:17:25.205297285    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:25.205298822      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:25.205300563    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:25.205320053      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:25.205322286    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:25.205324105      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:25.205325959  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:27.313269056  [2023-09-05 17:17:27] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:27.315790596  [2023-09-05 17:17:27] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:27.315904917  [2023-09-05 17:17:27] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:30.204566654  [2023-09-05 17:17:30] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:30.204570902  future: <Task finished name='Task-27228' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:30.204572560  Traceback (most recent call last):
2023-09-05 17:17:30.204574374    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:30.204575895      cpu_stats = get_cpu_stats()
2023-09-05 17:17:30.204577509    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:30.204579068      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:30.204580721    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:30.204582176      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:30.204583871    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:30.204585203      p = sp.run(
2023-09-05 17:17:30.204586869    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:30.204588185      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:30.204589627    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:30.204590934      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:30.204592332    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:30.204593659      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:30.204594989  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:30.205006473  [2023-09-05 17:17:30] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:30.205009473  future: <Task finished name='Task-27227' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:30.205010896  Traceback (most recent call last):
2023-09-05 17:17:30.205012502    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:30.205013943      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:30.205015549    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:30.205016800      p = sp.run(
2023-09-05 17:17:30.205018351    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:30.205019870      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:30.205021432    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:30.205022978      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:30.205024573    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:30.205025986      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:30.205027393  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:32.314275443  [2023-09-05 17:17:32] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:32.316057084  [2023-09-05 17:17:32] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:32.317895179  [2023-09-05 17:17:32] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:32.322800715  [2023-09-05 17:17:32] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:32.323554007  [2023-09-05 17:17:32] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:35.202517627  [2023-09-05 17:17:35] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:35.202521957  future: <Task finished name='Task-27231' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:35.202523640  Traceback (most recent call last):
2023-09-05 17:17:35.202525490    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:35.202527050      cpu_stats = get_cpu_stats()
2023-09-05 17:17:35.202528571    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:35.202530149      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:35.202531848    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:35.202533345      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:35.202535032    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:35.202536310      p = sp.run(
2023-09-05 17:17:35.202537803    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:35.202539188      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:35.202540794    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:35.202542257      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:35.202543744    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:35.202545138      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:35.202546529  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:35.202948161  [2023-09-05 17:17:35] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:35.202952181  future: <Task finished name='Task-27230' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:35.202953846  Traceback (most recent call last):
2023-09-05 17:17:35.202955624    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:35.202957048      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:35.202958586    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:35.202960003      p = sp.run(
2023-09-05 17:17:35.202961526    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:35.202963215      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:35.202964663    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:35.202966326      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:35.202967868    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:35.202969225      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:35.202970780  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:37.314123334  [2023-09-05 17:17:37] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:37.317023206  [2023-09-05 17:17:37] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:37.317194078  [2023-09-05 17:17:37] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:38.464771026  [2023-09-05 17:17:38] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:38.464774497  future: <Task finished name='Task-27234' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:38.464775847  Traceback (most recent call last):
2023-09-05 17:17:38.464777152    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:38.464778340      cpu_stats = get_cpu_stats()
2023-09-05 17:17:38.464779571    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:38.464780731      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:38.464781934    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:38.464783011      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:38.464784227    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:38.464785191      p = sp.run(
2023-09-05 17:17:38.464786364    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:38.464787465      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:38.464788609    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:38.464789723      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:38.464790878    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:38.464791947      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:38.464793006  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:38.464794273  [2023-09-05 17:17:38] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:38.464795798  future: <Task finished name='Task-27233' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:38.464796909  Traceback (most recent call last):
2023-09-05 17:17:38.464798041    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:38.464799081      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:38.464800254    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:38.464801257      p = sp.run(
2023-09-05 17:17:38.464802391    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:38.464803481      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:38.464804614    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:38.464805721      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:38.464806852    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:38.464807884      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:38.464808962  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:40.202771911  [2023-09-05 17:17:40] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:40.202775738  future: <Task finished name='Task-27237' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:40.202777087  Traceback (most recent call last):
2023-09-05 17:17:40.202778404    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:40.202779546      cpu_stats = get_cpu_stats()
2023-09-05 17:17:40.202780814    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:40.202782048      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:40.202783278    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:40.202784365      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:40.202785613    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:40.202786587      p = sp.run(
2023-09-05 17:17:40.202787743    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:40.202788857      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:40.202789979    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:40.202791084      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:40.202792183    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:40.202793867      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:40.202795614  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:40.203130415  [2023-09-05 17:17:40] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:40.203133266  future: <Task finished name='Task-27236' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:40.203134712  Traceback (most recent call last):
2023-09-05 17:17:40.203136450    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:40.203137990      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:40.203139634    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:40.203140926      p = sp.run(
2023-09-05 17:17:40.203142823    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:40.203156695      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:40.203158393    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:40.203159954      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:40.203161559    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:40.203162966      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:40.203164367  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:42.310875419  [2023-09-05 17:17:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:42.312083603  [2023-09-05 17:17:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:42.313625675  [2023-09-05 17:17:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:42.315004117  [2023-09-05 17:17:42] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:42.318238926  [2023-09-05 17:17:42] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:42.318315358  [2023-09-05 17:17:42] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:45.204622575  [2023-09-05 17:17:45] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:45.204627235  future: <Task finished name='Task-27240' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:45.204628911  Traceback (most recent call last):
2023-09-05 17:17:45.204630625    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:45.204632116      cpu_stats = get_cpu_stats()
2023-09-05 17:17:45.204633752    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:45.204635460      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:45.204637072    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:45.204638550      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:45.204640229    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:45.204641530      p = sp.run(
2023-09-05 17:17:45.204643268    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:45.204644673      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:45.204646269    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:45.204647813      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:45.204649315    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:45.204650706      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:45.204652221  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:45.204958465  [2023-09-05 17:17:45] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:45.204960877  future: <Task finished name='Task-27239' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:45.204961972  Traceback (most recent call last):
2023-09-05 17:17:45.204963185    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:45.204964250      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:45.204965396    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:45.204966366      p = sp.run(
2023-09-05 17:17:45.204967502    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:45.204968622      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:45.204969782    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:45.204970891      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:45.204972048    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:45.204973107      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:45.204974150  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:47.312875569  [2023-09-05 17:17:47] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:47.313047672  [2023-09-05 17:17:47] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:50.203428873  [2023-09-05 17:17:50] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:50.203433858  future: <Task finished name='Task-27243' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:50.203435602  Traceback (most recent call last):
2023-09-05 17:17:50.203437391    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:50.203438850      cpu_stats = get_cpu_stats()
2023-09-05 17:17:50.203440432    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:50.203442072      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:50.203443810    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:50.203445318      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:50.203446860    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:50.203448193      p = sp.run(
2023-09-05 17:17:50.203449772    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:50.203482573      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:50.203484437    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:50.203485903      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:50.203487375    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:50.203488801      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:50.203490268  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:50.203631634  [2023-09-05 17:17:50] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:50.203634559  future: <Task finished name='Task-27242' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:50.203635852  Traceback (most recent call last):
2023-09-05 17:17:50.203637433    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:50.203638768      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:50.203640256    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:50.203641469      p = sp.run(
2023-09-05 17:17:50.203642977    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:50.203644341      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:50.203645804    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:50.203647251      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:50.203648723    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:50.203650041      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:50.203651396  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:52.307045575  [2023-09-05 17:17:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:52.308073989  [2023-09-05 17:17:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garage. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:52.309132489  [2023-09-05 17:17:52] frigate.record                 WARNING : Unable to keep up with recording segments in cache for garden. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:52.311899620  [2023-09-05 17:17:52] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:52.311984897  [2023-09-05 17:17:52] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:17:55.202744121  [2023-09-05 17:17:55] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:55.202749144  future: <Task finished name='Task-27246' coro=<set_cpu_stats() done, defined at /opt/frigate/frigate/stats.py:112> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:55.202750885  Traceback (most recent call last):
2023-09-05 17:17:55.202752478    File "/opt/frigate/frigate/stats.py", line 114, in set_cpu_stats
2023-09-05 17:17:55.202753976      cpu_stats = get_cpu_stats()
2023-09-05 17:17:55.202755612    File "/opt/frigate/frigate/util.py", line 803, in get_cpu_stats
2023-09-05 17:17:55.202757268      docker_memlimit = get_docker_memlimit_bytes() / 1024
2023-09-05 17:17:55.202758954    File "/opt/frigate/frigate/util.py", line 774, in get_docker_memlimit_bytes
2023-09-05 17:17:55.202760418      if get_cgroups_version() == "cgroup2":
2023-09-05 17:17:55.202762050    File "/opt/frigate/frigate/util.py", line 747, in get_cgroups_version
2023-09-05 17:17:55.202763335      p = sp.run(
2023-09-05 17:17:55.202765013    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:55.202766436      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:55.202767951    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:55.202769322      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:55.202770966    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:55.202772347      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:55.202773760  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:55.203067896  [2023-09-05 17:17:55] asyncio                        ERROR   : Task exception was never retrieved
2023-09-05 17:17:55.203070586  future: <Task finished name='Task-27245' coro=<set_gpu_stats() done, defined at /opt/frigate/frigate/stats.py:120> exception=OSError(12, 'Cannot allocate memory')>
2023-09-05 17:17:55.203072038  Traceback (most recent call last):
2023-09-05 17:17:55.203073659    File "/opt/frigate/frigate/stats.py", line 184, in set_gpu_stats
2023-09-05 17:17:55.203075246      intel_usage = get_intel_gpu_stats()
2023-09-05 17:17:55.203076946    File "/opt/frigate/frigate/util.py", line 878, in get_intel_gpu_stats
2023-09-05 17:17:55.203078181      p = sp.run(
2023-09-05 17:17:55.203079819    File "/usr/lib/python3.9/subprocess.py", line 505, in run
2023-09-05 17:17:55.203081425      with Popen(*popenargs, **kwargs) as process:
2023-09-05 17:17:55.203083044    File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-09-05 17:17:55.203084867      self._execute_child(args, executable, preexec_fn, close_fds,
2023-09-05 17:17:55.203086533    File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
2023-09-05 17:17:55.203087828      self.pid = _posixsubprocess.fork_exec(
2023-09-05 17:17:55.203089184  OSError: [Errno 12] Cannot allocate memory
2023-09-05 17:17:57.310251334  [2023-09-05 17:17:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-09-05 17:17:57.313100475  [2023-09-05 17:17:57] frigate.record                 ERROR   : Error occurred when attempting to maintain recording cache
2023-09-05 17:17:57.313133500  [2023-09-05 17:17:57] frigate.record                 ERROR   : [Errno 12] Cannot allocate memory
2023-09-05 17:18:00.202688206  [2023-09-05 17:18:00] asyncio                        ERROR   : Task exception was never retrieved

ls -lah /dev/shm output:

# ls -lah /dev/shm
total 6.4M
drwxrwxrwt  3 root root  420 Sep  5 19:51 .
drwxr-xr-x 13 root root 3.2K Sep  5 02:59 ..
-rw-------  1 root root 300K Sep  5 03:00 back_porch
-rw-------  1 root root 450K Sep  5 19:51 back_porch1693961491.411047
-rw-------  1 root root 450K Sep  5 19:51 back_porch1693961491.432855
-rw-------  1 root root 3.0M Sep  5 03:00 birdseye
-rw-------  1 root root 300K Sep  5 03:00 front
-rw-------  1 root root 450K Sep  5 19:51 front1693961491.312198
-rw-------  1 root root 450K Sep  5 19:51 front1693961491.380324
-rw-------  1 root root 300K Sep  5 03:02 garage
-rw-------  1 root root 450K Sep  5 19:51 garage1693961491.156184
-rw-------  1 root root 450K Sep  5 19:51 garage1693961491.268624
-rw-------  1 root root 300K Sep  5 03:02 garden
-rw-------  1 root root 450K Sep  5 19:51 garden1693961491.353938
-rw-------  1 root root 450K Sep  5 19:51 garden1693961491.442802
-rw-r--r--  1 root root 1.6K Sep  5 02:59 go2rtc.yaml
drwxr-xr-x  5 root root  100 Sep  5 02:59 logs
-rw-------  1 root root  480 Sep  5 03:00 out-back_porch
-rw-------  1 root root  480 Sep  5 03:00 out-front
-rw-------  1 root root  480 Sep  5 03:02 out-garage
-rw-------  1 root root  480 Sep  5 03:02 out-garden

ps aux

# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0    212    68 ?        Ss   02:59   0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
root          15  0.0  0.0    212    68 ?        S    02:59   0:00 s6-supervise s6-linux-init-shutdownd
root          16  0.0  0.0    200     4 ?        Ss   02:59   0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B
root          24  0.0  0.0    212    64 ?        S    02:59   0:00 s6-supervise frigate
root          25  0.0  0.0    212    60 ?        S    02:59   0:00 s6-supervise s6rc-fdholder
root          26  0.0  0.0    212    64 ?        S    02:59   0:00 s6-supervise go2rtc-healthcheck
root          27  0.0  0.0    212    60 ?        S    02:59   0:00 s6-supervise nginx
root          28  0.0  0.0    212    64 ?        S    02:59   0:00 s6-supervise go2rtc-log
root          29  0.0  0.0    212    60 ?        S    02:59   0:00 s6-supervise s6rc-oneshot-runner
root          30  0.0  0.0    212    64 ?        S    02:59   0:00 s6-supervise go2rtc
root          31  0.0  0.0    212    68 ?        S    02:59   0:00 s6-supervise frigate-log
root          32  0.0  0.0    212    60 ?        S    02:59   0:00 s6-supervise nginx-log
root          41  0.0  0.0    520   156 ?        Ss   02:59   0:00 /package/admin/s6-2.11.3.0/command/s6-fdholderd -1 -i data/rules
root          42  0.0  0.0    188     4 ?        Ss   02:59   0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i da
nobody        80  0.0  0.0    280     4 ?        Ss   02:59   0:00 s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx
nobody        81  0.0  0.0    284   140 ?        Ss   02:59   0:02 s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate
nobody        82  0.0  0.0    276     4 ?        Ss   02:59   0:00 s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc
root          89  8.2  0.3 726600 29180 ?        Ssl  02:59  83:12 /config/go2rtc -config=/dev/shm/go2rtc.yaml
root         101  0.0  0.0   3892  3048 ?        Ss   02:59   0:01 bash ./run.user go2rtc-healthcheck
root         104  3.6 13.3 1062600316 1083240 ?  Ssl  02:59  36:43 python3 -u -m frigate
root         116  0.0  0.1 581680 15664 ?        Ss   02:59   0:00 nginx: master process nginx
root         125  0.0  0.6 886880 51072 ?        Sl   02:59   0:02 nginx: worker process
root         126  0.0  0.1 844872  8720 ?        Sl   02:59   0:00 nginx: worker process
root         127  0.0  0.3 864452 28340 ?        Sl   02:59   0:01 nginx: worker process
root         128  0.0  0.1 844476  8456 ?        Sl   02:59   0:00 nginx: worker process
root         283  0.0  0.9 593268 79888 ?        S    02:59   0:13 frigate.logger
root         289  0.5  0.1  14256 10680 ?        S    02:59   5:21 /usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(39)
root         290  2.8  1.3 1008468 109500 ?      Sl   02:59  28:25 frigate.detector.coral
root         292  5.3  1.3 1321720 112232 ?      Sl   02:59  54:25 frigate.output
root         298  1.8  1.2 1013276 99688 ?       Sl   02:59  18:55 frigate.process:garage
root         299  3.2  1.2 1013472 99932 ?       Sl   02:59  32:21 frigate.process:back_porch
root         300  6.2  1.2 1013316 99692 ?       Sl   02:59  63:17 frigate.process:garden
root         301  1.4  1.2 1012968 99564 ?       Sl   02:59  14:09 frigate.process:front
root         302  1.3  1.1 1009008 91056 ?       Sl   02:59  13:38 frigate.capture:garage
root         304  1.3  1.0 1013512 87124 ?       Sl   02:59  13:54 frigate.capture:back_porch
root         309  1.3  1.0 1009008 88100 ?       Sl   02:59  13:36 frigate.capture:garden
root         310  1.6  0.5 452492 48704 ?        Ssl  02:59  16:19 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         311  1.2  1.0 1013512 87056 ?       Sl   02:59  12:59 frigate.capture:front
root         317  2.3  0.4 167712 37416 ?        Ssl  02:59  24:13 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         322  2.4  0.4 167960 35456 ?        Ssl  02:59  24:14 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         326  2.5  0.4 167184 36816 ?        Ssl  02:59  25:34 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         329  1.6  0.6 453712 53256 ?        Ssl  02:59  16:29 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         331  3.5  0.4 169328 39776 ?        Sl   02:59  36:08 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.186/flv?port=1935&app=bcs
root         333  2.2  0.4 167140 36764 ?        Ssl  02:59  22:54 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         335  0.7  0.4 131128 36276 ?        S    02:59   7:47 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.187/flv?port=1935&app=bcs
root         336  0.7  0.3 131044 31828 ?        S    02:59   7:46 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.155/flv?port=1935&app=bcs
root         339  3.5  0.4 168296 38992 ?        Sl   02:59  35:27 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.187/flv?port=1935&app=bcs
root         340  4.1  0.5 168740 42128 ?        Sl   02:59  41:28 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.188/flv?port=1935&app=bcs
root         342  0.0  0.2 126820 20944 ?        Ss   02:59   0:00 ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q
root         343  3.4  0.4 168224 38528 ?        Sl   02:59  34:33 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.155/flv?port=1935&app=bcs
root         344  0.0  0.2 126820 19404 ?        Ss   02:59   0:00 ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q
root         345  0.0  0.2 126820 20968 ?        Ss   02:59   0:00 ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q
root         356  0.0  0.2 126820 19400 ?        Ss   02:59   0:00 ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q
root         357  5.0  1.0 247720 82984 ?        Ssl  02:59  50:56 ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1920x1080 -codec:v mpeg1vide
root         650  1.5  0.6 455132 55128 ?        Ssl  03:01  16:06 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root         652  0.7  0.4 131028 33832 ?        S    03:01   7:40 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.188/flv?port=1935&app=bcs
root       42279  1.6  0.7 453636 57080 ?        Ssl  13:12   6:42 ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwacc
root       42281  0.8  0.4 132012 34808 ?        S    13:12   3:13 ffmpeg -hide_banner -v error -fflags nobuffer -flags low_delay -i http://192.168.1.186/flv?port=1935&app=bcs
root       48871  0.0  0.0   2488   884 ?        S    19:49   0:00 sleep 30s
root       48872  0.0  0.0   2576   900 pts/0    Ss   19:49   0:00 sh
root       48881  0.0  0.0   6496  1504 pts/0    R+   19:49   0:00 ps aux

top -o %MEM:

Additional information, went digging through the containers logs as it seemed to be restarting out of the blue (after noticing the logs in the webgui seemed rather short), found this, pointing to python3.9?

2023-09-06 06:10:27.917715670  192.168.1.222 - - [06/Sep/2023:06:10:27 -0500] "GET /api/stats HTTP/1.1" 200 3373 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:10:34.069425205  [2023-09-06 06:10:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-06/11/front/10.23.mp4 in 0.18076801300048828 seconds.

2023-09-06 06:10:34.354868273  [2023-09-06 06:10:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-06/11/garage/10.22.mp4 in 0.19917607307434082 seconds.

2023-09-06 06:10:35.917883102  192.168.1.222 - - [06/Sep/2023:06:10:35 -0500] "GET /api/stats HTTP/1.1" 200 3375 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:10:43.807935756  Fatal Python error: Segmentation fault

2023-09-06 06:10:43.807940474  

2023-09-06 06:10:43.807942630  Thread 0x00007effdaffd6c0 (most recent call first):

2023-09-06 06:10:43.807991168    File "/usr/lib/python3.9/selectors.py", line 416 in select

2023-09-06 06:10:43.808064580    File "/usr/lib/python3.9/subprocess.py", line 1981 in _communicate

2023-09-06 06:10:43.808134736    File "/usr/lib/python3.9/subprocess.py", line 1134 in communicate

2023-09-06 06:10:43.808335489    File "/usr/lib/python3.9/subprocess.py", line 507 in run

2023-09-06 06:10:43.808337811    File "/opt/frigate/frigate/util.py", line 805 in get_cpu_stats

2023-09-06 06:10:43.808339771    File "/opt/frigate/frigate/stats.py", line 114 in set_cpu_stats

2023-09-06 06:10:43.808384657    File "/usr/lib/python3.9/asyncio/events.py", line 80 in _run

2023-09-06 06:10:43.808467704    File "/usr/lib/python3.9/asyncio/base_events.py", line 1890 in _run_once

2023-09-06 06:10:43.808547358    File "/usr/lib/python3.9/asyncio/base_events.py", line 596 in run_forever

2023-09-06 06:10:43.808640728    File "/usr/lib/python3.9/asyncio/base_events.py", line 629 in run_until_complete

2023-09-06 06:10:43.808715453    File "/opt/frigate/frigate/stats.py", line 108 in get_processing_stats

2023-09-06 06:10:43.808787149    File "/opt/frigate/frigate/stats.py", line 240 in stats_snapshot

2023-09-06 06:10:43.808844620    File "/opt/frigate/frigate/http.py", line 827 in stats

2023-09-06 06:10:43.808949576    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799 in dispatch_request

2023-09-06 06:10:43.809058178    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823 in full_dispatch_request

2023-09-06 06:10:43.809153662    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2529 in wsgi_app

2023-09-06 06:10:43.809242340    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2552 in __call__

2023-09-06 06:10:43.809341138    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 325 in execute

2023-09-06 06:10:43.809441423    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 364 in run_wsgi

2023-09-06 06:10:43.809523081    File "/usr/lib/python3.9/http/server.py", line 415 in handle_one_request

2023-09-06 06:10:43.809587331    File "/usr/lib/python3.9/http/server.py", line 427 in handle

2023-09-06 06:10:43.809684428    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 392 in handle

2023-09-06 06:10:43.809751711    File "/usr/lib/python3.9/socketserver.py", line 720 in __init__

2023-09-06 06:10:43.809827839    File "/usr/lib/python3.9/socketserver.py", line 360 in finish_request

2023-09-06 06:10:43.809917404    File "/usr/lib/python3.9/socketserver.py", line 650 in process_request_thread

2023-09-06 06:10:43.809977290    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.810050580    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.810114876    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.810116297  

2023-09-06 06:10:43.810130496  Thread 0x00007efff1ffb6c0 (most recent call first):

2023-09-06 06:10:43.810177510    File "/usr/lib/python3.9/threading.py", line 312 in wait

2023-09-06 06:10:43.810252879    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed

2023-09-06 06:10:43.810312551    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.810388060    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.810455887    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.810457366  

2023-09-06 06:10:43.810471211  Thread 0x00007efff3fff6c0 (most recent call first):

2023-09-06 06:10:43.810528382    File "/usr/lib/python3.9/threading.py", line 312 in wait

2023-09-06 06:10:43.810605848    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed

2023-09-06 06:10:43.810682497    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.810756904    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.810819450    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.810831820  

2023-09-06 06:10:43.810833738  Thread 0x00007efff2ffd6c0 (most recent call first):

2023-09-06 06:10:43.810883241    File "/usr/lib/python3.9/threading.py", line 312 in wait

2023-09-06 06:10:43.810958481    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed

2023-09-06 06:10:43.811012738    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.811083926    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.811148198    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.811149711  

2023-09-06 06:10:43.811163875  Thread 0x00007efff37fe6c0 (most recent call first):

2023-09-06 06:10:43.811207555    File "/usr/lib/python3.9/threading.py", line 312 in wait

2023-09-06 06:10:43.811282131    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed

2023-09-06 06:10:43.811335949    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.811406912    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.811471471    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.811485287  

2023-09-06 06:10:43.811487261  Thread 0x00007f0014ff96c0 (most recent call first):

2023-09-06 06:10:43.811558372    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait

2023-09-06 06:10:43.811643950    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait

2023-09-06 06:10:43.811705007    File "/opt/frigate/frigate/watchdog.py", line 24 in run

2023-09-06 06:10:43.811777037    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.811840738    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.811842187  

2023-09-06 06:10:43.811856267  Thread 0x00007f00157fa6c0 (most recent call first):

2023-09-06 06:10:43.811924719    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait

2023-09-06 06:10:43.812010052    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait

2023-09-06 06:10:43.812059848    File "/opt/frigate/frigate/stats.py", line 285 in run

2023-09-06 06:10:43.812131980    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.812192273    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.812206372  

2023-09-06 06:10:43.812208309  Thread 0x00007f0015ffb6c0 (most recent call first):

2023-09-06 06:10:43.812279819    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait

2023-09-06 06:10:43.812362950    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait

2023-09-06 06:10:43.812420253    File "/opt/frigate/frigate/record.py", line 626 in run

2023-09-06 06:10:43.812494620    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.812557601    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.812559017  

2023-09-06 06:10:43.812571212  Current thread 0x00007f00167fc6c0 (most recent call first):

2023-09-06 06:10:43.812620760    File "/usr/lib/python3.9/subprocess.py", line 1919 in _wait

2023-09-06 06:10:43.812726544    File "/usr/lib/python3.9/subprocess.py", line 1189 in wait

2023-09-06 06:10:43.812742709    File "/usr/lib/python3.9/subprocess.py", line 2007 in _communicate

2023-09-06 06:10:43.812813074    File "/usr/lib/python3.9/subprocess.py", line 1134 in communicate

2023-09-06 06:10:43.812867410    File "/usr/lib/python3.9/subprocess.py", line 507 in run

2023-09-06 06:10:43.812933737    File "/opt/frigate/frigate/record.py", line 161 in move_files

2023-09-06 06:10:43.812991015    File "/opt/frigate/frigate/record.py", line 397 in run

2023-09-06 06:10:43.813062265    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.813126089    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.813127594  

2023-09-06 06:10:43.813142559  Thread 0x00007f0016ffd6c0 (most recent call first):

2023-09-06 06:10:43.813209428    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait

2023-09-06 06:10:43.813293227    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait

2023-09-06 06:10:43.813350679    File "/opt/frigate/frigate/events.py", line 305 in run

2023-09-06 06:10:43.813421692    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.813485885    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.813487392  

2023-09-06 06:10:43.813499244  Thread 0x00007f00177fe6c0 (most recent call first):

2023-09-06 06:10:43.813552184    File "/usr/lib/python3.9/selectors.py", line 416 in select

2023-09-06 06:10:43.813633593    File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait

2023-09-06 06:10:43.813715870    File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll

2023-09-06 06:10:43.813794897    File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll

2023-09-06 06:10:43.813869184    File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get

2023-09-06 06:10:43.813923270    File "/opt/frigate/frigate/events.py", line 70 in run

2023-09-06 06:10:43.813995746    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.814059491    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.814072896  

2023-09-06 06:10:43.814074772  Thread 0x00007f0017fff6c0 (most recent call first):

2023-09-06 06:10:43.814149462    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait

2023-09-06 06:10:43.814231318    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait

2023-09-06 06:10:43.814292029    File "/opt/frigate/frigate/storage.py", line 181 in run

2023-09-06 06:10:43.814365729    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.814430030    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.814431518  

2023-09-06 06:10:43.814443760  Thread 0x00007f002ce5b6c0 (most recent call first):

2023-09-06 06:10:43.814493081    File "/usr/lib/python3.9/selectors.py", line 416 in select

2023-09-06 06:10:43.814579535    File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait

2023-09-06 06:10:43.814667949    File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll

2023-09-06 06:10:43.814745222    File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll

2023-09-06 06:10:43.814816683    File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get

2023-09-06 06:10:43.814885136    File "/opt/frigate/frigate/object_processing.py", line 904 in run

2023-09-06 06:10:43.814958293    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.815020926    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.815034669  

2023-09-06 06:10:43.815036540  Thread 0x00007f00350be6c0 (most recent call first):

2023-09-06 06:10:43.815086614    File "/usr/lib/python3.9/selectors.py", line 416 in select

2023-09-06 06:10:43.815160647    File "/usr/lib/python3.9/socketserver.py", line 232 in serve_forever

2023-09-06 06:10:43.815216556    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.815290796    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.815354883    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.815368470  

2023-09-06 06:10:43.815370351  Thread 0x00007f00341816c0 (most recent call first):

2023-09-06 06:10:43.815448473    File "/usr/local/lib/python3.9/dist-packages/ws4py/manager.py", line 141 in poll

2023-09-06 06:10:43.815537846    File "/usr/local/lib/python3.9/dist-packages/ws4py/manager.py", line 312 in run

2023-09-06 06:10:43.815608819    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.815672018    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.815673515  

2023-09-06 06:10:43.815687740  Thread 0x00007f004c1ff6c0 (most recent call first):

2023-09-06 06:10:43.815773688    File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1150 in _loop

2023-09-06 06:10:43.815877780    File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1756 in loop_forever

2023-09-06 06:10:43.815985542    File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 3591 in _thread_main

2023-09-06 06:10:43.816038749    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.816120041    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.816182440    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.816184037  

2023-09-06 06:10:43.816197915  Thread 0x00007f004b9fe6c0 (most recent call first):

2023-09-06 06:10:43.816241414    File "/usr/lib/python3.9/threading.py", line 312 in wait

2023-09-06 06:10:43.816294905    File "/usr/lib/python3.9/queue.py", line 171 in get

2023-09-06 06:10:43.816390934    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 159 in loop

2023-09-06 06:10:43.816483837    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 136 in run

2023-09-06 06:10:43.816577331    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 266 in run

2023-09-06 06:10:43.816631032    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.816758524    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.816774164    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.816775569  

2023-09-06 06:10:43.816786537  Thread 0x00007f00491fd6c0 (most recent call first):

2023-09-06 06:10:43.816836637    File "/usr/lib/python3.9/threading.py", line 312 in wait

2023-09-06 06:10:43.816932370    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed

2023-09-06 06:10:43.816996620    File "/usr/lib/python3.9/threading.py", line 892 in run

2023-09-06 06:10:43.817073912    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner

2023-09-06 06:10:43.817139574    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap

2023-09-06 06:10:43.817141040  

2023-09-06 06:10:43.817161857  Thread 0x00007f0050596740 (most recent call first):

2023-09-06 06:10:43.817202360    File "/usr/lib/python3.9/selectors.py", line 416 in select

2023-09-06 06:10:43.817279007    File "/usr/lib/python3.9/socketserver.py", line 232 in serve_forever

2023-09-06 06:10:43.817385781    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 804 in serve_forever

2023-09-06 06:10:43.817486259    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 1107 in run_simple

2023-09-06 06:10:43.817585992    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1191 in run

2023-09-06 06:10:43.817640025    File "/opt/frigate/frigate/app.py", line 403 in start

2023-09-06 06:10:43.817710946    File "/opt/frigate/frigate/__main__.py", line 16 in <module>

2023-09-06 06:10:43.817810287    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code

2023-09-06 06:10:43.817885658    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main

2023-09-06 06:10:43.997246716  2023/09/06 06:10:43 [error] 124#124: *1623 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.222, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001/stats", host: "192.168.69.69:5000"

2023-09-06 06:10:43.997367726  192.168.1.222 - - [06/Sep/2023:06:10:43 -0500] "GET /api/stats HTTP/1.1" 502 157 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:10:44.334949881  [INFO] Service Frigate exited with code 256 (by signal 11)

s6-rc: info: service legacy-services: stopping

s6-rc: info: service legacy-services successfully stopped

s6-rc: info: service nginx: stopping

s6-rc: info: service go2rtc-healthcheck: stopping

2023-09-06 06:10:44.353911233  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)

s6-rc: info: service go2rtc-healthcheck successfully stopped

2023-09-06 06:10:44.411409923  [INFO] Service NGINX exited with code 0 (by signal 0)

s6-rc: info: service nginx successfully stopped

s6-rc: info: service nginx-log: stopping

s6-rc: info: service frigate: stopping

s6-rc: info: service frigate successfully stopped

s6-rc: info: service go2rtc: stopping

s6-rc: info: service frigate-log: stopping

s6-rc: info: service nginx-log successfully stopped

2023-09-06 06:10:44.416159224  exit with signal: terminated

s6-rc: info: service frigate-log successfully stopped

2023-09-06 06:10:44.433867044  [INFO] The go2rtc service exited with code 0 (by signal 0)

s6-rc: info: service go2rtc successfully stopped

s6-rc: info: service go2rtc-log: stopping

s6-rc: info: service go2rtc-log successfully stopped

s6-rc: info: service log-prepare: stopping

s6-rc: info: service s6rc-fdholder: stopping

s6-rc: info: service log-prepare successfully stopped

s6-rc: info: service legacy-cont-init: stopping

s6-rc: info: service s6rc-fdholder successfully stopped

s6-rc: info: service legacy-cont-init successfully stopped

s6-rc: info: service fix-attrs: stopping

s6-rc: info: service fix-attrs successfully stopped

s6-rc: info: service s6rc-oneshot-runner: stopping

s6-rc: info: service s6rc-oneshot-runner successfully stopped

s6-rc: info: service s6rc-fdholder: starting

s6-rc: info: service s6rc-oneshot-runner: starting

s6-rc: info: service s6rc-oneshot-runner successfully started

s6-rc: info: service fix-attrs: starting

s6-rc: info: service s6rc-fdholder successfully started

s6-rc: info: service fix-attrs successfully started

s6-rc: info: service legacy-cont-init: starting

s6-rc: info: service legacy-cont-init successfully started

s6-rc: info: service log-prepare: starting

s6-rc: info: service log-prepare successfully started

s6-rc: info: service nginx-log: starting

s6-rc: info: service go2rtc-log: starting

s6-rc: info: service frigate-log: starting

s6-rc: info: service frigate-log successfully started

s6-rc: info: service nginx-log successfully started

s6-rc: info: service go2rtc-log successfully started

s6-rc: info: service go2rtc: starting

s6-rc: info: service go2rtc successfully started

s6-rc: info: service go2rtc-healthcheck: starting

s6-rc: info: service frigate: starting

s6-rc: info: service go2rtc-healthcheck successfully started

s6-rc: info: service frigate successfully started

s6-rc: info: service nginx: starting

2023-09-06 06:10:49.504921783  [INFO] Preparing go2rtc config...

s6-rc: info: service nginx successfully started

s6-rc: info: service legacy-services: starting

2023-09-06 06:10:49.509812339  [INFO] Starting Frigate...

2023-09-06 06:10:49.511198892  [INFO] Starting NGINX...

s6-rc: info: service legacy-services successfully started

2023-09-06 06:10:49.858653279  [WARN] Using go2rtc binary from '/config/go2rtc' instead of the embedded one

2023-09-06 06:10:49.858658730  [INFO] Starting go2rtc...

2023-09-06 06:10:49.953395695  06:10:49.953 INF go2rtc version 1.6.2 linux/amd64

2023-09-06 06:10:49.953915836  06:10:49.953 INF [api] listen addr=0.0.0.0:1984

2023-09-06 06:10:49.954583404  06:10:49.954 INF [rtsp] listen addr=0.0.0.0:8554

2023-09-06 06:10:49.955384641  06:10:49.955 INF [webrtc] listen addr=0.0.0.0:8555/tcp

2023-09-06 06:10:49.956235962  06:10:49.956 INF [srtp] listen addr=0.0.0.0:8443

2023-09-06 06:10:50.654560134  [2023-09-06 06:10:50] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)

2023-09-06 06:10:50.691670255  [2023-09-06 06:10:50] peewee_migrate                 INFO    : Starting migrations

2023-09-06 06:10:50.709660971  [2023-09-06 06:10:50] peewee_migrate                 INFO    : There is nothing to migrate

2023-09-06 06:10:50.732778245  [2023-09-06 06:10:50] detector.coral                 INFO    : Starting detection process: 289

2023-09-06 06:10:50.736979554  [2023-09-06 06:10:50] frigate.app                    INFO    : Output process started: 291

2023-09-06 06:10:50.745581400  [2023-09-06 06:10:50] frigate.app                    INFO    : Camera processor started for garage: 295

2023-09-06 06:10:50.749849744  [2023-09-06 06:10:50] frigate.app                    INFO    : Camera processor started for back_porch: 298

2023-09-06 06:10:50.757042892  [2023-09-06 06:10:50] frigate.app                    INFO    : Camera processor started for garden: 299

2023-09-06 06:10:50.761261442  [2023-09-06 06:10:50] frigate.app                    INFO    : Camera processor started for front: 300

2023-09-06 06:10:50.765479520  [2023-09-06 06:10:50] frigate.app                    INFO    : Capture process started for garage: 301

2023-09-06 06:10:50.769973519  [2023-09-06 06:10:50] frigate.app                    INFO    : Capture process started for back_porch: 302

2023-09-06 06:10:50.774193647  [2023-09-06 06:10:50] frigate.app                    INFO    : Capture process started for garden: 305

2023-09-06 06:10:50.778969278  [2023-09-06 06:10:50] frigate.app                    INFO    : Capture process started for front: 309

2023-09-06 06:10:50.883954677  [2023-09-06 06:10:50] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci

2023-09-06 06:10:50.888686232  [2023-09-06 06:10:50] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

2023-09-06 06:10:56.907384361  192.168.1.222 - - [06/Sep/2023:06:10:56 -0500] "GET /api/stats HTTP/1.1" 200 3359 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:10:59.509572736  [INFO] Starting go2rtc healthcheck service...

2023-09-06 06:11:04.906240968  192.168.1.222 - - [06/Sep/2023:06:11:04 -0500] "GET /api/stats HTTP/1.1" 200 3366 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:11:12.903992275  192.168.1.222 - - [06/Sep/2023:06:11:12 -0500] "GET /api/stats HTTP/1.1" 200 3368 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:11:20.911902692  192.168.1.222 - - [06/Sep/2023:06:11:20 -0500] "GET /api/stats HTTP/1.1" 200 3369 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:11:28.908087535  192.168.1.222 - - [06/Sep/2023:06:11:28 -0500] "GET /api/stats HTTP/1.1" 200 3367 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:11:31.482971149  [2023-09-06 06:11:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-06/11/garden/11.16.mp4 in 0.2458961009979248 seconds.

2023-09-06 06:11:36.908895902  192.168.1.222 - - [06/Sep/2023:06:11:36 -0500] "GET /api/stats HTTP/1.1" 200 3369 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:11:41.599615520  [2023-09-06 06:11:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-06/11/garden/11.26.mp4 in 0.2867710590362549 seconds.

2023-09-06 06:11:44.914044534  192.168.1.222 - - [06/Sep/2023:06:11:44 -0500] "GET /api/stats HTTP/1.1" 200 3369 "-" "HomeAssistant/2023.8.4 aiohttp/3.8.5 Python/3.11" "-"

2023-09-06 06:11:50.952743994  [2023-09-06 06:11:50] frigate.record                 DEBUG   : Start expire recordings (new).

2023-09-06 06:11:50.953020923  [2023-09-06 06:11:50] frigate.record                 DEBUG   : Start deleted cameras.

2023-09-06 06:11:50.963761285  [2023-09-06 06:11:50] frigate.record                 DEBUG   : Expiring 0 recordings

2023-09-06 06:11:50.964112367  [2023-09-06 06:11:50] frigate.record                 DEBUG   : End deleted cameras.

2023-09-06 06:11:50.964115909  [2023-09-06 06:11:50] frigate.record                 DEBUG   : Start all cameras.

2023-09-06 06:11:50.964168406  [2023-09-06 06:11:50] frigate.record                 DEBUG   : Start camera: garage.

2023-09-06 06:11:51.563281785  [2023-09-06 06:11:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-06/11/garden/11.38.mp4 in 0.24464797973632812 seconds.

2023-09-06 06:11:51.573965318  [2023-09-06 06:11:51] frigate.record                 DEBUG   : Expiring 38 recordings

2023-09-06 06:11:51.580415231  [2023-09-06 06:11:51] frigate.record                 DEBUG   : End camera: garage.

2023-09-06 06:11:51.580525417  [2023-09-06 06:11:51] frigate.record                 DEBUG   : Start camera: back_porch.

2023-09-06 06:11:52.168118017  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Expiring 72 recordings

2023-09-06 06:11:52.174962902  [2023-09-06 06:11:52] frigate.record                 DEBUG   : End camera: back_porch.

2023-09-06 06:11:52.175045121  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Start camera: garden.

2023-09-06 06:11:52.522943197  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Expiring 58 recordings

2023-09-06 06:11:52.530383168  [2023-09-06 06:11:52] frigate.record                 DEBUG   : End camera: garden.

2023-09-06 06:11:52.530528074  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Start camera: front.

2023-09-06 06:11:52.720612799  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Expiring 7 recordings

2023-09-06 06:11:52.727385851  [2023-09-06 06:11:52] frigate.record                 DEBUG   : End camera: front.

2023-09-06 06:11:52.727483315  [2023-09-06 06:11:52] frigate.record                 DEBUG   : End all cameras.

2023-09-06 06:11:52.727564737  [2023-09-06 06:11:52] frigate.record                 DEBUG   : End expire recordings (new).

2023-09-06 06:11:52.727653443  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Start expire files (legacy).

2023-09-06 06:11:52.759618445  [2023-09-06 06:11:52] frigate.record                 DEBUG   : Oldest recording in the db: 1691584742.060758

I'm going to close this for now as my initial issue was fixed, now I'm just dealing with weirdness. I'm just going to back up, recreate and then burn the old VM. Not sure what's going on, occasionally the container switches the storage to a directory on the SSD instead of the SkyHawks in my NAS, errors below:

2023-09-08 08:10:51.459975895  [INFO] Starting Frigate...
2023-09-08 08:10:53.224857632  [2023-09-08 08:10:53] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-08 08:10:53.261613161  [2023-09-08 08:10:53] frigate.app                    INFO    : Creating directory: /media/frigate/recordings
2023-09-08 08:10:53.262105200  [2023-09-08 08:10:53] frigate.app                    INFO    : Creating directory: /media/frigate/clips
2023-09-08 08:10:53.265373071  [2023-09-08 08:10:53] peewee_migrate                 INFO    : Starting migrations
2023-09-08 08:10:53.285036492  [2023-09-08 08:10:53] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-08 08:10:53.303701884  [2023-09-08 08:10:53] frigate.app                    INFO    : Output process started: 292
2023-09-08 08:10:53.310265834  [2023-09-08 08:10:53] frigate.app                    INFO    : Camera processor started for garage: 296
2023-09-08 08:10:53.311260076  [2023-09-08 08:10:53] detector.coral                 INFO    : Starting detection process: 291
2023-09-08 08:10:53.311264576  [2023-09-08 08:10:53] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2023-09-08 08:10:53.316159487  [2023-09-08 08:10:53] frigate.app                    INFO    : Camera processor started for back_porch: 298
2023-09-08 08:10:53.321128386  [2023-09-08 08:10:53] frigate.app                    INFO    : Camera processor started for garden: 301
2023-09-08 08:10:53.325031699  [2023-09-08 08:10:53] frigate.app                    INFO    : Camera processor started for front: 302
2023-09-08 08:10:53.328955505  [2023-09-08 08:10:53] frigate.app                    INFO    : Capture process started for garage: 303
2023-09-08 08:10:53.332862792  [2023-09-08 08:10:53] frigate.app                    INFO    : Capture process started for back_porch: 304
2023-09-08 08:10:53.336850410  [2023-09-08 08:10:53] frigate.app                    INFO    : Capture process started for garden: 306
2023-09-08 08:10:53.341119929  [2023-09-08 08:10:53] frigate.app                    INFO    : Capture process started for front: 309
2023-09-08 08:10:53.466834007  [2023-09-08 08:10:53] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-08 08:10:56.734016628  [2023-09-08 08:10:56] frigate.comms.mqtt             ERROR   : Unable to publish to garden/all: client is not connected
2023-09-08 08:10:56.940678696  [2023-09-08 08:10:56] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:10:56.940684557  [2023-09-08 08:10:56] frigate.comms.mqtt             ERROR   : Unable to publish to front_porch_area/all: client is not connected
2023-09-08 08:10:58.641104809  [2023-09-08 08:10:58] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-08 08:10:58.641348120  [2023-09-08 08:10:58] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-08 08:10:59.130616538  [2023-09-08 08:10:59] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/all: client is not connected
2023-09-08 08:11:13.452405662  [2023-09-08 08:11:13] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-08 08:11:13.452583952  [2023-09-08 08:11:13] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 08:11:13.452732948  [2023-09-08 08:11:13] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Invalid data found when processing input
2023-09-08 08:11:13.461205759  [2023-09-08 08:11:13] ffmpeg.back_porch.record       ERROR   : rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac: Invalid data found when processing input
2023-09-08 08:11:13.462293617  [2023-09-08 08:11:13] watchdog.back_porch            INFO    : Terminating the existing ffmpeg process...
2023-09-08 08:11:13.466233792  [2023-09-08 08:11:13] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 08:11:13.466994866  [2023-09-08 08:11:13] ffmpeg.garage.record           ERROR   : rtsp://127.0.0.1:8554/garage?video=copy&audio=aac: Invalid data found when processing input
2023-09-08 08:11:13.466998420  [2023-09-08 08:11:13] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-09-08 08:11:13.467000544  [2023-09-08 08:11:13] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 08:11:16.805753486  [2023-09-08 08:11:16] frigate.comms.mqtt             ERROR   : Unable to publish to garage/all: client is not connected
2023-09-08 08:11:53.720557659  [2023-09-08 08:11:53] frigate.record                 DEBUG   : Start expire recordings (new).
2023-09-08 08:11:53.720849512  [2023-09-08 08:11:53] frigate.record                 DEBUG   : Start deleted cameras.
2023-09-08 08:11:53.781223349  [2023-09-08 08:11:53] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-08 08:11:53.784181541  [2023-09-08 08:11:53] frigate.record                 DEBUG   : End deleted cameras.
2023-09-08 08:11:53.784285601  [2023-09-08 08:11:53] frigate.record                 DEBUG   : Start all cameras.
2023-09-08 08:11:53.784406252  [2023-09-08 08:11:53] frigate.record                 DEBUG   : Start camera: garage.
2023-09-08 08:11:55.497536790  [2023-09-08 08:11:55] frigate.record                 DEBUG   : Expiring 40 recordings
2023-09-08 08:11:55.509129753  [2023-09-08 08:11:55] frigate.record                 DEBUG   : End camera: garage.
2023-09-08 08:11:55.509279811  [2023-09-08 08:11:55] frigate.record                 DEBUG   : Start camera: back_porch.
2023-09-08 08:11:55.737225838  [2023-09-08 08:11:55] frigate.record                 DEBUG   : Expiring 239 recordings
2023-09-08 08:11:55.752770835  [2023-09-08 08:11:55] frigate.record                 DEBUG   : End camera: back_porch.
2023-09-08 08:11:55.752775425  [2023-09-08 08:11:55] frigate.record                 DEBUG   : Start camera: garden.
2023-09-08 08:11:55.890260507  [2023-09-08 08:11:55] frigate.record                 DEBUG   : Expiring 234 recordings
2023-09-08 08:11:55.896663085  [2023-09-08 08:11:55] frigate.record                 DEBUG   : End camera: garden.
2023-09-08 08:11:55.896743610  [2023-09-08 08:11:55] frigate.record                 DEBUG   : Start camera: front.
2023-09-08 08:11:56.303743357  [2023-09-08 08:11:56] frigate.record                 DEBUG   : Expiring 66 recordings
2023-09-08 08:11:56.311659327  [2023-09-08 08:11:56] frigate.record                 DEBUG   : End camera: front.
2023-09-08 08:11:56.311770003  [2023-09-08 08:11:56] frigate.record                 DEBUG   : End all cameras.
2023-09-08 08:11:56.311915120  [2023-09-08 08:11:56] frigate.record                 DEBUG   : End expire recordings (new).
2023-09-08 08:11:56.312001107  [2023-09-08 08:11:56] frigate.record                 DEBUG   : Start expire files (legacy).
2023-09-08 08:11:56.360154973  [2023-09-08 08:11:56] frigate.record                 WARNING : Unable to find file from recordings database: /media/frigate/recordings/2023-08-11/13/garage/20.26.mp4
2023-09-08 08:12:07.455415585  [2023-09-08 08:12:07] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:12:19.462275081  [2023-09-08 08:12:19] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:12:21.002892899  [2023-09-08 08:12:21] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:12:29.301483968  [2023-09-08 08:12:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/12.18.mp4 in 0.2070608139038086 seconds.
2023-09-08 08:12:29.631614838  [2023-09-08 08:12:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/12.17.mp4 in 0.16846203804016113 seconds.
2023-09-08 08:12:51.840000952  [2023-09-08 08:12:51] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:12:52.741014463  [2023-09-08 08:12:52] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:13:11.155170844  [2023-09-08 08:13:11] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:13:41.793928946  [2023-09-08 08:13:41] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:13:49.541335851  [2023-09-08 08:13:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/13.37.mp4 in 0.15833592414855957 seconds.
2023-09-08 08:13:57.541735435  [2023-09-08 08:13:57] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:13:59.271832615  [2023-09-08 08:13:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/13.48.mp4 in 0.18085908889770508 seconds.
2023-09-08 08:13:59.751568190  [2023-09-08 08:13:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/13.47.mp4 in 0.1679210662841797 seconds.
2023-09-08 08:14:03.470601983  [2023-09-08 08:14:03] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:14:09.232943335  [2023-09-08 08:14:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/13.58.mp4 in 0.18027305603027344 seconds.
2023-09-08 08:14:09.547962856  [2023-09-08 08:14:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/13.57.mp4 in 0.16673898696899414 seconds.
2023-09-08 08:14:14.085319816  [2023-09-08 08:14:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/13.58.mp4 in 0.17767596244812012 seconds.
2023-09-08 08:14:14.866814688  [2023-09-08 08:14:14] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:14:19.364372926  [2023-09-08 08:14:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/14.09.mp4 in 0.17144083976745605 seconds.
2023-09-08 08:14:19.687227800  [2023-09-08 08:14:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/14.07.mp4 in 0.16900992393493652 seconds.
2023-09-08 08:14:29.236574168  [2023-09-08 08:14:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/14.18.mp4 in 0.1902470588684082 seconds.
2023-09-08 08:14:29.520309715  [2023-09-08 08:14:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/14.17.mp4 in 0.15291285514831543 seconds.
2023-09-08 08:14:39.134983608  [2023-09-08 08:14:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/14.28.mp4 in 0.146622896194458 seconds.
2023-09-08 08:14:39.436595488  [2023-09-08 08:14:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/14.28.mp4 in 0.15073609352111816 seconds.
2023-09-08 08:14:43.210398786  [2023-09-08 08:14:43] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:14:43.221750373  [2023-09-08 08:14:43] frigate.comms.mqtt             ERROR   : Unable to publish to front/bus/snapshot: client is not connected
2023-09-08 08:14:43.221862476  [2023-09-08 08:14:43] frigate.comms.mqtt             ERROR   : Unable to publish to front/bus: client is not connected
2023-09-08 08:14:43.221968061  [2023-09-08 08:14:43] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:14:46.543199103  [2023-09-08 08:14:46] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:14:49.189326852  [2023-09-08 08:14:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/14.38.mp4 in 0.1659390926361084 seconds.
2023-09-08 08:14:49.522189052  [2023-09-08 08:14:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/14.38.mp4 in 0.17733478546142578 seconds.
2023-09-08 08:14:53.461687968  [2023-09-08 08:14:53] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:14:53.490091641  [2023-09-08 08:14:53] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:14:53.490096876  [2023-09-08 08:14:53] frigate.comms.mqtt             ERROR   : Unable to publish to front/bus: client is not connected
2023-09-08 08:14:53.494238252  [2023-09-08 08:14:53] frigate.comms.mqtt             ERROR   : Unable to publish to front/bus/snapshot: client is not connected
2023-09-08 08:14:59.146437485  [2023-09-08 08:14:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/14.48.mp4 in 0.15612196922302246 seconds.
2023-09-08 08:14:59.470761883  [2023-09-08 08:14:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/14.48.mp4 in 0.1685349941253662 seconds.
2023-09-08 08:15:09.182535890  [2023-09-08 08:15:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/14.58.mp4 in 0.15502500534057617 seconds.
2023-09-08 08:15:09.561866440  [2023-09-08 08:15:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/14.58.mp4 in 0.19624996185302734 seconds.
2023-09-08 08:15:18.570800545  [2023-09-08 08:15:18] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:15:19.210042718  [2023-09-08 08:15:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/15.08.mp4 in 0.18058204650878906 seconds.
2023-09-08 08:15:19.561588220  [2023-09-08 08:15:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/15.08.mp4 in 0.18723106384277344 seconds.
2023-09-08 08:15:29.192078283  [2023-09-08 08:15:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/15.18.mp4 in 0.173065185546875 seconds.
2023-09-08 08:15:29.548254490  [2023-09-08 08:15:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/15.18.mp4 in 0.17427492141723633 seconds.
2023-09-08 08:15:35.707649939  [2023-09-08 08:15:35] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:15:39.511690289  [2023-09-08 08:15:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/15.27.mp4 in 0.1770610809326172 seconds.
2023-09-08 08:15:49.511987349  [2023-09-08 08:15:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/15.37.mp4 in 0.1673109531402588 seconds.
2023-09-08 08:15:50.397214507  [2023-09-08 08:15:50] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:15:50.723190033  [2023-09-08 08:15:50] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:15:56.709032269  [2023-09-08 08:15:56] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:15:59.233225935  [2023-09-08 08:15:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/15.48.mp4 in 0.20029091835021973 seconds.
2023-09-08 08:15:59.580743097  [2023-09-08 08:15:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/15.48.mp4 in 0.18047094345092773 seconds.
2023-09-08 08:15:59.876268541  [2023-09-08 08:15:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/15.47.mp4 in 0.15121817588806152 seconds.
2023-09-08 08:16:09.008024212  [2023-09-08 08:16:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/15.56.mp4 in 0.13371706008911133 seconds.
2023-09-08 08:16:09.701909786  [2023-09-08 08:16:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/15.57.mp4 in 0.17664694786071777 seconds.
2023-09-08 08:16:19.585669016  [2023-09-08 08:16:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/16.07.mp4 in 0.18274283409118652 seconds.
2023-09-08 08:16:21.728501461  [2023-09-08 08:16:21] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:16:22.283848190  [2023-09-08 08:16:22] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:16:23.404563824  [2023-09-08 08:16:23] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:16:28.179806274  [2023-09-08 08:16:28] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:16:40.562460569  [2023-09-08 08:16:40] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:17:05.378870066  [2023-09-08 08:17:05] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:17:09.582410697  [2023-09-08 08:17:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/16.57.mp4 in 0.1924440860748291 seconds.
2023-09-08 08:17:22.824335620  [2023-09-08 08:17:22] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:17:23.715512563  [2023-09-08 08:17:23] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:17:25.991164983  [2023-09-08 08:17:25] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:17:29.212318858  [2023-09-08 08:17:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/17.18.mp4 in 0.1811070442199707 seconds.
2023-09-08 08:17:29.542082897  [2023-09-08 08:17:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/17.17.mp4 in 0.1757669448852539 seconds.
2023-09-08 08:17:35.686423825  [2023-09-08 08:17:35] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:17:54.204613097  [2023-09-08 08:17:54] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:17:56.237213419  [2023-09-08 08:17:56] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:18:00.319627666  [2023-09-08 08:18:00] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:18:09.536265386  [2023-09-08 08:18:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/17.57.mp4 in 0.16284990310668945 seconds.
2023-09-08 08:18:10.823113692  [2023-09-08 08:18:10] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:18:14.027373459  [2023-09-08 08:18:14] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:18:19.164909326  [2023-09-08 08:18:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/18.08.mp4 in 0.16032099723815918 seconds.
2023-09-08 08:18:19.530792192  [2023-09-08 08:18:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/18.07.mp4 in 0.17742109298706055 seconds.
2023-09-08 08:18:24.028582351  [2023-09-08 08:18:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/18.07.mp4 in 0.15013384819030762 seconds.
2023-09-08 08:18:29.702854781  [2023-09-08 08:18:29] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:18:41.093891613  [2023-09-08 08:18:41] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:18:41.529614168  [2023-09-08 08:18:41] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:18:44.439086068  [2023-09-08 08:18:44] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:18:44.509494983  [2023-09-08 08:18:44] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:18:49.463474562  [2023-09-08 08:18:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/18.37.mp4 in 0.17044997215270996 seconds.
2023-09-08 08:19:09.507660834  [2023-09-08 08:19:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/18.57.mp4 in 0.1545858383178711 seconds.
2023-09-08 08:19:29.213302453  [2023-09-08 08:19:29] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:19:33.407196492  [2023-09-08 08:19:33] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:19:46.489062739  [2023-09-08 08:19:46] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:19:59.058407461  [2023-09-08 08:19:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/19.46.mp4 in 0.16526412963867188 seconds.
2023-09-08 08:20:08.987361242  [2023-09-08 08:20:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/19.56.mp4 in 0.12353682518005371 seconds.
2023-09-08 08:20:19.015747994  [2023-09-08 08:20:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/20.06.mp4 in 0.137984037399292 seconds.
2023-09-08 08:20:20.914013111  [2023-09-08 08:20:20] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:20:29.065346899  [2023-09-08 08:20:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/20.16.mp4 in 0.16417288780212402 seconds.
2023-09-08 08:20:29.726387879  [2023-09-08 08:20:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/20.16.mp4 in 0.17022299766540527 seconds.
2023-09-08 08:20:37.115692625  [2023-09-08 08:20:37] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:20:39.452811269  [2023-09-08 08:20:39] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:20:41.289452380  [2023-09-08 08:20:41] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:20:49.261425149  [2023-09-08 08:20:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/20.37.mp4 in 0.19863295555114746 seconds.
2023-09-08 08:20:49.630780735  [2023-09-08 08:20:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/20.37.mp4 in 0.19240880012512207 seconds.
2023-09-08 08:20:54.170957424  [2023-09-08 08:20:54] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:20:55.641176939  [2023-09-08 08:20:55] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:21:12.528484685  [2023-09-08 08:21:12] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:21:13.434176224  [2023-09-08 08:21:13] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:21:16.192573161  [2023-09-08 08:21:16] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:21:17.208415387  [2023-09-08 08:21:17] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:21:19.192899803  [2023-09-08 08:21:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/21.07.mp4 in 0.16060400009155273 seconds.
2023-09-08 08:21:28.190506079  [2023-09-08 08:21:28] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:21:29.209826069  [2023-09-08 08:21:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/21.17.mp4 in 0.1766960620880127 seconds.
2023-09-08 08:21:29.545134174  [2023-09-08 08:21:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/21.17.mp4 in 0.17420101165771484 seconds.
2023-09-08 08:21:39.558742210  [2023-09-08 08:21:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/21.26.mp4 in 0.17342782020568848 seconds.
2023-09-08 08:21:40.818774104  [2023-09-08 08:21:40] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:21:47.470004730  [2023-09-08 08:21:47] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:21:49.706529770  [2023-09-08 08:21:49] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:21:58.737638657  [2023-09-08 08:21:58] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:22:18.535292494  [2023-09-08 08:22:18] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:22:29.217422365  [2023-09-08 08:22:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/22.17.mp4 in 0.16872382164001465 seconds.
2023-09-08 08:22:44.524089768  [2023-09-08 08:22:44] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:22:52.269946050  [2023-09-08 08:22:52] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:22:59.633336888  [2023-09-08 08:22:59] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:23:01.494659895  [2023-09-08 08:23:01] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:23:09.238615945  [2023-09-08 08:23:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/22.57.mp4 in 0.17923283576965332 seconds.
2023-09-08 08:23:14.114111557  [2023-09-08 08:23:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/22.56.mp4 in 0.17256402969360352 seconds.
2023-09-08 08:23:24.630270190  [2023-09-08 08:23:24] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:23:29.523652440  [2023-09-08 08:23:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/23.16.mp4 in 0.1638641357421875 seconds.
2023-09-08 08:23:31.668640048  [2023-09-08 08:23:31] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:23:32.460333942  [2023-09-08 08:23:32] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:23:39.531741785  [2023-09-08 08:23:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/23.26.mp4 in 0.1636209487915039 seconds.
2023-09-08 08:23:48.236311617  [2023-09-08 08:23:48] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:23:49.487880118  [2023-09-08 08:23:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/23.36.mp4 in 0.16262412071228027 seconds.
2023-09-08 08:23:59.490523211  [2023-09-08 08:23:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/23.46.mp4 in 0.16291308403015137 seconds.
2023-09-08 08:24:09.463625294  [2023-09-08 08:24:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/23.56.mp4 in 0.15581798553466797 seconds.
2023-09-08 08:24:19.509588204  [2023-09-08 08:24:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/24.06.mp4 in 0.16732192039489746 seconds.
2023-09-08 08:24:49.489300910  [2023-09-08 08:24:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/24.36.mp4 in 0.1642298698425293 seconds.
2023-09-08 08:24:51.945562676  [2023-09-08 08:24:51] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:25:03.828870189  [2023-09-08 08:25:03] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:25:05.129343222  [2023-09-08 08:25:05] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:25:09.212839764  [2023-09-08 08:25:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/24.57.mp4 in 0.162736177444458 seconds.
2023-09-08 08:25:09.568969965  [2023-09-08 08:25:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/24.56.mp4 in 0.17795586585998535 seconds.
2023-09-08 08:25:12.885776944  [2023-09-08 08:25:12] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:25:36.432158597  [2023-09-08 08:25:36] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:25:36.573361491  [2023-09-08 08:25:36] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:25:41.906527174  [2023-09-08 08:25:41] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:25:49.220018613  [2023-09-08 08:25:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/25.37.mp4 in 0.16795802116394043 seconds.
2023-09-08 08:25:55.394616339  [2023-09-08 08:25:55] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:25:55.659065202  [2023-09-08 08:25:55] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:25:59.217078565  [2023-09-08 08:25:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/25.47.mp4 in 0.1768660545349121 seconds.
2023-09-08 08:25:59.588447242  [2023-09-08 08:25:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/25.46.mp4 in 0.19553399085998535 seconds.
2023-09-08 08:26:09.402739251  [2023-09-08 08:26:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/25.56.mp4 in 0.18491506576538086 seconds.
2023-09-08 08:26:10.544127134  [2023-09-08 08:26:10] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:26:19.515025355  [2023-09-08 08:26:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/26.06.mp4 in 0.15446186065673828 seconds.
2023-09-08 08:26:20.304370260  [2023-09-08 08:26:20] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:26:26.558089222  [2023-09-08 08:26:26] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:26:26.584633382  [2023-09-08 08:26:26] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:26:29.065384547  [2023-09-08 08:26:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/26.15.mp4 in 0.16623210906982422 seconds.
2023-09-08 08:26:29.767588895  [2023-09-08 08:26:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/26.16.mp4 in 0.16780900955200195 seconds.
2023-09-08 08:26:49.576637365  [2023-09-08 08:26:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/26.36.mp4 in 0.17873311042785645 seconds.
2023-09-08 08:26:51.637685435  [2023-09-08 08:26:51] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:26:59.368655560  [2023-09-08 08:26:59] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:27:08.205082140  [2023-09-08 08:27:08] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:28:03.078037983  [2023-09-08 08:28:03] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:28:17.678603518  [2023-09-08 08:28:17] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:28:34.130111029  [2023-09-08 08:28:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/28.19.mp4 in 0.18925213813781738 seconds.
2023-09-08 08:28:47.517232281  [2023-09-08 08:28:47] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-08 08:28:47.517384906  [2023-09-08 08:28:47] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-08 08:28:52.338153382  [2023-09-08 08:28:52] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:28:53.508377198  [2023-09-08 08:28:53] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-08 08:28:53.508575747  [2023-09-08 08:28:53] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 08:28:53.508905078  [2023-09-08 08:28:53] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Connection timed out
2023-09-08 08:28:53.509041031  [2023-09-08 08:28:53] ffmpeg.garage.detect           ERROR   :     Last message repeated 1 times
2023-09-08 08:28:58.632732459  [2023-09-08 08:28:58] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-08 08:28:58.632877751  [2023-09-08 08:28:58] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-08 08:28:59.159099670  [2023-09-08 08:28:59] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:28:59.529174899  [2023-09-08 08:28:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/28.46.mp4 in 0.1731579303741455 seconds.
2023-09-08 08:29:03.518521084  [2023-09-08 08:29:03] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-08 08:29:03.518768777  [2023-09-08 08:29:03] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 08:29:03.518772077  [2023-09-08 08:29:03] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Invalid data found when processing input
2023-09-08 08:29:06.789162698  [2023-09-08 08:29:06] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:29:08.640132476  [2023-09-08 08:29:08] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-08 08:29:08.640320719  [2023-09-08 08:29:08] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-08 08:29:09.270928820  [2023-09-08 08:29:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/28.57.mp4 in 0.18683791160583496 seconds.
2023-09-08 08:29:09.789453283  [2023-09-08 08:29:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/28.56.mp4 in 0.17805814743041992 seconds.
2023-09-08 08:29:13.527276580  [2023-09-08 08:29:13] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-08 08:29:13.527785997  [2023-09-08 08:29:13] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 08:29:13.527790084  [2023-09-08 08:29:13] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Invalid data found when processing input
2023-09-08 08:29:18.649928917  [2023-09-08 08:29:18] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-08 08:29:18.650114416  [2023-09-08 08:29:18] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-08 08:29:23.537499659  [2023-09-08 08:29:23] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-08 08:29:23.537633371  [2023-09-08 08:29:23] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 08:29:23.537815711  [2023-09-08 08:29:23] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Invalid data found when processing input
2023-09-08 08:29:26.852945976  [2023-09-08 08:29:26] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:29:29.246021470  [2023-09-08 08:29:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/29.16.mp4 in 0.16005301475524902 seconds.
2023-09-08 08:29:29.767860216  [2023-09-08 08:29:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/29.16.mp4 in 0.18976807594299316 seconds.
2023-09-08 08:29:47.461130733  [2023-09-08 08:29:47] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:29:49.274109436  [2023-09-08 08:29:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/29.37.mp4 in 0.1873159408569336 seconds.
2023-09-08 08:29:49.601445460  [2023-09-08 08:29:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/29.35.mp4 in 0.16239595413208008 seconds.
2023-09-08 08:30:10.499079362  [2023-09-08 08:30:10] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:30:14.328531115  [2023-09-08 08:30:14] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:30:16.291642353  [2023-09-08 08:30:16] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:30:20.128734563  [2023-09-08 08:30:20] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:31:14.205521275  [2023-09-08 08:31:14] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:31:38.808704371  [2023-09-08 08:31:38] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:31:38.950099758  [2023-09-08 08:31:38] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:31:38.959222902  [2023-09-08 08:31:38] frigate.comms.mqtt             ERROR   : Unable to publish to front/person/snapshot: client is not connected
2023-09-08 08:31:38.959227307  [2023-09-08 08:31:38] frigate.comms.mqtt             ERROR   : Unable to publish to front/person: client is not connected
2023-09-08 08:31:38.959229340  [2023-09-08 08:31:38] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:31:39.677775135  [2023-09-08 08:31:39] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:31:39.681138673  [2023-09-08 08:31:39] frigate.comms.mqtt             ERROR   : Unable to publish to front/person: client is not connected
2023-09-08 08:31:39.683095904  [2023-09-08 08:31:39] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:31:40.231745744  [2023-09-08 08:31:40] frigate.comms.mqtt             ERROR   : Unable to publish to front/person/snapshot: client is not connected
2023-09-08 08:31:43.153089256  [2023-09-08 08:31:43] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:31:44.013226758  [2023-09-08 08:31:44] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:31:44.815836162  [2023-09-08 08:31:44] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:31:44.855144973  [2023-09-08 08:31:44] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:31:44.884653565  [2023-09-08 08:31:44] frigate.comms.mqtt             ERROR   : Unable to publish to front/person: client is not connected
2023-09-08 08:31:44.884965186  [2023-09-08 08:31:44] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:31:48.870943751  [2023-09-08 08:31:48] frigate.comms.mqtt             ERROR   : Unable to publish to events: client is not connected
2023-09-08 08:31:48.887893028  [2023-09-08 08:31:48] frigate.comms.mqtt             ERROR   : Unable to publish to front/all: client is not connected
2023-09-08 08:31:48.887897687  [2023-09-08 08:31:48] frigate.comms.mqtt             ERROR   : Unable to publish to front/person: client is not connected
2023-09-08 08:31:48.894326796  [2023-09-08 08:31:48] frigate.comms.mqtt             ERROR   : Unable to publish to front/person/snapshot: client is not connected
2023-09-08 08:31:49.280494726  [2023-09-08 08:31:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/31.36.mp4 in 0.1746349334716797 seconds.
2023-09-08 08:31:49.610375545  [2023-09-08 08:31:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/31.36.mp4 in 0.17331290245056152 seconds.
2023-09-08 08:31:59.430903110  [2023-09-08 08:31:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/31.45.mp4 in 0.18931198120117188 seconds.
2023-09-08 08:32:13.689571206  [2023-09-08 08:32:13] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:32:16.152981271  [2023-09-08 08:32:16] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:32:16.636711302  [2023-09-08 08:32:16] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:32:16.921748793  [2023-09-08 08:32:16] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:32:17.918499832  [2023-09-08 08:32:17] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:32:29.274605973  [2023-09-08 08:32:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/32.16.mp4 in 0.18933606147766113 seconds.
2023-09-08 08:32:34.103610209  [2023-09-08 08:32:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/32.15.mp4 in 0.1653590202331543 seconds.
2023-09-08 08:32:48.728685485  [2023-09-08 08:32:48] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:32:49.383637814  [2023-09-08 08:32:49] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:33:21.636515307  [2023-09-08 08:33:21] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:34:25.339818047  [2023-09-08 08:34:25] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:34:53.317902887  [2023-09-08 08:34:53] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:34:54.875893635  [2023-09-08 08:34:54] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:34:59.215262314  [2023-09-08 08:34:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/34.48.mp4 in 0.16472983360290527 seconds.
2023-09-08 08:34:59.595064896  [2023-09-08 08:34:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/34.47.mp4 in 0.19755101203918457 seconds.
2023-09-08 08:35:25.569402272  [2023-09-08 08:35:25] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:35:26.525089075  [2023-09-08 08:35:26] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:35:29.047043739  [2023-09-08 08:35:29] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:35:56.615514682  [2023-09-08 08:35:56] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:35:57.673740999  [2023-09-08 08:35:57] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:35:59.191666198  [2023-09-08 08:35:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/35.48.mp4 in 0.14998292922973633 seconds.
2023-09-08 08:36:06.183189137  [2023-09-08 08:36:06] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:36:09.270421143  [2023-09-08 08:36:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/35.58.mp4 in 0.1807260513305664 seconds.
2023-09-08 08:36:09.620315031  [2023-09-08 08:36:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/35.57.mp4 in 0.18164801597595215 seconds.
2023-09-08 08:36:09.879417688  [2023-09-08 08:36:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/35.57.mp4 in 0.13298511505126953 seconds.
2023-09-08 08:36:19.543094829  [2023-09-08 08:36:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/36.07.mp4 in 0.17087912559509277 seconds.
2023-09-08 08:36:28.217387585  [2023-09-08 08:36:28] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:36:28.873131150  [2023-09-08 08:36:28] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:36:32.778200173  [2023-09-08 08:36:32] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:36:49.588820051  [2023-09-08 08:36:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/36.37.mp4 in 0.18526697158813477 seconds.
2023-09-08 08:37:12.787028142  [2023-09-08 08:37:12] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:37:13.405474631  [2023-09-08 08:37:13] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:37:14.351879123  [2023-09-08 08:37:14] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:37:19.052468838  [2023-09-08 08:37:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/37.06.mp4 in 0.13520193099975586 seconds.
2023-09-08 08:37:19.700019054  [2023-09-08 08:37:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/37.07.mp4 in 0.17260408401489258 seconds.
2023-09-08 08:37:29.081845240  [2023-09-08 08:37:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/37.16.mp4 in 0.1589949131011963 seconds.
2023-09-08 08:37:36.494282197  [2023-09-08 08:37:36] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:37:41.076660680  [2023-09-08 08:37:41] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:37:44.304476809  [2023-09-08 08:37:44] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:37:47.218317659  [2023-09-08 08:37:47] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:37:49.261652843  [2023-09-08 08:37:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/37.38.mp4 in 0.17394399642944336 seconds.
2023-09-08 08:37:50.694280990  [2023-09-08 08:37:50] frigate.comms.mqtt             ERROR   : Unable to publish to back_porch/motion: client is not connected
2023-09-08 08:37:59.558084214  [2023-09-08 08:37:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/37.47.mp4 in 0.17426109313964844 seconds.
2023-09-08 08:38:09.581714134  [2023-09-08 08:38:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/37.57.mp4 in 0.18414807319641113 seconds.
2023-09-08 08:38:13.385381757  [2023-09-08 08:38:13] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:38:29.613145257  [2023-09-08 08:38:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/38.17.mp4 in 0.19486308097839355 seconds.
2023-09-08 08:38:40.204214564  [2023-09-08 08:38:40] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:38:40.646841940  [2023-09-08 08:38:40] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:38:41.202179481  [2023-09-08 08:38:41] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:38:49.232251348  [2023-09-08 08:38:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/38.38.mp4 in 0.16112494468688965 seconds.
2023-09-08 08:38:49.524855334  [2023-09-08 08:38:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/38.37.mp4 in 0.13926386833190918 seconds.
2023-09-08 08:38:49.899845049  [2023-09-08 08:38:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/38.37.mp4 in 0.19750499725341797 seconds.
2023-09-08 08:38:59.304945923  [2023-09-08 08:38:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/38.48.mp4 in 0.18906688690185547 seconds.
2023-09-08 08:38:59.610355018  [2023-09-08 08:38:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/38.47.mp4 in 0.15526986122131348 seconds.
2023-09-08 08:38:59.936941803  [2023-09-08 08:38:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/38.47.mp4 in 0.17526793479919434 seconds.
2023-09-08 08:39:09.395754491  [2023-09-08 08:39:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/38.57.mp4 in 0.1561110019683838 seconds.
2023-09-08 08:39:19.531435153  [2023-09-08 08:39:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/39.07.mp4 in 0.16279911994934082 seconds.
2023-09-08 08:39:24.648610706  [2023-09-08 08:39:24] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:39:25.325152183  [2023-09-08 08:39:25] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:39:39.575993305  [2023-09-08 08:39:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/39.27.mp4 in 0.15747594833374023 seconds.
2023-09-08 08:39:43.916258589  [2023-09-08 08:39:43] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:40:01.912840251  [2023-09-08 08:40:01] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:40:04.749315203  [2023-09-08 08:40:04] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:40:09.609044305  [2023-09-08 08:40:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/39.56.mp4 in 0.17401409149169922 seconds.
2023-09-08 08:40:19.515759247  [2023-09-08 08:40:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/40.06.mp4 in 0.14156603813171387 seconds.
2023-09-08 08:40:20.402388591  [2023-09-08 08:40:20] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:40:22.158555791  [2023-09-08 08:40:22] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:40:29.229505386  [2023-09-08 08:40:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/40.17.mp4 in 0.1594228744506836 seconds.
2023-09-08 08:40:29.541507774  [2023-09-08 08:40:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/40.17.mp4 in 0.15822505950927734 seconds.
2023-09-08 08:40:29.873023942  [2023-09-08 08:40:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/40.16.mp4 in 0.1657261848449707 seconds.
2023-09-08 08:40:39.418807197  [2023-09-08 08:40:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/40.28.mp4 in 0.17275285720825195 seconds.
2023-09-08 08:40:39.755175365  [2023-09-08 08:40:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/40.26.mp4 in 0.18567109107971191 seconds.
2023-09-08 08:40:47.626703760  [2023-09-08 08:40:47] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:40:49.577679963  [2023-09-08 08:40:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/40.36.mp4 in 0.17540502548217773 seconds.
2023-09-08 08:40:54.207927460  [2023-09-08 08:40:54] frigate.comms.mqtt             ERROR   : Unable to publish to front/motion: client is not connected
2023-09-08 08:40:59.671100156  [2023-09-08 08:40:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/40.47.mp4 in 0.1799910068511963 seconds.
2023-09-08 08:41:01.038259686  [2023-09-08 08:41:01] frigate.comms.mqtt             ERROR   : Unable to publish to garage/motion: client is not connected
2023-09-08 08:41:18.987579803  [2023-09-08 08:41:18] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:41:51.334064496  [2023-09-08 08:41:51] frigate.comms.mqtt             ERROR   : Unable to publish to stats: client is not connected
2023-09-08 08:41:55.456001205  [2023-09-08 08:41:55] frigate.comms.mqtt             ERROR   : Unable to publish to garden/motion: client is not connected
2023-09-08 08:41:59.633692810  [2023-09-08 08:41:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/41.46.mp4 in 0.19918322563171387 seconds.
2023-09-08 08:42:59.542314551  [2023-09-08 08:42:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/42.46.mp4 in 0.1706850528717041 seconds.
2023-09-08 08:43:09.257981816  [2023-09-08 08:43:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/42.57.mp4 in 0.15700507164001465 seconds.
2023-09-08 08:43:09.591319074  [2023-09-08 08:43:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/42.56.mp4 in 0.17217588424682617 seconds.
2023-09-08 08:43:09.921808604  [2023-09-08 08:43:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/42.56.mp4 in 0.17327594757080078 seconds.
2023-09-08 08:43:19.250896382  [2023-09-08 08:43:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/43.07.mp4 in 0.17126011848449707 seconds.
2023-09-08 08:43:19.734215248  [2023-09-08 08:43:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/43.06.mp4 in 0.17723608016967773 seconds.
2023-09-08 08:43:29.497738732  [2023-09-08 08:43:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/43.16.mp4 in 0.16453909873962402 seconds.
2023-09-08 08:43:49.564608440  [2023-09-08 08:43:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/43.36.mp4 in 0.17494702339172363 seconds.
2023-09-08 08:43:59.530244742  [2023-09-08 08:43:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/43.46.mp4 in 0.16930484771728516 seconds.
2023-09-08 08:44:09.515204750  [2023-09-08 08:44:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/43.56.mp4 in 0.15970706939697266 seconds.
2023-09-08 08:44:19.546565183  [2023-09-08 08:44:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/44.06.mp4 in 0.17131805419921875 seconds.
2023-09-08 08:44:29.050689461  [2023-09-08 08:44:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/44.15.mp4 in 0.12497282028198242 seconds.
2023-09-08 08:44:39.044790152  [2023-09-08 08:44:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/44.25.mp4 in 0.12290310859680176 seconds.
2023-09-08 08:44:39.683459213  [2023-09-08 08:44:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/44.26.mp4 in 0.1703019142150879 seconds.
2023-09-08 08:44:49.070995625  [2023-09-08 08:44:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/44.35.mp4 in 0.1343088150024414 seconds.
2023-09-08 08:44:59.095166576  [2023-09-08 08:44:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/44.45.mp4 in 0.14230704307556152 seconds.
2023-09-08 08:45:09.072926300  [2023-09-08 08:45:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/44.55.mp4 in 0.1344900131225586 seconds.
2023-09-08 08:45:19.093964228  [2023-09-08 08:45:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/45.05.mp4 in 0.14457488059997559 seconds.
2023-09-08 08:45:29.107436125  [2023-09-08 08:45:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/45.15.mp4 in 0.1544480323791504 seconds.
2023-09-08 08:45:59.079975529  [2023-09-08 08:45:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/45.45.mp4 in 0.14266514778137207 seconds.
2023-09-08 08:45:59.435161395  [2023-09-08 08:45:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/45.47.mp4 in 0.18316984176635742 seconds.
2023-09-08 08:45:59.758475608  [2023-09-08 08:45:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/45.46.mp4 in 0.16632795333862305 seconds.
2023-09-08 08:46:09.087052939  [2023-09-08 08:46:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/45.55.mp4 in 0.1488499641418457 seconds.
2023-09-08 08:46:09.393176328  [2023-09-08 08:46:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/45.57.mp4 in 0.1649940013885498 seconds.
2023-09-08 08:47:09.258178188  [2023-09-08 08:47:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/46.57.mp4 in 0.1617269515991211 seconds.
2023-09-08 08:47:09.586782176  [2023-09-08 08:47:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/46.56.mp4 in 0.1643660068511963 seconds.
2023-09-08 08:47:29.586761219  [2023-09-08 08:47:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/47.16.mp4 in 0.17322182655334473 seconds.
2023-09-08 08:47:39.518216547  [2023-09-08 08:47:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/47.26.mp4 in 0.17790508270263672 seconds.
2023-09-08 08:48:39.584348887  [2023-09-08 08:48:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/48.26.mp4 in 0.17311716079711914 seconds.
2023-09-08 08:48:49.606434618  [2023-09-08 08:48:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/48.36.mp4 in 0.1834239959716797 seconds.
2023-09-08 08:51:59.293953689  [2023-09-08 08:51:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/51.46.mp4 in 0.1844630241394043 seconds.
2023-09-08 08:51:59.602807748  [2023-09-08 08:51:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/51.46.mp4 in 0.16357707977294922 seconds.
2023-09-08 08:53:49.681170956  [2023-09-08 08:53:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/53.37.mp4 in 0.2112109661102295 seconds.
2023-09-08 08:53:59.124286131  [2023-09-08 08:53:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/53.47.mp4 in 0.14671587944030762 seconds.
2023-09-08 08:53:59.791899364  [2023-09-08 08:53:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/53.47.mp4 in 0.18095183372497559 seconds.
2023-09-08 08:54:19.625264169  [2023-09-08 08:54:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/54.07.mp4 in 0.2061598300933838 seconds.
2023-09-08 08:54:59.304832239  [2023-09-08 08:54:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/54.48.mp4 in 0.18421196937561035 seconds.
2023-09-08 08:54:59.666143955  [2023-09-08 08:54:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/54.47.mp4 in 0.19351506233215332 seconds.
2023-09-08 08:55:04.208636085  [2023-09-08 08:55:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/54.47.mp4 in 0.1966090202331543 seconds.
2023-09-08 08:55:09.261888271  [2023-09-08 08:55:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/54.58.mp4 in 0.15165019035339355 seconds.
2023-09-08 08:55:09.606574553  [2023-09-08 08:55:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/54.57.mp4 in 0.17832207679748535 seconds.
2023-09-08 08:55:14.196222780  [2023-09-08 08:55:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/54.58.mp4 in 0.19809985160827637 seconds.
2023-09-08 08:55:19.438790572  [2023-09-08 08:55:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/55.07.mp4 in 0.16292309761047363 seconds.
2023-09-08 08:55:29.684141263  [2023-09-08 08:55:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/55.17.mp4 in 0.1953449249267578 seconds.
2023-09-08 08:55:39.728282916  [2023-09-08 08:55:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/55.27.mp4 in 0.18886494636535645 seconds.
2023-09-08 08:55:49.695882863  [2023-09-08 08:55:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/55.37.mp4 in 0.2202298641204834 seconds.
2023-09-08 08:56:29.675736070  [2023-09-08 08:56:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/56.17.mp4 in 0.21134090423583984 seconds.
2023-09-08 08:56:39.687485412  [2023-09-08 08:56:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/56.27.mp4 in 0.22121381759643555 seconds.
2023-09-08 08:56:49.684780000  [2023-09-08 08:56:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/56.37.mp4 in 0.21112298965454102 seconds.
2023-09-08 08:56:59.638692098  [2023-09-08 08:56:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/56.47.mp4 in 0.18737220764160156 seconds.
2023-09-08 08:57:09.700822690  [2023-09-08 08:57:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/56.57.mp4 in 0.18762993812561035 seconds.
2023-09-08 08:57:19.643400293  [2023-09-08 08:57:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/57.07.mp4 in 0.18004608154296875 seconds.
2023-09-08 08:57:29.610384857  [2023-09-08 08:57:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/57.17.mp4 in 0.17167997360229492 seconds.
2023-09-08 08:57:39.598512111  [2023-09-08 08:57:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/57.27.mp4 in 0.17133784294128418 seconds.
2023-09-08 08:57:49.652775884  [2023-09-08 08:57:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/57.37.mp4 in 0.1665189266204834 seconds.
2023-09-08 08:57:59.251800158  [2023-09-08 08:57:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/57.48.mp4 in 0.14781522750854492 seconds.
2023-09-08 08:57:59.620206926  [2023-09-08 08:57:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garage/57.47.mp4 in 0.18775296211242676 seconds.
2023-09-08 08:57:59.933019174  [2023-09-08 08:57:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/57.47.mp4 in 0.1614389419555664 seconds.
2023-09-08 08:58:09.593198311  [2023-09-08 08:58:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/57.57.mp4 in 0.16410183906555176 seconds.
2023-09-08 08:58:19.620569207  [2023-09-08 08:58:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/58.07.mp4 in 0.1609649658203125 seconds.
2023-09-08 08:58:29.664494258  [2023-09-08 08:58:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/58.17.mp4 in 0.16734099388122559 seconds.
2023-09-08 08:58:39.626627957  [2023-09-08 08:58:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/58.27.mp4 in 0.20036697387695312 seconds.
2023-09-08 08:58:49.618634924  [2023-09-08 08:58:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/58.37.mp4 in 0.1639089584350586 seconds.
2023-09-08 08:58:59.627417551  [2023-09-08 08:58:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/58.47.mp4 in 0.19489216804504395 seconds.
2023-09-08 08:59:29.174657772  [2023-09-08 08:59:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/back_porch/59.16.mp4 in 0.1817610263824463 seconds.
2023-09-08 08:59:39.668908683  [2023-09-08 08:59:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/59.27.mp4 in 0.20638680458068848 seconds.
2023-09-08 08:59:49.688427047  [2023-09-08 08:59:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/59.37.mp4 in 0.189223051071167 seconds.
2023-09-08 08:59:59.620931964  [2023-09-08 08:59:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/59.47.mp4 in 0.16596508026123047 seconds.
2023-09-08 09:00:09.290133923  [2023-09-08 09:00:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/front/59.58.mp4 in 0.16246604919433594 seconds.
2023-09-08 09:00:09.798666492  [2023-09-08 09:00:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/13/garden/59.56.mp4 in 0.16289687156677246 seconds.
2023-09-08 09:00:19.641629382  [2023-09-08 09:00:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/00.06.mp4 in 0.1737079620361328 seconds.
2023-09-08 09:00:29.158211157  [2023-09-08 09:00:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/00.16.mp4 in 0.16269683837890625 seconds.
2023-09-08 09:00:39.694109062  [2023-09-08 09:00:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/00.27.mp4 in 0.18597197532653809 seconds.
2023-09-08 09:00:49.650718667  [2023-09-08 09:00:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/00.36.mp4 in 0.19010305404663086 seconds.
2023-09-08 09:00:59.742574920  [2023-09-08 09:00:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/00.46.mp4 in 0.19764995574951172 seconds.
2023-09-08 09:01:09.667012778  [2023-09-08 09:01:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/00.57.mp4 in 0.190201997756958 seconds.
2023-09-08 09:01:59.667690988  [2023-09-08 09:01:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/01.46.mp4 in 0.19147801399230957 seconds.
2023-09-08 09:02:09.672986937  [2023-09-08 09:02:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/01.56.mp4 in 0.1874680519104004 seconds.
2023-09-08 09:02:19.634630226  [2023-09-08 09:02:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/02.06.mp4 in 0.1723170280456543 seconds.
2023-09-08 09:02:29.698614784  [2023-09-08 09:02:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/02.16.mp4 in 0.19027495384216309 seconds.
2023-09-08 09:02:39.673760941  [2023-09-08 09:02:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/02.26.mp4 in 0.17759990692138672 seconds.
2023-09-08 09:03:19.687960594  [2023-09-08 09:03:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/03.06.mp4 in 0.19513583183288574 seconds.
2023-09-08 09:03:29.259617485  [2023-09-08 09:03:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/03.17.mp4 in 0.14090681076049805 seconds.
2023-09-08 09:03:29.638662009  [2023-09-08 09:03:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/03.16.mp4 in 0.19710803031921387 seconds.
2023-09-08 09:03:29.963736185  [2023-09-08 09:03:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/03.16.mp4 in 0.16724300384521484 seconds.
2023-09-08 09:03:39.565057204  [2023-09-08 09:03:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/03.28.mp4 in 0.20114994049072266 seconds.
2023-09-08 09:03:39.956128744  [2023-09-08 09:03:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/03.26.mp4 in 0.1875290870666504 seconds.
2023-09-08 09:03:49.676905278  [2023-09-08 09:03:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/03.36.mp4 in 0.18477392196655273 seconds.
2023-09-08 09:04:09.720613230  [2023-09-08 09:04:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/03.56.mp4 in 0.19428682327270508 seconds.
2023-09-08 09:04:19.322207031  [2023-09-08 09:04:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/04.07.mp4 in 0.17721295356750488 seconds.
2023-09-08 09:04:24.219736630  [2023-09-08 09:04:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/04.06.mp4 in 0.1840829849243164 seconds.
2023-09-08 09:04:29.237295034  [2023-09-08 09:04:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/04.17.mp4 in 0.127000093460083 seconds.
2023-09-08 09:05:19.663730214  [2023-09-08 09:05:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/05.06.mp4 in 0.18968415260314941 seconds.
2023-09-08 09:05:29.667920970  [2023-09-08 09:05:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/05.16.mp4 in 0.18331599235534668 seconds.
2023-09-08 09:05:39.620375076  [2023-09-08 09:05:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/05.26.mp4 in 0.1806778907775879 seconds.
2023-09-08 09:05:49.615252862  [2023-09-08 09:05:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/05.36.mp4 in 0.16688108444213867 seconds.
2023-09-08 09:05:59.656298500  [2023-09-08 09:05:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/05.46.mp4 in 0.1764841079711914 seconds.
2023-09-08 09:06:09.108510983  [2023-09-08 09:06:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/05.55.mp4 in 0.12647700309753418 seconds.
2023-09-08 09:06:09.753937555  [2023-09-08 09:06:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/05.56.mp4 in 0.1796419620513916 seconds.
2023-09-08 09:07:19.314577176  [2023-09-08 09:07:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/07.07.mp4 in 0.16926217079162598 seconds.
2023-09-08 09:07:19.841528419  [2023-09-08 09:07:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/07.06.mp4 in 0.18205595016479492 seconds.
2023-09-08 09:07:29.692942014  [2023-09-08 09:07:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/07.16.mp4 in 0.20323491096496582 seconds.
2023-09-08 09:07:39.705833716  [2023-09-08 09:07:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/07.26.mp4 in 0.19210219383239746 seconds.
2023-09-08 09:07:59.739554943  [2023-09-08 09:07:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/07.46.mp4 in 0.19440913200378418 seconds.
2023-09-08 09:08:09.712990192  [2023-09-08 09:08:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/07.56.mp4 in 0.20184898376464844 seconds.
2023-09-08 09:08:19.688871501  [2023-09-08 09:08:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/08.06.mp4 in 0.1965489387512207 seconds.
2023-09-08 09:08:29.317111478  [2023-09-08 09:08:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/08.17.mp4 in 0.1631159782409668 seconds.
2023-09-08 09:08:29.691559189  [2023-09-08 09:08:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/08.16.mp4 in 0.19671106338500977 seconds.
2023-09-08 09:08:30.007500415  [2023-09-08 09:08:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/08.16.mp4 in 0.17033910751342773 seconds.
2023-09-08 09:08:39.713709927  [2023-09-08 09:08:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/08.26.mp4 in 0.17966103553771973 seconds.
2023-09-08 09:08:49.651686760  [2023-09-08 09:08:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/08.36.mp4 in 0.18224883079528809 seconds.
2023-09-08 09:08:59.301521925  [2023-09-08 09:08:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/08.47.mp4 in 0.16254806518554688 seconds.
2023-09-08 09:08:59.812802160  [2023-09-08 09:08:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/08.46.mp4 in 0.18165922164916992 seconds.
2023-09-08 09:09:09.672973703  [2023-09-08 09:09:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/08.56.mp4 in 0.19788694381713867 seconds.
2023-09-08 09:09:19.648535532  [2023-09-08 09:09:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/09.06.mp4 in 0.17354202270507812 seconds.
2023-09-08 09:09:29.609482114  [2023-09-08 09:09:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/09.16.mp4 in 0.17015600204467773 seconds.
2023-09-08 09:09:49.635354436  [2023-09-08 09:09:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/09.36.mp4 in 0.1790008544921875 seconds.
2023-09-08 09:10:39.694678881  [2023-09-08 09:10:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/10.26.mp4 in 0.2002100944519043 seconds.
2023-09-08 09:10:49.340092375  [2023-09-08 09:10:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/10.37.mp4 in 0.18282413482666016 seconds.
2023-09-08 09:10:49.728634399  [2023-09-08 09:10:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/10.36.mp4 in 0.20299506187438965 seconds.
2023-09-08 09:10:50.034872739  [2023-09-08 09:10:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/10.35.mp4 in 0.1556379795074463 seconds.
2023-09-08 09:10:59.306200733  [2023-09-08 09:10:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/10.47.mp4 in 0.15371298789978027 seconds.
2023-09-08 09:10:59.610433700  [2023-09-08 09:10:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/10.46.mp4 in 0.15024995803833008 seconds.
2023-09-08 09:10:59.927766134  [2023-09-08 09:10:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/10.45.mp4 in 0.16657686233520508 seconds.
2023-09-08 09:11:49.349941078  [2023-09-08 09:11:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/11.36.mp4 in 0.16906499862670898 seconds.
2023-09-08 09:11:49.746498456  [2023-09-08 09:11:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/11.36.mp4 in 0.20971012115478516 seconds.
2023-09-08 09:11:50.125020501  [2023-09-08 09:11:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/11.35.mp4 in 0.19305682182312012 seconds.
2023-09-08 09:11:56.393651065  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start expire recordings (new).
2023-09-08 09:11:56.393824184  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start deleted cameras.
2023-09-08 09:11:56.396546482  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-08 09:11:56.397986782  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End deleted cameras.
2023-09-08 09:11:56.397991408  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start all cameras.
2023-09-08 09:11:56.397993753  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start camera: garage.
2023-09-08 09:11:56.425184396  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-08 09:11:56.425341126  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End camera: garage.
2023-09-08 09:11:56.425473469  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start camera: back_porch.
2023-09-08 09:11:56.456213576  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-08 09:11:56.456309805  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End camera: back_porch.
2023-09-08 09:11:56.456516185  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start camera: garden.
2023-09-08 09:11:56.493758481  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-08 09:11:56.493764607  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End camera: garden.
2023-09-08 09:11:56.493766806  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start camera: front.
2023-09-08 09:11:56.516661419  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-09-08 09:11:56.517132496  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End camera: front.
2023-09-08 09:11:56.517136237  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End all cameras.
2023-09-08 09:11:56.517138688  [2023-09-08 09:11:56] frigate.record                 DEBUG   : End expire recordings (new).
2023-09-08 09:11:56.517140479  [2023-09-08 09:11:56] frigate.record                 DEBUG   : Start expire files (legacy).
2023-09-08 09:11:56.556949847  [2023-09-08 09:11:56] frigate.record                 WARNING : Unable to find file from recordings database: /media/frigate/recordings/2023-08-26/13/garage/34.03.mp4
2023-09-08 09:11:59.336667524  [2023-09-08 09:11:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/11.46.mp4 in 0.16265511512756348 seconds.
2023-09-08 09:12:59.357276963  [2023-09-08 09:12:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/12.46.mp4 in 0.1644458770751953 seconds.
2023-09-08 09:12:59.750593751  [2023-09-08 09:12:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/12.46.mp4 in 0.20476698875427246 seconds.
2023-09-08 09:13:19.736479802  [2023-09-08 09:13:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/13.05.mp4 in 0.1896989345550537 seconds.
2023-09-08 09:13:29.184896702  [2023-09-08 09:13:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/13.15.mp4 in 0.15519213676452637 seconds.
2023-09-08 09:13:29.834471396  [2023-09-08 09:13:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/13.15.mp4 in 0.18548893928527832 seconds.
2023-09-08 09:13:39.742131314  [2023-09-08 09:13:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/13.25.mp4 in 0.1887350082397461 seconds.
2023-09-08 09:13:49.720539250  [2023-09-08 09:13:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/13.35.mp4 in 0.2223348617553711 seconds.
2023-09-08 09:13:59.633808977  [2023-09-08 09:13:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/13.47.mp4 in 0.1801748275756836 seconds.
2023-09-08 09:14:09.693303327  [2023-09-08 09:14:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/13.57.mp4 in 0.20960712432861328 seconds.
2023-09-08 09:14:39.674992706  [2023-09-08 09:14:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/14.27.mp4 in 0.19937896728515625 seconds.
2023-09-08 09:14:49.716567655  [2023-09-08 09:14:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/14.37.mp4 in 0.20085811614990234 seconds.
2023-09-08 09:14:59.148151976  [2023-09-08 09:14:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/14.46.mp4 in 0.14622092247009277 seconds.
2023-09-08 09:14:59.886135709  [2023-09-08 09:14:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/14.47.mp4 in 0.18849611282348633 seconds.
2023-09-08 09:15:59.619022683  [2023-09-08 09:15:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/15.47.mp4 in 0.17988204956054688 seconds.
2023-09-08 09:16:19.713239741  [2023-09-08 09:16:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/16.07.mp4 in 0.20463299751281738 seconds.
2023-09-08 09:16:29.136887740  [2023-09-08 09:16:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/16.16.mp4 in 0.13500595092773438 seconds.
2023-09-08 09:16:29.840048198  [2023-09-08 09:16:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/16.17.mp4 in 0.2046799659729004 seconds.
2023-09-08 09:18:09.314683834  [2023-09-08 09:18:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/17.58.mp4 in 0.17747092247009277 seconds.
2023-09-08 09:18:14.247431890  [2023-09-08 09:18:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/17.57.mp4 in 0.19566106796264648 seconds.
2023-09-08 09:19:09.234113190  [2023-09-08 09:19:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/18.56.mp4 in 0.18643593788146973 seconds.
2023-09-08 09:19:29.387951859  [2023-09-08 09:19:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/19.18.mp4 in 0.18198609352111816 seconds.
2023-09-08 09:19:39.330244659  [2023-09-08 09:19:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/19.28.mp4 in 0.1487879753112793 seconds.
2023-09-08 09:19:44.261467475  [2023-09-08 09:19:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/19.27.mp4 in 0.20401310920715332 seconds.
2023-09-08 09:19:49.366034732  [2023-09-08 09:19:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/19.38.mp4 in 0.1625831127166748 seconds.
2023-09-08 09:19:54.218119126  [2023-09-08 09:19:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/19.39.mp4 in 0.18035387992858887 seconds.
2023-09-08 09:20:29.401496216  [2023-09-08 09:20:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/20.18.mp4 in 0.17627811431884766 seconds.
2023-09-08 09:20:39.304366472  [2023-09-08 09:20:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/20.27.mp4 in 0.13872814178466797 seconds.
2023-09-08 09:20:49.350727557  [2023-09-08 09:20:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/20.38.mp4 in 0.16355299949645996 seconds.
2023-09-08 09:20:49.723112630  [2023-09-08 09:20:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/20.37.mp4 in 0.1859140396118164 seconds.
2023-09-08 09:21:09.736801135  [2023-09-08 09:21:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/20.57.mp4 in 0.21004915237426758 seconds.
2023-09-08 09:21:19.678571205  [2023-09-08 09:21:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/21.06.mp4 in 0.18278288841247559 seconds.
2023-09-08 09:21:29.194185287  [2023-09-08 09:21:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/21.16.mp4 in 0.15581703186035156 seconds.
2023-09-08 09:22:59.151633663  [2023-09-08 09:22:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/22.46.mp4 in 0.13987016677856445 seconds.
2023-09-08 09:23:09.210628207  [2023-09-08 09:23:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/22.56.mp4 in 0.16986799240112305 seconds.
2023-09-08 09:23:29.684287770  [2023-09-08 09:23:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/23.16.mp4 in 0.16848993301391602 seconds.
2023-09-08 09:23:49.684342135  [2023-09-08 09:23:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/23.36.mp4 in 0.18335700035095215 seconds.
2023-09-08 09:23:59.709469038  [2023-09-08 09:23:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/23.46.mp4 in 0.18537092208862305 seconds.
2023-09-08 09:24:19.642437601  [2023-09-08 09:24:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/24.06.mp4 in 0.16707396507263184 seconds.
2023-09-08 09:24:59.729883683  [2023-09-08 09:24:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/24.46.mp4 in 0.20522499084472656 seconds.
2023-09-08 09:26:49.203379300  [2023-09-08 09:26:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/26.35.mp4 in 0.1553339958190918 seconds.
2023-09-08 09:26:59.205692828  [2023-09-08 09:26:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/26.45.mp4 in 0.1569831371307373 seconds.
2023-09-08 09:27:09.175480432  [2023-09-08 09:27:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/26.55.mp4 in 0.14389491081237793 seconds.
2023-09-08 09:27:19.214305812  [2023-09-08 09:27:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/27.05.mp4 in 0.163726806640625 seconds.
2023-09-08 09:27:49.708782203  [2023-09-08 09:27:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/27.36.mp4 in 0.18845200538635254 seconds.
2023-09-08 09:27:59.232078080  [2023-09-08 09:27:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/27.45.mp4 in 0.16451096534729004 seconds.
2023-09-08 09:28:09.753611985  [2023-09-08 09:28:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/27.56.mp4 in 0.18970894813537598 seconds.
2023-09-08 09:28:19.737870567  [2023-09-08 09:28:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/28.06.mp4 in 0.21128106117248535 seconds.
2023-09-08 09:28:29.179423303  [2023-09-08 09:28:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/28.15.mp4 in 0.14079713821411133 seconds.
2023-09-08 09:28:39.701876396  [2023-09-08 09:28:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/28.26.mp4 in 0.18425703048706055 seconds.
2023-09-08 09:28:49.144540896  [2023-09-08 09:28:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/28.35.mp4 in 0.12410902976989746 seconds.
2023-09-08 09:28:49.807376974  [2023-09-08 09:28:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/28.36.mp4 in 0.16964101791381836 seconds.
2023-09-08 09:28:59.204116776  [2023-09-08 09:28:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/28.45.mp4 in 0.15329790115356445 seconds.
2023-09-08 09:28:59.879643263  [2023-09-08 09:28:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/28.46.mp4 in 0.18999409675598145 seconds.
2023-09-08 09:29:09.200544699  [2023-09-08 09:29:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/28.55.mp4 in 0.15259695053100586 seconds.
2023-09-08 09:29:09.868823208  [2023-09-08 09:29:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/28.56.mp4 in 0.17597603797912598 seconds.
2023-09-08 09:29:19.165945288  [2023-09-08 09:29:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/29.05.mp4 in 0.13577508926391602 seconds.
2023-09-08 09:29:59.253555758  [2023-09-08 09:29:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/29.45.mp4 in 0.17987489700317383 seconds.
2023-09-08 09:30:49.752888366  [2023-09-08 09:30:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/30.36.mp4 in 0.18646883964538574 seconds.
2023-09-08 09:32:09.243166962  [2023-09-08 09:32:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/31.55.mp4 in 0.16677498817443848 seconds.
2023-09-08 09:32:09.902740008  [2023-09-08 09:32:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/31.55.mp4 in 0.17890000343322754 seconds.
2023-09-08 09:32:19.237847293  [2023-09-08 09:32:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/32.05.mp4 in 0.1598529815673828 seconds.
2023-09-08 09:32:19.575493173  [2023-09-08 09:32:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/32.06.mp4 in 0.1750659942626953 seconds.
2023-09-08 09:32:19.953577216  [2023-09-08 09:32:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/32.06.mp4 in 0.18552303314208984 seconds.
2023-09-08 09:32:29.264086890  [2023-09-08 09:32:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/32.15.mp4 in 0.18100810050964355 seconds.
2023-09-08 09:32:29.569261597  [2023-09-08 09:32:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/32.16.mp4 in 0.15660691261291504 seconds.
2023-09-08 09:32:59.385207766  [2023-09-08 09:32:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/32.46.mp4 in 0.1621098518371582 seconds.
2023-09-08 09:33:09.390611339  [2023-09-08 09:33:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/32.56.mp4 in 0.157210111618042 seconds.
2023-09-08 09:33:14.213500502  [2023-09-08 09:33:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/32.55.mp4 in 0.15503406524658203 seconds.
2023-09-08 09:33:29.239042564  [2023-09-08 09:33:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/33.15.mp4 in 0.16937589645385742 seconds.
2023-09-08 09:33:39.753895471  [2023-09-08 09:33:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/33.25.mp4 in 0.20185303688049316 seconds.
2023-09-08 09:33:49.391523988  [2023-09-08 09:33:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/33.36.mp4 in 0.169173002243042 seconds.
2023-09-08 09:33:49.715518842  [2023-09-08 09:33:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/33.35.mp4 in 0.1667318344116211 seconds.
2023-09-08 09:33:59.354862567  [2023-09-08 09:33:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/33.46.mp4 in 0.1406569480895996 seconds.
2023-09-08 09:33:59.682242126  [2023-09-08 09:33:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/33.47.mp4 in 0.1634681224822998 seconds.
2023-09-08 09:34:43.640139355  [2023-09-08 09:34:43] watchdog.garden                INFO    : No frames received from garden in 20 seconds. Exiting ffmpeg...
2023-09-08 09:34:43.640594747  [2023-09-08 09:34:43] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:34:43.649506669  [2023-09-08 09:34:43] watchdog.back_porch            INFO    : No frames received from back_porch in 20 seconds. Exiting ffmpeg...
2023-09-08 09:34:43.649741526  [2023-09-08 09:34:43] watchdog.front                 INFO    : No frames received from front in 20 seconds. Exiting ffmpeg...
2023-09-08 09:34:43.649825239  [2023-09-08 09:34:43] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:34:43.649992325  [2023-09-08 09:34:43] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:34:43.658546470  [2023-09-08 09:34:43] frigate.video                  ERROR   : front: Unable to read frames from ffmpeg process.
2023-09-08 09:34:43.658759954  [2023-09-08 09:34:43] frigate.video                  ERROR   : front: ffmpeg process is not running. exiting capture thread...
2023-09-08 09:34:43.660710553  [2023-09-08 09:34:43] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-09-08 09:34:43.661001193  [2023-09-08 09:34:43] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-09-08 09:34:43.675408087  [2023-09-08 09:34:43] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-08 09:34:43.675550676  [2023-09-08 09:34:43] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-08 09:34:53.660194527  [2023-09-08 09:34:53] watchdog.front                 ERROR   : Ffmpeg process crashed unexpectedly for front.
2023-09-08 09:34:53.660431227  [2023-09-08 09:34:53] watchdog.front                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 09:34:53.663312223  [2023-09-08 09:34:53] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-09-08 09:34:53.663317141  [2023-09-08 09:34:53] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 09:34:53.680779102  [2023-09-08 09:34:53] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-08 09:34:53.682170520  [2023-09-08 09:34:53] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 09:36:13.673688832  [2023-09-08 09:36:13] watchdog.front                 ERROR   : No new recording segments were created for front in the last 120s. restarting the ffmpeg record process...
2023-09-08 09:36:13.674072909  [2023-09-08 09:36:13] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-09-08 09:36:13.674076428  [2023-09-08 09:36:13] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:36:13.677236839  [2023-09-08 09:36:13] watchdog.garden                ERROR   : No new recording segments were created for garden in the last 120s. restarting the ffmpeg record process...
2023-09-08 09:36:13.677370993  [2023-09-08 09:36:13] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-09-08 09:36:13.677498801  [2023-09-08 09:36:13] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:36:33.691476845  [2023-09-08 09:36:33] watchdog.back_porch            ERROR   : No new recording segments were created for back_porch in the last 120s. restarting the ffmpeg record process...
2023-09-08 09:36:33.691571571  [2023-09-08 09:36:33] watchdog.back_porch            INFO    : Terminating the existing ffmpeg process...
2023-09-08 09:36:33.691680648  [2023-09-08 09:36:33] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:36:33.703334230  [2023-09-08 09:36:33] watchdog.garage                ERROR   : No new recording segments were created for garage in the last 120s. restarting the ffmpeg record process...
2023-09-08 09:36:33.703338736  [2023-09-08 09:36:33] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-09-08 09:36:33.703754113  [2023-09-08 09:36:33] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:37:09.149883853  [2023-09-08 09:37:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/36.56.mp4 in 0.11868906021118164 seconds.
2023-09-08 09:37:09.732040175  [2023-09-08 09:37:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/36.57.mp4 in 0.14300203323364258 seconds.
2023-09-08 09:38:09.257107063  [2023-09-08 09:38:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/37.56.mp4 in 0.17452478408813477 seconds.
2023-09-08 09:40:29.233440880  [2023-09-08 09:40:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/40.16.mp4 in 0.15251493453979492 seconds.
2023-09-08 09:41:19.212718568  [2023-09-08 09:41:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/41.06.mp4 in 0.135329008102417 seconds.
2023-09-08 09:41:19.845934834  [2023-09-08 09:41:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/41.07.mp4 in 0.1798410415649414 seconds.
2023-09-08 09:41:29.648134256  [2023-09-08 09:41:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/41.17.mp4 in 0.17981791496276855 seconds.
2023-09-08 09:41:39.745858034  [2023-09-08 09:41:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/41.27.mp4 in 0.19469308853149414 seconds.
2023-09-08 09:42:49.233149120  [2023-09-08 09:42:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/42.36.mp4 in 0.14734196662902832 seconds.
2023-09-08 09:42:59.206825985  [2023-09-08 09:42:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/42.46.mp4 in 0.14110708236694336 seconds.
2023-09-08 09:43:09.247061845  [2023-09-08 09:43:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/42.56.mp4 in 0.16416692733764648 seconds.
2023-09-08 09:44:29.254773576  [2023-09-08 09:44:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/44.16.mp4 in 0.15392208099365234 seconds.
2023-09-08 09:44:49.230205105  [2023-09-08 09:44:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/44.36.mp4 in 0.14166498184204102 seconds.
2023-09-08 09:45:19.204245121  [2023-09-08 09:45:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/45.05.mp4 in 0.13314318656921387 seconds.
2023-09-08 09:48:09.676925697  [2023-09-08 09:48:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/47.56.mp4 in 0.16677498817443848 seconds.
2023-09-08 09:48:29.172949652  [2023-09-08 09:48:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/48.15.mp4 in 0.11211895942687988 seconds.
2023-09-08 09:48:39.205689637  [2023-09-08 09:48:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/48.25.mp4 in 0.1309490203857422 seconds.
2023-09-08 09:49:09.237638752  [2023-09-08 09:49:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/48.55.mp4 in 0.15509390830993652 seconds.
2023-09-08 09:49:49.243638246  [2023-09-08 09:49:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/49.35.mp4 in 0.15427017211914062 seconds.
2023-09-08 09:49:59.224577362  [2023-09-08 09:49:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/49.45.mp4 in 0.14029693603515625 seconds.
2023-09-08 09:50:59.238368180  [2023-09-08 09:50:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/50.45.mp4 in 0.15030407905578613 seconds.
2023-09-08 09:51:39.729573266  [2023-09-08 09:51:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/51.25.mp4 in 0.1920299530029297 seconds.
2023-09-08 09:51:49.635947223  [2023-09-08 09:51:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/51.35.mp4 in 0.1354351043701172 seconds.
2023-09-08 09:52:09.749012193  [2023-09-08 09:52:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/51.55.mp4 in 0.18807482719421387 seconds.
2023-09-08 09:53:13.780052761  [2023-09-08 09:53:13] watchdog.garage                INFO    : No frames received from garage in 20 seconds. Exiting ffmpeg...
2023-09-08 09:53:13.780210762  [2023-09-08 09:53:13] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-08 09:53:14.242988659  [2023-09-08 09:53:14] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-08 09:53:14.243208743  [2023-09-08 09:53:14] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-08 09:53:19.427691920  [2023-09-08 09:53:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/53.04.mp4 in 0.16116809844970703 seconds.
2023-09-08 09:53:24.244608043  [2023-09-08 09:53:24] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-08 09:53:24.244769338  [2023-09-08 09:53:24] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-08 09:53:29.385632821  [2023-09-08 09:53:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/53.14.mp4 in 0.1566629409790039 seconds.
2023-09-08 09:53:29.867466756  [2023-09-08 09:53:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/53.15.mp4 in 0.166748046875 seconds.
2023-09-08 09:53:39.783123696  [2023-09-08 09:53:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/53.25.mp4 in 0.19939184188842773 seconds.
2023-09-08 09:54:19.226622352  [2023-09-08 09:54:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/54.05.mp4 in 0.14020609855651855 seconds.
2023-09-08 09:54:29.737871965  [2023-09-08 09:54:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/54.15.mp4 in 0.18522882461547852 seconds.
2023-09-08 09:54:39.709258018  [2023-09-08 09:54:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/54.25.mp4 in 0.1724379062652588 seconds.
2023-09-08 09:54:49.608101358  [2023-09-08 09:54:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/54.35.mp4 in 0.1579909324645996 seconds.
2023-09-08 09:54:59.190882110  [2023-09-08 09:54:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/54.44.mp4 in 0.12395191192626953 seconds.
2023-09-08 09:55:09.757999056  [2023-09-08 09:55:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/54.55.mp4 in 0.1999831199645996 seconds.
2023-09-08 09:55:59.655242935  [2023-09-08 09:55:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/55.45.mp4 in 0.17173409461975098 seconds.
2023-09-08 09:56:09.712398205  [2023-09-08 09:56:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/55.55.mp4 in 0.17791390419006348 seconds.
2023-09-08 09:56:19.381288403  [2023-09-08 09:56:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/56.04.mp4 in 0.1589820384979248 seconds.
2023-09-08 09:56:19.743694256  [2023-09-08 09:56:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/56.05.mp4 in 0.18599200248718262 seconds.
2023-09-08 09:57:09.253341514  [2023-09-08 09:57:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/56.54.mp4 in 0.15602684020996094 seconds.
2023-09-08 09:57:29.755049976  [2023-09-08 09:57:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garden/57.15.mp4 in 0.1876389980316162 seconds.
2023-09-08 09:58:09.396446788  [2023-09-08 09:58:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/57.56.mp4 in 0.16168713569641113 seconds.
2023-09-08 09:58:14.282623612  [2023-09-08 09:58:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/57.57.mp4 in 0.16102004051208496 seconds.
2023-09-08 09:59:09.234889796  [2023-09-08 09:59:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/58.56.mp4 in 0.1336970329284668 seconds.
2023-09-08 09:59:19.266875915  [2023-09-08 09:59:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/59.06.mp4 in 0.1462080478668213 seconds.
2023-09-08 09:59:29.238511405  [2023-09-08 09:59:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/59.16.mp4 in 0.13690900802612305 seconds.
2023-09-08 09:59:39.266273918  [2023-09-08 09:59:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/59.26.mp4 in 0.16192293167114258 seconds.
2023-09-08 09:59:49.277240016  [2023-09-08 09:59:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/back_porch/59.36.mp4 in 0.1575319766998291 seconds.
2023-09-08 09:59:59.465697224  [2023-09-08 09:59:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/front/59.46.mp4 in 0.17541790008544922 seconds.
2023-09-08 09:59:59.767047588  [2023-09-08 09:59:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/14/garage/59.47.mp4 in 0.14853405952453613 seconds.
2023-09-08 10:01:09.399916040  [2023-09-08 10:01:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/front/00.56.mp4 in 0.14545798301696777 seconds.
2023-09-08 10:01:09.748236303  [2023-09-08 10:01:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garage/00.57.mp4 in 0.18093419075012207 seconds.
2023-09-08 10:01:19.464472473  [2023-09-08 10:01:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/front/01.06.mp4 in 0.17302393913269043 seconds.
2023-09-08 10:01:49.400683941  [2023-09-08 10:01:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/front/01.36.mp4 in 0.15331220626831055 seconds.
2023-09-08 10:01:54.273092249  [2023-09-08 10:01:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garage/01.37.mp4 in 0.15282607078552246 seconds.
2023-09-08 10:01:59.423063855  [2023-09-08 10:01:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/front/01.46.mp4 in 0.16459202766418457 seconds.
2023-09-08 10:02:14.307676164  [2023-09-08 10:02:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garage/02.03.mp4 in 0.172288179397583 seconds.
2023-09-08 10:02:19.856373075  [2023-09-08 10:02:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garden/02.07.mp4 in 0.20841717720031738 seconds.
2023-09-08 10:02:29.437791555  [2023-09-08 10:02:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/front/02.16.mp4 in 0.1637098789215088 seconds.
2023-09-08 10:02:54.357858422  [2023-09-08 10:02:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garage/02.42.mp4 in 0.19031310081481934 seconds.
2023-09-08 10:02:59.400052626  [2023-09-08 10:02:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/front/02.45.mp4 in 0.16088008880615234 seconds.
2023-09-08 10:03:04.302674987  [2023-09-08 10:03:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garage/02.52.mp4 in 0.17255210876464844 seconds.
2023-09-08 10:03:29.776633118  [2023-09-08 10:03:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garden/03.17.mp4 in 0.20196199417114258 seconds.
2023-09-08 10:03:39.297407826  [2023-09-08 10:03:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/back_porch/03.26.mp4 in 0.15405511856079102 seconds.
2023-09-08 10:03:49.238930935  [2023-09-08 10:03:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/back_porch/03.36.mp4 in 0.12685489654541016 seconds.
2023-09-08 10:03:59.860843621  [2023-09-08 10:03:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/garden/03.46.mp4 in 0.28009700775146484 seconds.
2023-09-08 10:04:09.254191554  [2023-09-08 10:04:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-08/15/back_porch/03.56.mp4 in 0.13859295845031738 seconds.
2023-09-08 10:04:09.875887667  [2023-09-08 10:04:09] frigate.record                 ERROR   : Unable to convert /tmp/cache/garden-20230908100356.mp4 to /media/frigate/recordings/2023-09-08/15/garden/03.56.mp4
2023-09-08 10:04:09.876035533  [2023-09-08 10:04:09] frigate.record                 ERROR   : ffmpeg version n5.1-2-g915ef932a3-20220731 Copyright (c) 2000-2022 the FFmpeg developers
2023-09-08 10:04:09.876038324    built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
2023-09-08 10:04:09.876079687    configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
2023-09-08 10:04:09.876095959    libavutil      57. 28.100 / 57. 28.100
2023-09-08 10:04:09.876097827    libavcodec     59. 37.100 / 59. 37.100
2023-09-08 10:04:09.876099615    libavformat    59. 27.100 / 59. 27.100
2023-09-08 10:04:09.876101299    libavdevice    59.  7.100 / 59.  7.100
2023-09-08 10:04:09.876125295    libavfilter     8. 44.100 /  8. 44.100
2023-09-08 10:04:09.876126956    libswscale      6.  7.100 /  6.  7.100
2023-09-08 10:04:09.876128274    libswresample   4.  7.100 /  4.  7.100
2023-09-08 10:04:09.876129703    libpostproc    56.  6.100 / 56.  6.100
2023-09-08 10:04:09.876132110  Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/tmp/cache/garden-20230908100356.mp4':
2023-09-08 10:04:09.876133378    Metadata:
2023-09-08 10:04:09.876134746      major_brand     : isom
2023-09-08 10:04:09.876136226      minor_version   : 512
2023-09-08 10:04:09.876137669      compatible_brands: isomiso2avc1mp41
2023-09-08 10:04:09.876173865      title           : go2rtc/1.6.2
2023-09-08 10:04:09.876175808      encoder         : Lavf59.27.100
2023-09-08 10:04:09.876177452    Duration: 00:00:09.99, start: 0.000000, bitrate: 8460 kb/s
2023-09-08 10:04:09.876179595    Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 2560x1920, 8404 kb/s, 30.05 fps, 30 tbr, 90k tbn (default)
2023-09-08 10:04:09.876181069      Metadata:
2023-09-08 10:04:09.876182487        handler_name    : VideoHandler
2023-09-08 10:04:09.876183934        vendor_id       : [0][0][0][0]
2023-09-08 10:04:09.876185784    Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, mono, fltp, 59 kb/s (default)
2023-09-08 10:04:09.876187017      Metadata:
2023-09-08 10:04:09.876188399        handler_name    : SoundHandler
2023-09-08 10:04:09.876203034        vendor_id       : [0][0][0][0]
2023-09-08 10:04:09.876204821  Output #0, mp4, to '/media/frigate/recordings/2023-09-08/15/garden/03.56.mp4':
2023-09-08 10:04:09.876206042    Metadata:
2023-09-08 10:04:09.876207357      major_brand     : isom
2023-09-08 10:04:09.876208693      minor_version   : 512
2023-09-08 10:04:09.876210052      compatible_brands: isomiso2avc1mp41
2023-09-08 10:04:09.876211412      title           : go2rtc/1.6.2
2023-09-08 10:04:09.876236940      encoder         : Lavf59.27.100
2023-09-08 10:04:09.876239270    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 2560x1920, q=2-31, 8404 kb/s, 30.05 fps, 30 tbr, 90k tbn (default)
2023-09-08 10:04:09.876240524      Metadata:
2023-09-08 10:04:09.876241874        handler_name    : VideoHandler
2023-09-08 10:04:09.876243280        vendor_id       : [0][0][0][0]
2023-09-08 10:04:09.876245062    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, mono, fltp, 59 kb/s (default)
2023-09-08 10:04:09.876246447      Metadata:
2023-09-08 10:04:09.876247790        handler_name    : SoundHandler
2023-09-08 10:04:09.876249158        vendor_id       : [0][0][0][0]
2023-09-08 10:04:09.876250339  Stream mapping:
2023-09-08 10:04:09.876251665    Stream #0:0 -> #0:0 (copy)
2023-09-08 10:04:09.876252940    Stream #0:1 -> #0:1 (copy)
2023-09-08 10:04:09.876254239  Press [q] to stop, [?] for help
2023-09-08 10:04:09.876256009  frame=    1 fps=0.0 q=-1.0 size=       0kB time=00:00:00.00 bitrate=34909.1kbits/s speed=N/A    
2023-09-08 10:04:09.876257491  av_interleaved_write_frame(): No space left on device
2023-09-08 10:04:09.876282524  [mp4 @ 0x5622844eeac0] Starting second pass: moving the moov atom to the beginning of the file
2023-09-08 10:04:09.876284531  Error writing trailer of /media/frigate/recordings/2023-09-08/15/garden/03.56.mp4: No space left on device
2023-09-08 10:04:09.876286325  frame=    1 fps=0.0 q=-1.0 Lsize=     512kB time=00:00:00.07 bitrate=57461.5kbits/s speed=56.6x    
2023-09-08 10:04:09.876288089  video:1207kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
2023-09-08 10:04:09.876289864  Error closing file /media/frigate/recordings/2023-09-08/15/garden/03.56.mp4: No space left on device
2023-09-08 10:04:09.876291101  Conversion failed!

@NickM-27 Not sure if I should create a new issue for this - I recreated the VM, everything is kept as same as they can be, I'm still running into a mountain of issues:

2023-09-09 16:52:51.715190002  [2023-09-09 16:52:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/52.35.mp4 in 0.1154789924621582 seconds.
2023-09-09 16:53:01.011497303  [2023-09-09 16:53:01] ffmpeg.front.record            ERROR   : [segment @ 0x55d42d904a40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-09 16:53:01.011671157  [2023-09-09 16:53:01] ffmpeg.front.record            ERROR   : [segment @ 0x55d42d904a40] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
2023-09-09 16:53:01.011739382  [2023-09-09 16:53:01] ffmpeg.front.record            ERROR   : rtsp://127.0.0.1:8554/front?video=copy&audio=aac: Connection timed out
2023-09-09 16:53:01.012057880  [2023-09-09 16:53:01] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:01.012368992  [2023-09-09 16:53:01] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:01.037834335  [2023-09-09 16:53:01] ffmpeg.garden.record           ERROR   : [segment @ 0x5618fe51d740] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-09 16:53:01.046353992  [2023-09-09 16:53:01] ffmpeg.back_porch.record       ERROR   : [segment @ 0x561439c8b9c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-09 16:53:01.046359063  [2023-09-09 16:53:01] ffmpeg.back_porch.record       ERROR   : [segment @ 0x561439c8b9c0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
2023-09-09 16:53:01.046384412  [2023-09-09 16:53:01] ffmpeg.back_porch.record       ERROR   : rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac: Connection timed out
2023-09-09 16:53:01.046387226  [2023-09-09 16:53:01] watchdog.back_porch            INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:01.046389303  [2023-09-09 16:53:01] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:01.046416223  [2023-09-09 16:53:01] ffmpeg.garden.record           ERROR   : [segment @ 0x5618fe51d740] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
2023-09-09 16:53:01.046449974  [2023-09-09 16:53:01] ffmpeg.garden.record           ERROR   : rtsp://127.0.0.1:8554/garden?video=copy&audio=aac: Connection timed out
2023-09-09 16:53:01.046452206  [2023-09-09 16:53:01] watchdog.garden                INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:01.046453994  [2023-09-09 16:53:01] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:01.047286130  [2023-09-09 16:53:01] ffmpeg.garage.record           ERROR   : [segment @ 0x5591d916d040] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-09 16:53:01.047290345  [2023-09-09 16:53:01] ffmpeg.garage.record           ERROR   : [segment @ 0x5591d916d040] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
2023-09-09 16:53:01.047292803  [2023-09-09 16:53:01] ffmpeg.garage.record           ERROR   : rtsp://127.0.0.1:8554/garage?video=copy&audio=aac: Connection timed out
2023-09-09 16:53:01.047295067  [2023-09-09 16:53:01] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:01.047297111  [2023-09-09 16:53:01] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:01.676099664  [2023-09-09 16:53:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/52.45.mp4 in 0.09118294715881348 seconds.
2023-09-09 16:53:02.011945162  [2023-09-09 16:53:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/52.47.mp4 in 0.10941290855407715 seconds.
2023-09-09 16:53:03.910221424  [2023-09-09 16:53:03] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-09 16:53:03.911020620  [2023-09-09 16:53:03] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-09 16:53:04.202174983  [2023-09-09 16:53:04] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-09-09 16:53:04.202390614  [2023-09-09 16:53:04] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-09-09 16:53:04.296485041  [2023-09-09 16:53:04] frigate.video                  ERROR   : front: Unable to read frames from ffmpeg process.
2023-09-09 16:53:04.297299756  [2023-09-09 16:53:04] frigate.video                  ERROR   : front: ffmpeg process is not running. exiting capture thread...
2023-09-09 16:53:04.299500782  [2023-09-09 16:53:04] frigate.video                  ERROR   : back_porch: Unable to read frames from ffmpeg process.
2023-09-09 16:53:04.299738866  [2023-09-09 16:53:04] frigate.video                  ERROR   : back_porch: ffmpeg process is not running. exiting capture thread...
2023-09-09 16:53:11.021872987  [2023-09-09 16:53:11] watchdog.front                 ERROR   : Ffmpeg process crashed unexpectedly for front.
2023-09-09 16:53:11.022071893  [2023-09-09 16:53:11] watchdog.front                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-09 16:53:11.022203566  [2023-09-09 16:53:11] ffmpeg.front.detect            ERROR   : rtsp://127.0.0.1:8554/front_sub?video=copy: Connection timed out
2023-09-09 16:53:11.022223380  [2023-09-09 16:53:11] ffmpeg.front.detect            ERROR   :     Last message repeated 1 times
2023-09-09 16:53:11.044916252  [2023-09-09 16:53:11] watchdog.back_porch            ERROR   : Ffmpeg process crashed unexpectedly for back_porch.
2023-09-09 16:53:11.045037834  [2023-09-09 16:53:11] watchdog.back_porch            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-09 16:53:11.045253123  [2023-09-09 16:53:11] ffmpeg.back_porch.detect       ERROR   : rtsp://127.0.0.1:8554/back_porch_sub?video=copy: Connection timed out
2023-09-09 16:53:11.045355306  [2023-09-09 16:53:11] ffmpeg.back_porch.detect       ERROR   :     Last message repeated 1 times
2023-09-09 16:53:11.048573849  [2023-09-09 16:53:11] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-09 16:53:11.048578508  [2023-09-09 16:53:11] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-09 16:53:11.048580586  [2023-09-09 16:53:11] ffmpeg.garage.detect           ERROR   : rtsp://127.0.0.1:8554/garage_sub?video=copy: Connection timed out
2023-09-09 16:53:11.048582451  [2023-09-09 16:53:11] ffmpeg.garage.detect           ERROR   :     Last message repeated 1 times
2023-09-09 16:53:11.056432859  [2023-09-09 16:53:11] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-09-09 16:53:11.056437397  [2023-09-09 16:53:11] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-09 16:53:11.056439925  [2023-09-09 16:53:11] ffmpeg.garden.detect           ERROR   : rtsp://127.0.0.1:8554/garden_sub?video=copy: Connection timed out
2023-09-09 16:53:11.056441778  [2023-09-09 16:53:11] ffmpeg.garden.detect           ERROR   :     Last message repeated 1 times
2023-09-09 16:53:11.058580714  [2023-09-09 16:53:11] ffmpeg.garage.record           ERROR   : rtsp://127.0.0.1:8554/garage?video=copy&audio=aac: Invalid data found when processing input
2023-09-09 16:53:11.058585746  [2023-09-09 16:53:11] watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:11.058588189  [2023-09-09 16:53:11] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:11.061421207  [2023-09-09 16:53:11] ffmpeg.back_porch.record       ERROR   : rtsp://127.0.0.1:8554/back_porch?video=copy&audio=aac: Invalid data found when processing input
2023-09-09 16:53:11.061426283  [2023-09-09 16:53:11] watchdog.back_porch            INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:11.061428215  [2023-09-09 16:53:11] watchdog.back_porch            INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:11.517074135  [2023-09-09 16:53:11] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/front-20230909165311.mp4
2023-09-09 16:53:21.026639035  [2023-09-09 16:53:21] ffmpeg.front.record            ERROR   : [rtsp @ 0x55acbe292f00] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
2023-09-09 16:53:21.026752448  [2023-09-09 16:53:21] ffmpeg.front.record            ERROR   : Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
2023-09-09 16:53:21.026863139  [2023-09-09 16:53:21] ffmpeg.front.record            ERROR   : rtsp://127.0.0.1:8554/front?video=copy&audio=aac: Connection timed out
2023-09-09 16:53:21.026968127  [2023-09-09 16:53:21] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:21.027089833  [2023-09-09 16:53:21] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:26.775287220  [2023-09-09 16:53:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/53.13.mp4 in 0.1332230567932129 seconds.
2023-09-09 16:53:27.020311053  [2023-09-09 16:53:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/53.13.mp4 in 0.12800097465515137 seconds.
2023-09-09 16:53:27.384720787  [2023-09-09 16:53:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garden/53.12.mp4 in 0.18088603019714355 seconds.
2023-09-09 16:53:31.531394633  [2023-09-09 16:53:31] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/front-20230909165331.mp4
2023-09-09 16:53:41.037163088  [2023-09-09 16:53:41] ffmpeg.front.record            ERROR   : [rtsp @ 0x562c496a9f00] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
2023-09-09 16:53:41.037301539  [2023-09-09 16:53:41] ffmpeg.front.record            ERROR   : Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
2023-09-09 16:53:41.037426646  [2023-09-09 16:53:41] ffmpeg.front.record            ERROR   : rtsp://127.0.0.1:8554/front?video=copy&audio=aac: Connection timed out
2023-09-09 16:53:41.037558107  [2023-09-09 16:53:41] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:53:41.037640262  [2023-09-09 16:53:41] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:53:51.528973686  [2023-09-09 16:53:51] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/front-20230909165351.mp4
2023-09-09 16:54:01.044931566  [2023-09-09 16:54:01] ffmpeg.front.record            ERROR   : [rtsp @ 0x56550aa65f00] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
2023-09-09 16:54:01.045172223  [2023-09-09 16:54:01] ffmpeg.front.record            ERROR   : Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
2023-09-09 16:54:01.045176187  [2023-09-09 16:54:01] ffmpeg.front.record            ERROR   : rtsp://127.0.0.1:8554/front?video=copy&audio=aac: Connection timed out
2023-09-09 16:54:01.045394210  [2023-09-09 16:54:01] watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2023-09-09 16:54:01.045397193  [2023-09-09 16:54:01] watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-09 16:54:11.256132055  [2023-09-09 16:54:11] frigate.storage                ERROR   : Could not clear 12383.35 currently 210.50000000000003, retained recordings must be deleted.
2023-09-09 16:54:16.778689114  [2023-09-09 16:54:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/54.04.mp4 in 0.13643193244934082 seconds.
2023-09-09 16:54:26.785538970  [2023-09-09 16:54:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/54.14.mp4 in 0.14322304725646973 seconds.
2023-09-09 16:54:27.211227442  [2023-09-09 16:54:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/54.15.mp4 in 0.14078402519226074 seconds.
2023-09-09 16:54:36.920765100  [2023-09-09 16:54:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/front/54.24.mp4 in 0.15339088439941406 seconds.
2023-09-09 16:54:41.766517622  [2023-09-09 16:54:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/54.25.mp4 in 0.13355684280395508 seconds.
2023-09-09 16:54:46.766963451  [2023-09-09 16:54:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/54.34.mp4 in 0.1320960521697998 seconds.
2023-09-09 16:54:47.228040754  [2023-09-09 16:54:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garden/54.34.mp4 in 0.17151379585266113 seconds.
2023-09-09 16:55:15.051904099  [2023-09-09 16:55:15] frigate.http                   ERROR   : No recordings found for the requested time range
2023-09-09 16:55:15.070208203  [2023-09-09 16:55:15] frigate.http                   ERROR   : Event does not have recordings: 1694295877.026649-sq3xc1
2023-09-09 16:55:16.108153635  [2023-09-09 16:55:16] frigate.http                   ERROR   : Event does not have recordings: 1694295877.026649-sq3xc1
2023-09-09 16:56:56.933363831  [2023-09-09 16:56:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/front/56.44.mp4 in 0.14562511444091797 seconds.
2023-09-09 16:56:57.251983572  [2023-09-09 16:56:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/56.45.mp4 in 0.14220404624938965 seconds.
2023-09-09 16:57:07.223827935  [2023-09-09 16:57:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garden/56.54.mp4 in 0.1711559295654297 seconds.
2023-09-09 16:57:16.945746899  [2023-09-09 16:57:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/front/57.04.mp4 in 0.1631789207458496 seconds.
2023-09-09 16:57:17.254658197  [2023-09-09 16:57:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garden/57.04.mp4 in 0.15850400924682617 seconds.
2023-09-09 16:57:21.805051474  [2023-09-09 16:57:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/57.05.mp4 in 0.14785408973693848 seconds.
2023-09-09 16:57:27.043071413  [2023-09-09 16:57:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/57.16.mp4 in 0.13039302825927734 seconds.
2023-09-09 16:57:36.776669438  [2023-09-09 16:57:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/57.24.mp4 in 0.13633203506469727 seconds.
2023-09-09 16:57:37.198528588  [2023-09-09 16:57:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/57.25.mp4 in 0.13718795776367188 seconds.
2023-09-09 16:57:46.781597276  [2023-09-09 16:57:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/57.34.mp4 in 0.13717293739318848 seconds.
2023-09-09 16:57:56.783326050  [2023-09-09 16:57:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/57.44.mp4 in 0.13543200492858887 seconds.
2023-09-09 16:58:06.800539837  [2023-09-09 16:58:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/57.54.mp4 in 0.16049695014953613 seconds.
2023-09-09 16:58:07.397445695  [2023-09-09 16:58:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garden/57.53.mp4 in 0.16087794303894043 seconds.
2023-09-09 16:58:16.766553949  [2023-09-09 16:58:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/58.04.mp4 in 0.13731718063354492 seconds.
2023-09-09 16:59:11.315329191  [2023-09-09 16:59:11] frigate.storage                ERROR   : Could not clear 12366.580000000002 currently 177.10000000000002, retained recordings must be deleted.
2023-09-09 16:59:17.050599666  [2023-09-09 16:59:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/59.04.mp4 in 0.13213706016540527 seconds.
2023-09-09 16:59:31.801053404  [2023-09-09 16:59:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garage/59.16.mp4 in 0.14418411254882812 seconds.
2023-09-09 16:59:57.296938592  [2023-09-09 16:59:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/garden/59.43.mp4 in 0.17932987213134766 seconds.
2023-09-09 17:00:06.788428970  [2023-09-09 17:00:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-09-09/21/back_porch/59.53.mp4 in 0.1376199722290039 seconds.

You can create a new issue, looks like issues outside of frigate though (check go2rtc logs) Connection timed out

blakeblackshear / frigate