search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.09k stars 1.65k forks source link

[Support]: Database is locked occurs randomly #7182

Closed YBonline closed 1 year ago

YBonline commented 1 year ago

Describe the problem you are having

I've been having a "Database is locked" error randomly on the latest dev build. I haven't been able to figure out a cause yet, but it happens much more frequently when my storage drive for the videos is full and Frigate is deleting them continuously to free up space for the next recording. It does still happen without it.

After the database is locked error occurs, the Home Assistant's object count becomes permanently too high. Even restarting Frigate does not cause it to return to normal (although restarting the plugin in HA without restarting Frigate does fix it) Frigate does seem to start working normally again after just dropping some recordings while the database is locked.

Version

0.13.0-8E584CF

Frigate config file

ui:
  use_experimental: true

database:
  path: /db/frigate.db

birdseye:
  enabled: True
#  restream: True
  mode: objects
  width: 1920
  height: 1200

mqtt:
  host: 10.10.22.11
  user: frigate
  password: frigatePASS1

detectors:
  coral:
    type: edgetpu

record:
  enabled: True
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 21
      mode: active_objects

objects:
  track:
    - person
    - bicycle
    - car
    - cat
    - dog
  filters:
    person:
      # Optional: minimum width*height of the bounding box for the detected object (default: 0)
      min_area: 500
      # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
      max_area: 100000
      # Optional: minimum width/height of the bounding box for the detected object (default: 0)
      min_ratio: 0.5
      # Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
      max_ratio: 2.0
      # Optional: minimum score for the object to initiate tracking (default: shown below)
      min_score: 0.60
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.75

audio:
  enabled: false
  listen:
    - bark
    - scream
    - speech
    - yell
    - goat
    - chicken
    - skidding
    - tire_squeal
    - emergency_vehicle
    - smoke_detector
    - gunshot
    - glass
    - crack
    - shatter
    - smash
    - breaking

snapshots:
  enabled: true
  clean_copy: True
  timestamp: False
  bounding_box: True
  crop: True
  retain:
    default: 180

go2rtc:
  streams:
[...]

  webrtc:
    candidates:
      - 10.10.22.11:8555
      - stun:8555

cameras:
  warehouse_exterior_doorbell:
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_doorbell
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 2560
      height: 1920
    motion:
      mask:
        - 0,0,2560,0,2560,986,2045,956,2034,1016,2406,1054,2332,1121,371,1118,357,910,0,846

  warehouse_exterior_nw:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_nw
          roles:
            - record
        #- path: rtsp://127.0.0.1:8554/warehouse_exterior_nw_sub
        #  roles:
            - detect
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: False
    detect:
      width: 1920
      height: 1080

  warehouse_exterior_sw:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_sw
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_sw_sub
          roles:
            - detect
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: False
    detect:
      width: 1920
      height: 1080

  warehouse_exterior_se:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_se
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_se_sub
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: False
    detect:
      width: 480
      height: 360
    motion:
      mask:
        - 480,0,480,55,109,50,0,80,0,0
#    detect:
#      width: 2592
#      height: 1944

  warehouse_exterior_front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_exterior_front_sub
          roles:
            - record
        #- path: rtsp://127.0.0.1:8554/warehouse_exterior_front_sub
        #  roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 1536
      height: 576
    #birdseye:
    #  enabled: False
    #detect:
    #  width: 4608
    #  height: 1728

  warehouse_workbench:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_workbench
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/warehouse_workbench_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    #detect:
    #  width: 2560
    #  height: 1920
    detect:
      width: 640
      height: 480
    motion:
      mask:
        - 220,480,100,480,98,459,217,453

  warehouse_interior_front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_interior_front
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/warehouse_interior_front_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    #detect:
    #  width: 2560
    #  height: 1920
    detect:
      width: 640
      height: 480
    motion:
      mask:
        - 376,0,397,31,205,27,205,0

  warehouse_interior_office:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_interior_office
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/warehouse_interior_office_sub
          roles:
            - detect
            - audio
#          input_args: -ss 00:01
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 704
      height: 480
    motion:
      mask:
        - 682,21,683,52,418,47,405,22
#    detect:
#      width: 3840
#      height: 2160

  warehouse_interior_overview:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/warehouse_interior_overview
          roles:
            - record
#        - path: rtsp://127.0.0.1:8554/warehouse_interior_overview_sub
#          roles:
#            - detect
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: False
    detect:
      width: 1920
      height: 1080
#    detect:
#      width: 704
#      height: 480

  gp_clubhouse_gym_1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_1
          roles:
            - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_1_sub
#          roles:
            - detect
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 2592
      height: 1944
#    detect:
#      width: 480
#      height: 360
    audio:
      enabled: False
    motion:
      mask:
        - 2592,173,2592,0,0,0,0,337
    objects:
      filters:
        bicycle:
          mask:
            - 1635,80,1886,1019,408,1059,350,177
        motorcycle:
          mask:
            - 1635,80,1886,1019,408,1059,350,177

  gp_clubhouse_gym_2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_2
          roles:
            - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_2_sub
#          roles:
            - detect
      hwaccel_args: preset-nvidia-h265
#    detect:
#      width: 480
#      height: 360
    detect:
      width: 2592
      height: 1944
    audio:
      enabled: False
    motion:
      mask:
        - 2592,0,0,0,0,385
    objects:
      filters:
        bicycle:
          mask:
            - 2391,1369,1261,677,1331,135,2542,274
        motorcycle:
          mask:
            - 2391,1369,1261,677,1331,135,2542,274

  gp_clubhouse_front_door:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_front_door
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_front_door_sub
          roles:
            - detect
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 480
      height: 360
#    detect:
#      width: 2592
#      height: 1944
    audio:
      enabled: False
    motion:
      mask:
        - 0,0,0,34,265,30,287,0

  gp_clubhouse_back_door:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_back_door
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_back_door_sub
          roles:
            - detect
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 480
      height: 360
    audio:
      enabled: False
    motion:
      mask:
        - 480,0,480,45,0,56,0,0
#    detect:
#      width: 2592
#      height: 1944

  gp_clubhouse_banquet:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_banquet
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_banquet_sub
          roles:
            - detect
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 480
      height: 360
    audio:
      enabled: False
    motion:
      mask:
        - 0,0,0,31,277,31,373,0
#    detect:
#      width: 2592
#      height: 1944

  gp_clubhouse_pool_porch:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool_porch
          roles:
            - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool_porch_sub
#          roles:
            - detect
      hwaccel_args: preset-nvidia-h264
    zones:
      gp_clubhouse_pool_deck:
        coordinates: 1920,0,1920,1080,0,1080,0,0
    detect:
      width: 1920
      height: 1080
    audio:
      enabled: False
    motion:
      mask:
        - 1000,130,674,122,678,76,993,67
#    detect:
#      width: 704
#      height: 480

  gp_clubhouse_pool:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
#    zones:
#      gp_clubhouse_pool_deck:
#        coordinates: 5120,1440,5120,0,2701,0,2004,1440
    detect:
#      width: 5120
#      height: 1440
      width: 1920
      height: 536

  gp_clubhouse_firepit:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_firepit
          roles:
            - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_firepit_sub
#          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    motion:
      mask:
        - 2211,0,0,0,0,495,1795,224,1795,67,2202,62
        - 2061,1203,1952,361,1771,361,1702,1182
    objects:
      filters:
        person:
          mask:
            - 2114,1213,1610,1210,1612,366,2080,361
    zones:
      gp_clubhouse_pool_deck:
        coordinates: 3840,2160,3657,599,1531,331,0,725,0,2160
    detect:
      width: 3840
      height: 2160

  gp_clubhouse_parking:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_clubhouse_parking
          roles:
            - record
        - path: rtsp://admin:campass123@10.10.23.30:554/Streaming/channels/102
        #- path: rtsp://127.0.0.1:8554/gp_clubhouse_parking_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
#      width: 5120
#      height: 1440

  gp636_doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp636_doorbell
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 1920
      height: 1080
    motion:
      mask:
        - 0,0,1667,0,1643,67,928,361,926,599,0,499

  gp525_doorbell:
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp525_doorbell
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 2560
      height: 1920
    motion:
      mask:
        - 1641,0,827,995,333,1019,173,861,0,904,0,0

  gp_entrance_75:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_entrance_75
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 1080
#    record:
#      retain:
#        days: 5
#        mode: all
#      events:
#        retain:
#          default: 45

  gp_entrance_archer:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_entrance_archer
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 1080
    record:
      retain:
        days: 5
        mode: all
      events:
        retain:
          default: 45

  gp_bldg_08_front:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_bldg_08_front
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_bldg_08_front_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
#      width: 5120
#      height: 1440
#    zones:
#      gp_bldg_alert:
#        coordinates: 0,961,0,1440,5120,1440,5120,1030,4090,139,1293,0

  gp_bldg_08_rear:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_bldg_08_rear
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_bldg_08_rear_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
#      width: 5120
#      height: 1440
#    zones:
#      gp_bldg_alert:
#        coordinates: 5120,1440,0,1440,0,767,1113,393,2685,308,4223,514,5120,976

  gp_bldg_03_side:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gp_bldg_03_side
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gp_bldg_03_side_sub
          roles:
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
    motion:
      mask:
        - 764,93,1255,111,1920,260,1920,0,0,0,0,175
#      width: 5120
#      height: 1440
    zones:
      gp_bldg_alert:
        coordinates: 0,536,0,306,240,239,1189,370,1352,536

Relevant log output

2023-07-15 21:55:19.360407809  [INFO] Preparing Frigate...
2023-07-15 21:55:19.395359015  [INFO] Starting Frigate...
[...]
2023-07-16 02:00:34.962268248  [2023-07-16 02:00:34] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-16 02:00:34.963699927  [2023-07-16 02:00:34] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-16 02:00:39.748448339  [2023-07-16 02:00:39] watchdog.warehouse_workbench   ERROR   : Ffmpeg process crashed unexpectedly for warehouse_workbench.
2023-07-16 02:00:39.748682491  [2023-07-16 02:00:39] watchdog.warehouse_workbench   ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-16 02:00:39.748795780  [2023-07-16 02:00:39] ffmpeg.warehouse_workbench.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_workbench_sub: Invalid data found when processing input
2023-07-16 02:00:39.920583749  [2023-07-16 02:00:39] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-16 02:00:39.920770389  [2023-07-16 02:00:39] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-16 02:00:39.920879055  [2023-07-16 02:00:39] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Invalid data found when processing input
2023-07-16 05:08:54.880840806  Exception in thread event_cleanup:
2023-07-16 05:08:54.880957267  Traceback (most recent call last):
2023-07-16 05:08:54.880964388    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-16 05:08:54.885747660      cursor.execute(sql, params or ())
2023-07-16 05:08:54.885934081  sqlite3.OperationalError: database is locked
2023-07-16 05:08:54.885942699  
2023-07-16 05:08:54.885950139  During handling of the above exception, another exception occurred:
2023-07-16 05:08:54.885955892  
2023-07-16 05:08:54.885994303  Traceback (most recent call last):
2023-07-16 05:08:54.886001108    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-16 05:08:54.887289177      self.run()
2023-07-16 05:08:54.887301351    File "/opt/frigate/frigate/events/cleanup.py", line 165, in run
2023-07-16 05:08:54.887781373      self.expire("snapshots")
2023-07-16 05:08:54.887789574    File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-16 05:08:54.887962952      update_query.execute()
2023-07-16 05:08:54.887970011    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-16 05:08:54.888868107      return method(self, database, *args, **kwargs)
2023-07-16 05:08:54.888873947    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-16 05:08:54.889780318      return self._execute(database)
2023-07-16 05:08:54.889799574    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-16 05:08:54.891055401      return self.handle_result(database, cursor)
2023-07-16 05:08:54.891077653    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-16 05:08:54.892213170      return database.rows_affected(cursor)
2023-07-16 05:08:54.892218655    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-16 05:08:54.893860338      return cursor.rowcount
2023-07-16 05:08:54.893880817    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-16 05:08:54.894159169      self._wait()
2023-07-16 05:08:54.894178540    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-16 05:08:54.894411260      raise self._exc
2023-07-16 05:08:54.894416942    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-16 05:08:54.894601830      cursor = self.database._execute(obj.sql, obj.params)
2023-07-16 05:08:54.894636415    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-16 05:08:54.896070653      cursor.execute(sql, params or ())
2023-07-16 05:08:54.896076514    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-16 05:08:54.898242283      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-16 05:08:54.898291171    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-16 05:08:54.899354803      raise value.with_traceback(tb)
2023-07-16 05:08:54.899374907    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-16 05:08:54.901400803      cursor.execute(sql, params or ())
2023-07-16 05:08:54.901666460  peewee.OperationalError: database is locked

FFprobe output from your camera

NA

Frigate stats

{"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"18":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"41":{"cmdline":"/package/admin/s6-2.11.3.0/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"42":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"80":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"81":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"82":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"89":{"cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"64.9","cpu_average":"123","mem":"0.1"},"98":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"99":{"cmdline":"python3 -u -m frigate","cpu":"19.9","cpu_average":"32","mem":"3.3"},"119":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"145":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"146":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"147":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"148":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"153":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"166":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"191":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"223":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"256":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"292":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"325":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"362":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"397":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"437":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"475":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"511":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"549":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"585":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"621":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"659":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"693":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"724":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"762":{"cmdline":"nginx: worker process","cpu":"0.3","cpu_average":"0","mem":"0.0"},"800":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"841":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"878":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"916":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"953":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"991":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1028":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1066":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1099":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1300":{"cmdline":"frigate.logger       ","cpu":"0.0","cpu_average":"0","mem":"0.2"},"1301":{"cmdline":"frigate.recording_manager","cpu":"112.4","cpu_average":"123","mem":"1.3"},"1310":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(87)","cpu":"1.7","cpu_average":"3","mem":"0.0"},"1311":{"cmdline":"frigate.detector.coral","cpu":"61.6","cpu_average":"78","mem":"0.3"},"1312":{"cmdline":"frigate.output       ","cpu":"164.4","cpu_average":"196","mem":"0.3"},"1319":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2560x1920 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1320":{"cmdline":"frigate.process:warehouse_exterior_nw","cpu":"2.1","cpu_average":"10","mem":"0.3"},"1321":{"cmdline":"frigate.process:warehouse_exterior_sw","cpu":"44.4","cpu_average":"39","mem":"0.3"},"1322":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1323":{"cmdline":"frigate.process:warehouse_exterior_se","cpu":"50.9","cpu_average":"3","mem":"0.3"},"1324":{"cmdline":"frigate.process:warehouse_exterior_front","cpu":"28.5","cpu_average":"36","mem":"0.3"},"1325":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1326":{"cmdline":"frigate.process:warehouse_workbench","cpu":"49.5","cpu_average":"4","mem":"0.3"},"1327":{"cmdline":"frigate.process:warehouse_interior_front","cpu":"43.3","cpu_average":"3","mem":"0.3"},"1328":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1329":{"cmdline":"frigate.process:warehouse_interior_office","cpu":"2.1","cpu_average":"3","mem":"0.3"},"1330":{"cmdline":"frigate.process:warehouse_interior_overview","cpu":"15.8","cpu_average":"5","mem":"0.3"},"1331":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1536x576 -i pipe: -f mpegts -s 1920x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1332":{"cmdline":"frigate.process:gp_clubhouse_gym_1","cpu":"2.4","cpu_average":"46","mem":"0.3"},"1333":{"cmdline":"frigate.process:gp_clubhouse_gym_2","cpu":"1.7","cpu_average":"52","mem":"0.4"},"1334":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1335":{"cmdline":"frigate.process:gp_clubhouse_front_door","cpu":"2.1","cpu_average":"3","mem":"0.3"},"1336":{"cmdline":"frigate.process:gp_clubhouse_back_door","cpu":"2.1","cpu_average":"3","mem":"0.3"},"1337":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1338":{"cmdline":"frigate.process:gp_clubhouse_banquet","cpu":"4.5","cpu_average":"7","mem":"0.3"},"1339":{"cmdline":"frigate.process:gp_clubhouse_pool_porch","cpu":"2.1","cpu_average":"9","mem":"0.3"},"1340":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 704x480 -i pipe: -f mpegts -s 1056x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1341":{"cmdline":"frigate.process:gp_clubhouse_pool","cpu":"19.3","cpu_average":"26","mem":"0.3"},"1342":{"cmdline":"frigate.process:gp_clubhouse_firepit","cpu":"21.3","cpu_average":"7","mem":"0.3"},"1343":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1344":{"cmdline":"frigate.process:gp_clubhouse_parking","cpu":"16.2","cpu_average":"33","mem":"0.3"},"1345":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1346":{"cmdline":"frigate.process:gp636_doorbell","cpu":"11.0","cpu_average":"20","mem":"0.3"},"1347":{"cmdline":"frigate.process:gp_entrance_75","cpu":"5.8","cpu_average":"31","mem":"0.3"},"1348":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1349":{"cmdline":"frigate.process:gp_entrance_archer","cpu":"4.8","cpu_average":"44","mem":"0.3"},"1350":{"cmdline":"frigate.process:gp_bldg_08_front","cpu":"16.5","cpu_average":"34","mem":"0.3"},"1351":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1352":{"cmdline":"frigate.process:gp_bldg_08_rear","cpu":"14.1","cpu_average":"35","mem":"0.3"},"1353":{"cmdline":"frigate.process:gp_bldg_03_side","cpu":"31.3","cpu_average":"9","mem":"0.3"},"1354":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1355":{"cmdline":"frigate.capture:warehouse_exterior_nw","cpu":"5.2","cpu_average":"9","mem":"0.3"},"1358":{"cmdline":"frigate.capture:warehouse_exterior_sw","cpu":"4.5","cpu_average":"9","mem":"0.3"},"1359":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1360":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_nw -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/warehouse_exterior_nw-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"8.6","cpu_average":"16","mem":"0.3"},"1363":{"cmdline":"frigate.capture:warehouse_exterior_se","cpu":"1.0","cpu_average":"1","mem":"0.3"},"1364":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_sw_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"8.3","cpu_average":"14","mem":"0.2"},"1368":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1369":{"cmdline":"frigate.capture:warehouse_exterior_front","cpu":"2.4","cpu_average":"4","mem":"0.3"},"1370":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_se_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.1","cpu_average":"5","mem":"0.2"},"1375":{"cmdline":"frigate.capture:warehouse_workbench","cpu":"0.7","cpu_average":"1","mem":"0.3"},"1376":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_sw -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/warehouse_exterior_sw-%Y%m%d%H%M%S.mp4","cpu":"0.7","cpu_average":"2","mem":"0.0"},"1380":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1383":{"cmdline":"frigate.capture:warehouse_interior_front","cpu":"1.0","cpu_average":"1","mem":"0.3"},"1384":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_se -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_exterior_se-%Y%m%d%H%M%S.mp4","cpu":"0.7","cpu_average":"1","mem":"0.0"},"1390":{"cmdline":"frigate.capture:warehouse_interior_office","cpu":"1.0","cpu_average":"2","mem":"0.3"},"1391":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 3840x2160 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1398":{"cmdline":"frigate.capture:warehouse_interior_overview","cpu":"4.1","cpu_average":"8","mem":"0.3"},"1402":{"cmdline":"frigate.capture:gp_clubhouse_gym_1","cpu":"9.3","cpu_average":"21","mem":"0.3"},"1403":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1407":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_gym_1 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_gym_1-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=2592:h=1944:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"14.1","cpu_average":"26","mem":"0.4"},"1408":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_office_sub -r 5 -vf fps=5,scale_cuda=w=704:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.1","cpu_average":"5","mem":"0.2"},"1409":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_overview -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/warehouse_interior_overview-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"7.2","cpu_average":"15","mem":"0.3"},"1411":{"cmdline":"frigate.capture:gp_clubhouse_gym_2","cpu":"9.6","cpu_average":"23","mem":"0.3"},"1413":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1420":{"cmdline":"frigate.capture:gp_clubhouse_front_door","cpu":"0.3","cpu_average":"1","mem":"0.3"},"1426":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_gym_2 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_gym_2-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=2592:h=1944:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"14.8","cpu_average":"29","mem":"0.3"},"1427":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_office -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_interior_office-%Y%m%d%H%M%S.mp4","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1430":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2560x1920 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1431":{"cmdline":"frigate.capture:gp_clubhouse_back_door","cpu":"0.7","cpu_average":"1","mem":"0.3"},"1433":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_front_door_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"2.8","cpu_average":"5","mem":"0.2"},"1438":{"cmdline":"frigate.capture:gp_clubhouse_banquet","cpu":"0.7","cpu_average":"1","mem":"0.3"},"1439":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_back_door_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.1","cpu_average":"5","mem":"0.2"},"1440":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1445":{"cmdline":"frigate.capture:gp_clubhouse_pool_porch","cpu":"4.5","cpu_average":"9","mem":"0.3"},"1446":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_front_door -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_front_door-%Y%m%d%H%M%S.mp4","cpu":"1.0","cpu_average":"1","mem":"0.0"},"1450":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_banquet_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.8","cpu_average":"6","mem":"0.2"},"1452":{"cmdline":"frigate.capture:gp_clubhouse_pool","cpu":"2.8","cpu_average":"4","mem":"0.3"},"1453":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_back_door -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_back_door-%Y%m%d%H%M%S.mp4","cpu":"1.0","cpu_average":"1","mem":"0.0"},"1454":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1455":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_pool_porch -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_pool_porch-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"6.9","cpu_average":"16","mem":"0.3"},"1460":{"cmdline":"frigate.capture:gp_clubhouse_firepit","cpu":"12.7","cpu_average":"32","mem":"0.3"},"1463":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_pool_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"7.2","cpu_average":"11","mem":"0.2"},"1464":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_banquet -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_banquet-%Y%m%d%H%M%S.mp4","cpu":"1.0","cpu_average":"2","mem":"0.0"},"1466":{"cmdline":"frigate.capture:gp_clubhouse_parking","cpu":"2.1","cpu_average":"4","mem":"0.3"},"1467":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1470":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_firepit -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_clubhouse_firepit-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"28.5","cpu_average":"67","mem":"0.4"},"1473":{"cmdline":"frigate.capture:gp636_doorbell","cpu":"3.8","cpu_average":"9","mem":"0.3"},"1474":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:campass123@10.10.23.30:554/Streaming/channels/102 -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"5.2","cpu_average":"12","mem":"0.2"},"1478":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_pool -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_clubhouse_pool-%Y%m%d%H%M%S.mp4","cpu":"5.8","cpu_average":"10","mem":"0.0"},"1479":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1480":{"cmdline":"frigate.capture:gp_entrance_75","cpu":"4.1","cpu_average":"9","mem":"0.3"},"1482":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp636_doorbell -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp636_doorbell-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"14.4","cpu_average":"30","mem":"0.3"},"1488":{"cmdline":"frigate.capture:gp_entrance_archer","cpu":"4.8","cpu_average":"10","mem":"0.3"},"1490":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_parking -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_clubhouse_parking-%Y%m%d%H%M%S.mp4","cpu":"5.8","cpu_average":"12","mem":"0.0"},"1491":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_entrance_75 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_entrance_75-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"16.5","cpu_average":"32","mem":"0.3"},"1492":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1496":{"cmdline":"frigate.capture:gp_bldg_08_front","cpu":"3.1","cpu_average":"4","mem":"0.3"},"1497":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_entrance_archer -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_entrance_archer-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"17.9","cpu_average":"34","mem":"0.3"},"1501":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1200 -i pipe: -f mpegts -s 1920x1200 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"25.5","cpu_average":"46","mem":"0.1"},"1502":{"cmdline":"frigate.capture:gp_bldg_08_rear","cpu":"2.4","cpu_average":"4","mem":"0.3"},"1503":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_front_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"6.5","cpu_average":"11","mem":"0.2"},"1515":{"cmdline":"frigate.capture:gp_bldg_03_side","cpu":"2.8","cpu_average":"4","mem":"0.3"},"1538":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_rear_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"5.8","cpu_average":"11","mem":"0.2"},"1541":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_front -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_bldg_08_front-%Y%m%d%H%M%S.mp4","cpu":"5.2","cpu_average":"12","mem":"0.0"},"1546":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_03_side_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"6.2","cpu_average":"11","mem":"0.2"},"1549":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_rear -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_bldg_08_rear-%Y%m%d%H%M%S.mp4","cpu":"6.9","cpu_average":"12","mem":"0.0"},"1552":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_03_side -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_bldg_03_side-%Y%m%d%H%M%S.mp4","cpu":"2.8","cpu_average":"10","mem":"0.0"},"510368":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_front_sub -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_exterior_front-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1536:h=576:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"7.9","cpu_average":"14","mem":"0.2"},"510533":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_workbench -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_workbench-%Y%m%d%H%M%S.mp4","cpu":"6.5","cpu_average":"13","mem":"0.0"},"510536":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_front -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_interior_front-%Y%m%d%H%M%S.mp4","cpu":"6.5","cpu_average":"13","mem":"0.0"},"510885":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_workbench_sub -r 5 -vf fps=5,scale_cuda=w=640:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"2.8","cpu_average":"5","mem":"0.2"},"510887":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-8e584cf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_front_sub -r 5 -vf fps=5,scale_cuda=w=640:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"2.8","cpu_average":"5","mem":"0.2"},"1350392":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":107.6,"detectors":{"coral":{"detection_start":1689510989.004715,"inference_speed":9.48,"pid":1311}},"gp525_doorbell":{"camera_fps":0.0,"capture_pid":null,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":0,"pid":null,"process_fps":0.0,"skipped_fps":0.0},"gp636_doorbell":{"camera_fps":5.1,"capture_pid":1473,"detection_enabled":1,"detection_fps":4.4,"ffmpeg_pid":1482,"pid":1346,"process_fps":5.0,"skipped_fps":0.0},"gp_bldg_03_side":{"camera_fps":5.0,"capture_pid":1515,"detection_enabled":1,"detection_fps":7.7,"ffmpeg_pid":1546,"pid":1353,"process_fps":5.1,"skipped_fps":0.0},"gp_bldg_08_front":{"camera_fps":5.0,"capture_pid":1496,"detection_enabled":1,"detection_fps":4.9,"ffmpeg_pid":1503,"pid":1350,"process_fps":5.0,"skipped_fps":0.0},"gp_bldg_08_rear":{"camera_fps":5.1,"capture_pid":1502,"detection_enabled":1,"detection_fps":5.1,"ffmpeg_pid":1538,"pid":1352,"process_fps":5.1,"skipped_fps":0.0},"gp_clubhouse_back_door":{"camera_fps":5.0,"capture_pid":1431,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1439,"pid":1336,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_banquet":{"camera_fps":5.0,"capture_pid":1438,"detection_enabled":1,"detection_fps":0.7,"ffmpeg_pid":1450,"pid":1338,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_firepit":{"camera_fps":5.0,"capture_pid":1460,"detection_enabled":1,"detection_fps":6.0,"ffmpeg_pid":1470,"pid":1342,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_front_door":{"camera_fps":5.1,"capture_pid":1420,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1433,"pid":1335,"process_fps":5.1,"skipped_fps":0.0},"gp_clubhouse_gym_1":{"camera_fps":5.0,"capture_pid":1402,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1407,"pid":1332,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_gym_2":{"camera_fps":5.1,"capture_pid":1411,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1426,"pid":1333,"process_fps":5.1,"skipped_fps":0.0},"gp_clubhouse_parking":{"camera_fps":5.1,"capture_pid":1466,"detection_enabled":1,"detection_fps":5.0,"ffmpeg_pid":1474,"pid":1344,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_pool":{"camera_fps":5.0,"capture_pid":1452,"detection_enabled":1,"detection_fps":5.9,"ffmpeg_pid":1463,"pid":1341,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_pool_porch":{"camera_fps":5.0,"capture_pid":1445,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1455,"pid":1339,"process_fps":5.0,"skipped_fps":0.0},"gp_entrance_75":{"camera_fps":5.1,"capture_pid":1480,"detection_enabled":1,"detection_fps":3.1,"ffmpeg_pid":1491,"pid":1347,"process_fps":5.1,"skipped_fps":0.0},"gp_entrance_archer":{"camera_fps":5.0,"capture_pid":1488,"detection_enabled":1,"detection_fps":2.0,"ffmpeg_pid":1497,"pid":1349,"process_fps":5.0,"skipped_fps":0.0},"gpu_usages":{"NVIDIA T1000 8GB":{"gpu":"9.0%","mem":"40.5%"}},"processes":{"go2rtc":{"pid":89},"logger":{"pid":1300},"recording":{"pid":1301}},"service":{"last_updated":1689510989,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":1344.9,"mount_type":"tmpfs","total":1521.0,"used":176.1},"/media/frigate/clips":{"free":613031.8,"mount_type":"zfs","total":3683156.4,"used":3070124.6},"/media/frigate/recordings":{"free":613031.8,"mount_type":"zfs","total":3683156.4,"used":3070124.6},"/tmp/cache":{"free":615350.5,"mount_type":"overlay","total":670697.8,"used":55347.2}},"temperatures":{},"uptime":38463,"version":"0.13.0-8e584cf"},"warehouse_exterior_doorbell":{"camera_fps":0.0,"capture_pid":null,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":0,"pid":null,"process_fps":0.0,"skipped_fps":0.0},"warehouse_exterior_front":{"camera_fps":5.0,"capture_pid":1369,"detection_enabled":1,"detection_fps":9.3,"ffmpeg_pid":510368,"pid":1324,"process_fps":5.0,"skipped_fps":0.0},"warehouse_exterior_nw":{"camera_fps":5.0,"capture_pid":1355,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1360,"pid":1320,"process_fps":5.0,"skipped_fps":0.0},"warehouse_exterior_se":{"camera_fps":5.1,"capture_pid":1363,"detection_enabled":1,"detection_fps":12.1,"ffmpeg_pid":1370,"pid":1323,"process_fps":4.6,"skipped_fps":0.3},"warehouse_exterior_sw":{"camera_fps":5.1,"capture_pid":1358,"detection_enabled":1,"detection_fps":8.1,"ffmpeg_pid":1364,"pid":1321,"process_fps":5.0,"skipped_fps":0.0},"warehouse_interior_front":{"camera_fps":5.1,"capture_pid":1383,"detection_enabled":1,"detection_fps":12.2,"ffmpeg_pid":510887,"pid":1327,"process_fps":4.6,"skipped_fps":0.3},"warehouse_interior_office":{"camera_fps":5.1,"capture_pid":1390,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1408,"pid":1329,"process_fps":5.1,"skipped_fps":0.0},"warehouse_interior_overview":{"camera_fps":5.0,"capture_pid":1398,"detection_enabled":1,"detection_fps":5.3,"ffmpeg_pid":1409,"pid":1330,"process_fps":5.0,"skipped_fps":0.0},"warehouse_workbench":{"camera_fps":5.1,"capture_pid":1375,"detection_enabled":1,"detection_fps":15.8,"ffmpeg_pid":510885,"pid":1326,"process_fps":2.9,"skipped_fps":2.1}}

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

NA

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

What type of mount is /db/?

YBonline commented 1 year ago

Filesystem is ext4, SSD, separate drive from /media/frigate

xconverge commented 1 year ago

I started seeing this too, let me know what I can do to help debug.

0.13.0-dacf45c
2023-07-18 09:50:05.972838129  Exception in thread timeline_processor:
2023-07-18 09:50:05.972852518  Traceback (most recent call last):
2023-07-18 09:50:05.972855499    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 09:50:05.975147184      cursor.execute(sql, params or ())
2023-07-18 09:50:05.975158221  sqlite3.OperationalError: database is locked
2023-07-18 09:50:05.975159517  
2023-07-18 09:50:05.975162406  During handling of the above exception, another exception occurred:
2023-07-18 09:50:05.975163536  
2023-07-18 09:50:05.975165202  Traceback (most recent call last):
2023-07-18 09:50:05.975169462    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-18 09:50:05.985262396      self.run()
2023-07-18 09:50:05.985274137    File "/opt/frigate/frigate/timeline.py", line 46, in run
2023-07-18 09:50:05.985275859      self.handle_object_detection(
2023-07-18 09:50:05.985278155    File "/opt/frigate/frigate/timeline.py", line 92, in handle_object_detection
2023-07-18 09:50:05.985279914      Timeline.insert(timeline_entry).execute()
2023-07-18 09:50:05.985282137    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-18 09:50:05.985283970      return method(self, database, *args, **kwargs)
2023-07-18 09:50:05.985292933    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-18 09:50:05.985379413      return self._execute(database)
2023-07-18 09:50:05.985382413    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-18 09:50:05.985385062      return super(Insert, self)._execute(database)
2023-07-18 09:50:05.985387339    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-18 09:50:05.985500801      return self.handle_result(database, cursor)
2023-07-18 09:50:05.985504061    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-18 09:50:05.985572801      return database.last_insert_id(cursor, self._query_type)
2023-07-18 09:50:05.985576153    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-18 09:50:05.985577967      return cursor.lastrowid
2023-07-18 09:50:05.985580449    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-18 09:50:05.985581819      self._wait()
2023-07-18 09:50:05.985584264    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-18 09:50:05.985585690      raise self._exc
2023-07-18 09:50:05.985588097    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-18 09:50:05.985645059      cursor = self.database._execute(obj.sql, obj.params)
2023-07-18 09:50:05.985648226    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 09:50:05.986163018      cursor.execute(sql, params or ())
2023-07-18 09:50:05.986285387    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-18 09:50:05.987740153      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-18 09:50:05.987876578    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-18 09:50:05.988446832      raise value.with_traceback(tb)
2023-07-18 09:50:05.988571202    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 09:50:05.990128541      cursor.execute(sql, params or ())
2023-07-18 09:50:05.990474519  peewee.OperationalError: database is locked
2023-07-18 12:55:50.776418134  Exception in thread event_cleanup:
2023-07-18 12:55:50.776429856  Traceback (most recent call last):
2023-07-18 12:55:50.776481448    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 12:55:50.777896732      cursor.execute(sql, params or ())
2023-07-18 12:55:50.777944121  sqlite3.OperationalError: database is locked
2023-07-18 12:55:50.777946121  
2023-07-18 12:55:50.777948491  During handling of the above exception, another exception occurred:
2023-07-18 12:55:50.777949750  
2023-07-18 12:55:50.778240618  Traceback (most recent call last):
2023-07-18 12:55:50.778247618    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-18 12:55:50.778670929      self.run()
2023-07-18 12:55:50.778706855    File "/opt/frigate/frigate/events/cleanup.py", line 164, in run
2023-07-18 12:55:50.779064556      self.expire("clips")
2023-07-18 12:55:50.779100407    File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-18 12:55:50.779403479      update_query.execute()
2023-07-18 12:55:50.779438015    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-18 12:55:50.780254842      return method(self, database, *args, **kwargs)
2023-07-18 12:55:50.780291212    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-18 12:55:50.781140019      return self._execute(database)
2023-07-18 12:55:50.781146408    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-18 12:55:50.782277361      return self.handle_result(database, cursor)
2023-07-18 12:55:50.782289621    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-18 12:55:50.783319408      return database.rows_affected(cursor)
2023-07-18 12:55:50.783329704    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-18 12:55:50.784712729      return cursor.rowcount
2023-07-18 12:55:50.784725248    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-18 12:55:50.785003412      self._wait()
2023-07-18 12:55:50.785041560    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-18 12:55:50.785281336      raise self._exc
2023-07-18 12:55:50.785288761    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-18 12:55:50.785544111      cursor = self.database._execute(obj.sql, obj.params)
2023-07-18 12:55:50.785551944    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 12:55:50.786711971      cursor.execute(sql, params or ())
2023-07-18 12:55:50.786721953    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-18 12:55:50.787886128      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-18 12:55:50.787897350    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-18 12:55:50.788201199      raise value.with_traceback(tb)
2023-07-18 12:55:50.788209088    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-18 12:55:50.789472114      cursor.execute(sql, params or ())
2023-07-18 12:55:50.789526855  peewee.OperationalError: database is locked
YBonline commented 1 year ago

This didn't fix it for me:

2023-07-19 08:36:32.553079912  [2023-07-19 08:36:32] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/warehouse_workbench-20230719083338.mp4
2023-07-19 08:36:32.553236125  [2023-07-19 08:36:32] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/warehouse_workbench-20230719083338.mp4
2023-07-19 09:37:30.323723133  Exception in thread event_processor:
2023-07-19 09:37:30.323897768  Traceback (most recent call last):
2023-07-19 09:37:30.323918369    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 09:37:30.331318561      cursor.execute(sql, params or ())
2023-07-19 09:37:30.331342251  sqlite3.OperationalError: database is locked
2023-07-19 09:37:30.331344599  
2023-07-19 09:37:30.331347339  During handling of the above exception, another exception occurred:
2023-07-19 09:37:30.331349284  
2023-07-19 09:37:30.331351318  Traceback (most recent call last):
2023-07-19 09:37:30.331359311    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 09:37:30.331632840      self.run()
2023-07-19 09:37:30.331635196    File "/opt/frigate/frigate/events/maintainer.py", line 93, in run
2023-07-19 09:37:30.331723972      self.handle_object_detection(event_type, camera, event_data)
2023-07-19 09:37:30.331726662    File "/opt/frigate/frigate/events/maintainer.py", line 205, in handle_object_detection
2023-07-19 09:37:30.331789908      Event.insert(event)
2023-07-19 09:37:30.331792344    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 09:37:30.332129113      return method(self, database, *args, **kwargs)
2023-07-19 09:37:30.332139754    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 09:37:30.332569313      return self._execute(database)
2023-07-19 09:37:30.332579136    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 09:37:30.333172138      return super(Insert, self)._execute(database)
2023-07-19 09:37:30.333179884    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 09:37:30.333545795      return self.handle_result(database, cursor)
2023-07-19 09:37:30.333552348    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 09:37:30.333964995      return database.last_insert_id(cursor, self._query_type)
2023-07-19 09:37:30.333971215    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 09:37:30.334540225      return cursor.lastrowid
2023-07-19 09:37:30.334547590    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 09:37:30.334622120      self._wait()
2023-07-19 09:37:30.334628949    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 09:37:30.334667776      raise self._exc
2023-07-19 09:37:30.334675984    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 09:37:30.334724209      cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 09:37:30.334732501    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 09:37:30.335214326      cursor.execute(sql, params or ())
2023-07-19 09:37:30.335221475    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 09:37:30.335653469      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 09:37:30.335659650    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 09:37:30.335711924      raise value.with_traceback(tb)
2023-07-19 09:37:30.335717649    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 09:37:30.336193640      cursor.execute(sql, params or ())
2023-07-19 09:37:30.336199661  peewee.OperationalError: database is locked
2023-07-19 10:02:14.205352389  Exception in thread event_cleanup:
2023-07-19 10:02:14.205530698  Traceback (most recent call last):
2023-07-19 10:02:14.205540822    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 10:02:14.214614187      cursor.execute(sql, params or ())
2023-07-19 10:02:14.214620420  sqlite3.OperationalError: database is locked
2023-07-19 10:02:14.214622804  
2023-07-19 10:02:14.214625694  During handling of the above exception, another exception occurred:
2023-07-19 10:02:14.214627684  
2023-07-19 10:02:14.214629899  Traceback (most recent call last):
2023-07-19 10:02:14.214632570    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 10:02:14.215264752      self.run()
2023-07-19 10:02:14.215289119    File "/opt/frigate/frigate/events/cleanup.py", line 164, in run
2023-07-19 10:02:14.216935461      self.expire("clips")
2023-07-19 10:02:14.216959001    File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-19 10:02:14.218279606      update_query.execute()
2023-07-19 10:02:14.219404135    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 10:02:14.223553696      return method(self, database, *args, **kwargs)
2023-07-19 10:02:14.223558200    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 10:02:14.225136113      return self._execute(database)
2023-07-19 10:02:14.225146470    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 10:02:14.226925532      return self.handle_result(database, cursor)
2023-07-19 10:02:14.226937722    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-19 10:02:14.228378208      return database.rows_affected(cursor)
2023-07-19 10:02:14.228579252    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-19 10:02:14.232971992      return cursor.rowcount
2023-07-19 10:02:14.232989583    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-19 10:02:14.240234468      self._wait()
2023-07-19 10:02:14.240249777    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 10:02:14.242112263      raise self._exc
2023-07-19 10:02:14.249262345    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 10:02:14.251695665      cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 10:02:14.251710549    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 10:02:14.265251398      cursor.execute(sql, params or ())
2023-07-19 10:02:14.265266152    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 10:02:14.271666464      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 10:02:14.273429818    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 10:02:14.285790808      raise value.with_traceback(tb)
2023-07-19 10:02:14.285811216    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 10:02:14.303426253      cursor.execute(sql, params or ())
2023-07-19 10:02:14.303631589  peewee.OperationalError: database is locked
NickM-27 commented 1 year ago

This doesn't seem right, if I'm correct you have 25 cameras which means the db has a timeout of 250 seconds. If the db is unable to write that change in 250 seconds that seems like an issue with the host / storage

YBonline commented 1 year ago

Your camera count is indeed correct. So yesterday I was playing with a few things to try to troubleshoot this issue, and tried moving the DB file from my main host drive to the RAID array where I store my backup files, which is ZFS, still local. Using dd to do a speed test I was getting 6x-10x faster read/write speeds, but no effect at all on this issue.

During further troubleshooting, I've discovered that the auto-expiration process where the recordings storage drive fills up takes a TON of resources, in fact, I achieved a 60% reduction in CPU usage by adjusting my auto expiration settings to make sure that enough space is freed for the entire days recordings a day in advance. I was going to make a separate issue for that once I test one more time to make sure that was in fact true. But I still get my DB lock errors, even with drastically less load and many less hits on the DB.

I guess the next thing I could do is delete the database file and start over to make sure there isn't something in there that is causing it to move really slow if you want me to try that

NickM-27 commented 1 year ago

I guess the next thing I could do is delete the database file and start over to make sure there isn't something in there that is causing it to move really slow if you want me to try that

Could be worth a try, can you also show your docker compose for frigate?

YBonline commented 1 year ago

Ok, my suspicion is an empty DB will not give me the issue, and I believe it will take like 2 weeks for it to fill back up, which is why I didn't jump to that as of yet, but here's my docker-compose:

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:dev-f2ff55d
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    shm_size: "1521mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri/card0 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/frigate/config.yml:/config/config.yml
      - /media/frigate/storage:/media/frigate
      - /var/lib/frigate:/db
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 2500000000
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: removed
      PLUS_API_KEY: removed
NickM-27 commented 1 year ago

How large is the db currently?

YBonline commented 1 year ago

3.2G Jul 19 11:57 frigate.db 64K Jul 19 12:10 frigate.db-shm 21M Jul 19 12:10 frigate.db-wal

NickM-27 commented 1 year ago

Oh wow, that is massive. Okay so I see a couple things:

  1. I'd 100% recommend mapping the entire config directory not just the config file for docker. This will allow frigate to correctly run its vacuuming
  2. Can you try running a manual VACUUM and show the new sizes after running that?
YBonline commented 1 year ago

So to run the manual vacuum I had to shutdown Frigate as DB was always locked, but after that: 3.2G Jul 19 12:20 frigate.db 6.3M Jul 19 12:21 frigate.db-shm 3.2G Jul 19 12:20 frigate.db-wal

Everything got bigger? I changed my config directory to expose the entire directory, will report back if that makes any difference

NickM-27 commented 1 year ago

Once the write ahead log is merged to the db it should get smaller. It makes sense that it got bigger because the write ahead log is keeping the changes separately until they are merged.

YBonline commented 1 year ago

So for reference, the SHM and WAL files went back down to size (160K and 79M)... and since the change enabling full config directory access, had it happen a few more times, here's the full log from end of startup till now (1.5 hours). Have not yet tried deleting the DB and running it yet, wanted to see if you had other ideas before I tried that. Also my order of the dual edge Google Coral is finally getting delivered today from my order last year (currently running USB), and wanted to try to make sure I'm not changing too much at once, not that I really expect much difference from that.

2023-07-19 14:05:04.935783849  [2023-07-19 14:05:04] frigate.app                    INFO    : Capture process started for gp_bldg_03_side: 1514
2023-07-19 14:05:06.993078352  [2023-07-19 14:05:06] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-07-19 14:05:09.284169961  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624032480] moov atom not found
2023-07-19 14:05:09.370823522  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp636_doorbell-20230719140434.mp4
2023-07-19 14:05:09.370925365  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp636_doorbell-20230719140434.mp4
2023-07-19 14:05:09.478313324  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.483466714  [ERROR:0@7.197] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.483470895  
2023-07-19 14:05:09.483473733  OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.483480065  
2023-07-19 14:05:09.483481632  
2023-07-19 14:05:09.511689656  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_firepit-20230719140433.mp4
2023-07-19 14:05:09.511978141  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_firepit-20230719140433.mp4
2023-07-19 14:05:09.525444910  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.556675963  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_gym_1-20230719140441.mp4
2023-07-19 14:05:09.556830086  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_gym_1-20230719140441.mp4
2023-07-19 14:05:09.559521070  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.597711532  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_gym_2-20230719140441.mp4
2023-07-19 14:05:09.597829647  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_gym_2-20230719140441.mp4
2023-07-19 14:05:09.627912344  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.628009285  [ERROR:0@7.342] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.628012755  
2023-07-19 14:05:09.628017077  OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.628019815  
2023-07-19 14:05:09.628022305  
2023-07-19 14:05:09.665363215  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp_clubhouse_pool_porch-20230719140442.mp4
2023-07-19 14:05:09.665956867  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_clubhouse_pool_porch-20230719140442.mp4
2023-07-19 14:05:09.667042493  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.699408018  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp_entrance_75-20230719140433.mp4
2023-07-19 14:05:09.699874644  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_entrance_75-20230719140433.mp4
2023-07-19 14:05:09.699979262  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.700097107  [ERROR:0@7.414] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.700100732  
2023-07-19 14:05:09.700105043  OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.700107732  
2023-07-19 14:05:09.700110122  
2023-07-19 14:05:09.737633344  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/gp_entrance_archer-20230719140441.mp4
2023-07-19 14:05:09.738021571  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/gp_entrance_archer-20230719140441.mp4
2023-07-19 14:05:09.743464666  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:09.744108921  [ERROR:0@7.458] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:09.744121871  
2023-07-19 14:05:09.744126825  OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:09.744130023  
2023-07-19 14:05:09.744132524  
2023-07-19 14:05:09.777590583  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/warehouse_exterior_nw-20230719140434.mp4
2023-07-19 14:05:09.778164379  [2023-07-19 14:05:09] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/warehouse_exterior_nw-20230719140434.mp4
2023-07-19 14:05:10.062995434  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f7624440b00] moov atom not found
2023-07-19 14:05:10.063151669  [ERROR:0@7.777] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
2023-07-19 14:05:10.063154037  
2023-07-19 14:05:10.063156859  OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'icvExtractPattern'
2023-07-19 14:05:10.063158864  
2023-07-19 14:05:10.063160374  
2023-07-19 14:05:10.096662415  [2023-07-19 14:05:10] frigate.record.maintainer      WARNING : Failed to probe corrupt segment /tmp/cache/warehouse_interior_overview-20230719140443.mp4
2023-07-19 14:05:10.097464861  [2023-07-19 14:05:10] frigate.record.maintainer      WARNING : Discarding a corrupt recording segment: /tmp/cache/warehouse_interior_overview-20230719140443.mp4
2023-07-19 14:28:03.633009785  [2023-07-19 14:28:03] frigate.object_processing      WARNING : Unable to create jpg because frame 1689791280.763218 is not in the cache
2023-07-19 14:28:41.539640682  [2023-07-19 14:28:41] frigate.object_processing      WARNING : Unable to create jpg because frame 1689791280.763218 is not in the cache
2023-07-19 14:28:41.539740623  [2023-07-19 14:28:41] frigate.object_processing      WARNING : Unable to save snapshot for 1689791259.540438-63sls2.
2023-07-19 14:28:41.539803005  [2023-07-19 14:28:41] frigate.object_processing      WARNING : Unable to create clean png because frame 1689791280.763218 is not in the cache
2023-07-19 14:28:41.539854370  [2023-07-19 14:28:41] frigate.object_processing      WARNING : Unable to save clean snapshot for 1689791259.540438-63sls2.
2023-07-19 14:28:41.540489346  [2023-07-19 14:28:41] frigate.object_processing      WARNING : Unable to create jpg because frame 1689791280.763218 is not in the cache
2023-07-19 14:29:51.403473181  [2023-07-19 14:29:51] frigate.object_processing      WARNING : Unable to create jpg because frame 1689791390.547153 is not in the cache
2023-07-19 14:30:03.539377830  [2023-07-19 14:30:03] frigate.object_processing      WARNING : Unable to create jpg because frame 1689791390.547153 is not in the cache
2023-07-19 14:30:03.539512287  [2023-07-19 14:30:03] frigate.object_processing      WARNING : Unable to save snapshot for 1689791317.765963-afqor4.
2023-07-19 14:30:03.539595885  [2023-07-19 14:30:03] frigate.object_processing      WARNING : Unable to create clean png because frame 1689791390.547153 is not in the cache
2023-07-19 14:30:03.539671668  [2023-07-19 14:30:03] frigate.object_processing      WARNING : Unable to save clean snapshot for 1689791317.765963-afqor4.
2023-07-19 14:30:03.541160656  [2023-07-19 14:30:03] frigate.object_processing      WARNING : Unable to create jpg because frame 1689791390.547153 is not in the cache
2023-07-19 14:49:41.306030604  Exception in thread timeline_processor:
2023-07-19 14:49:41.306416853  Traceback (most recent call last):
2023-07-19 14:49:41.306426516    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:49:41.308120146      cursor.execute(sql, params or ())
2023-07-19 14:49:41.308151042  sqlite3.OperationalError: database is locked
2023-07-19 14:49:41.308157901  
2023-07-19 14:49:41.308159676  During handling of the above exception, another exception occurred:
2023-07-19 14:49:41.308160993  
2023-07-19 14:49:41.308162528  Traceback (most recent call last):
2023-07-19 14:49:41.308170216    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 14:49:41.309166539      self.run()
2023-07-19 14:49:41.309185550    File "/opt/frigate/frigate/timeline.py", line 46, in run
2023-07-19 14:49:41.309591198      self.handle_object_detection(
2023-07-19 14:49:41.309695258    File "/opt/frigate/frigate/timeline.py", line 92, in handle_object_detection
2023-07-19 14:49:41.309785574      Timeline.insert(timeline_entry).execute()
2023-07-19 14:49:41.309792028    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 14:49:41.310130337      return method(self, database, *args, **kwargs)
2023-07-19 14:49:41.310138295    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 14:49:41.310457614      return self._execute(database)
2023-07-19 14:49:41.310467678    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 14:49:41.310898007      return super(Insert, self)._execute(database)
2023-07-19 14:49:41.310908975    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 14:49:41.311272285      return self.handle_result(database, cursor)
2023-07-19 14:49:41.311440908    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 14:49:41.316831388      return database.last_insert_id(cursor, self._query_type)
2023-07-19 14:49:41.316835305    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 14:49:41.317430543      return cursor.lastrowid
2023-07-19 14:49:41.317441077    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 14:49:41.317857224      self._wait()
2023-07-19 14:49:41.317889316    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 14:49:41.318009519      raise self._exc
2023-07-19 14:49:41.318035054    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 14:49:41.318157035      cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 14:49:41.318186073    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:49:41.318997723      cursor.execute(sql, params or ())
2023-07-19 14:49:41.319008706    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 14:49:41.319752205      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 14:49:41.319765290    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 14:49:41.319841621      raise value.with_traceback(tb)
2023-07-19 14:49:41.319849799    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:49:41.320656212      cursor.execute(sql, params or ())
2023-07-19 14:49:41.320669587  peewee.OperationalError: database is locked
2023-07-19 14:53:14.763705979  [2023-07-19 14:53:14] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:14.764016476  [2023-07-19 14:53:14] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:15.095173898  [2023-07-19 14:53:15] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:15.095381689  [2023-07-19 14:53:15] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:15.095584574  [2023-07-19 14:53:15] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Connection timed out
2023-07-19 14:53:15.095745892  [2023-07-19 14:53:15] ffmpeg.warehouse_interior_front.detect ERROR   :     Last message repeated 1 times
2023-07-19 14:53:20.241772808  [2023-07-19 14:53:20] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:20.242285765  [2023-07-19 14:53:20] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:25.125974072  [2023-07-19 14:53:25] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:25.126114786  [2023-07-19 14:53:25] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:25.126223301  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Invalid data found when processing input
2023-07-19 14:53:25.156582141  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : [segment @ 0x556e1fda4580] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-07-19 14:53:25.172398503  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : [aac @ 0x556e1fda3740] channel element 0.0 is not allocated
2023-07-19 14:53:25.172559015  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : Error while decoding stream #0:1: Invalid data found when processing input
2023-07-19 14:53:25.172644575  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : [aac @ 0x556e1fda3740] channel element 0.0 is not allocated
2023-07-19 14:53:25.172735064  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : Error while decoding stream #0:1: Invalid data found when processing input
2023-07-19 14:53:25.172807134  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : [aac @ 0x556e1fda3740] Input buffer exhausted before END element found
2023-07-19 14:53:25.172877940  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : Error while decoding stream #0:1: Invalid data found when processing input
2023-07-19 14:53:25.172950672  [2023-07-19 14:53:25] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Connection timed out
2023-07-19 14:53:25.173033584  [2023-07-19 14:53:25] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:53:25.173110939  [2023-07-19 14:53:25] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:27.744338974  [2023-07-19 14:53:27] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:27.744830786  [2023-07-19 14:53:27] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:35.191596475  [2023-07-19 14:53:35] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:35.191735098  [2023-07-19 14:53:35] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:35.191823828  [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x5564b1ac8f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:35.191906075  [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:53:35.226665625  [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x55dfcb431000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:35.226853984  [2023-07-19 14:53:35] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:53:35.227014950  [2023-07-19 14:53:35] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:53:35.243611619  [2023-07-19 14:53:35] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:35.245432689  [2023-07-19 14:53:35] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:35.246636847  [2023-07-19 14:53:35] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:45.261952582  [2023-07-19 14:53:45] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:45.262069500  [2023-07-19 14:53:45] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:45.262156610  [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x559925dd0f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:45.262241758  [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:53:45.298182131  [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x55a330ac6000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:45.298318891  [2023-07-19 14:53:45] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:53:45.298413437  [2023-07-19 14:53:45] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:53:45.298490568  [2023-07-19 14:53:45] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:45.315449319  [2023-07-19 14:53:45] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:45.315818754  [2023-07-19 14:53:45] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:53:55.330039150  [2023-07-19 14:53:55] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:53:55.330118729  [2023-07-19 14:53:55] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:53:55.330186618  [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x563eee392f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:55.330246707  [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:53:55.364380958  [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x556e0d041000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:53:55.364716917  [2023-07-19 14:53:55] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:53:55.364776006  [2023-07-19 14:53:55] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:53:55.364850965  [2023-07-19 14:53:55] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:53:55.382431141  [2023-07-19 14:53:55] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:53:55.383017393  [2023-07-19 14:53:55] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:05.398514691  [2023-07-19 14:54:05] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:05.398680563  [2023-07-19 14:54:05] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:05.398837422  [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x5630e5f43f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:05.398981089  [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:05.438075497  [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x5648aaedf000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:05.438562790  [2023-07-19 14:54:05] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:05.438575256  [2023-07-19 14:54:05] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:54:05.438750594  [2023-07-19 14:54:05] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:05.458499396  [2023-07-19 14:54:05] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:05.458648523  [2023-07-19 14:54:05] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:15.472376672  [2023-07-19 14:54:15] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:15.472479926  [2023-07-19 14:54:15] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:15.472560808  [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x55b4f3f8ef00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:15.472654510  [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:15.506323201  [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x55a1067f7000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:15.506531602  [2023-07-19 14:54:15] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:15.506675749  [2023-07-19 14:54:15] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:54:15.506788169  [2023-07-19 14:54:15] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:15.519536333  [2023-07-19 14:54:15] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:15.519856166  [2023-07-19 14:54:15] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:25.536218035  [2023-07-19 14:54:25] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:25.536510239  [2023-07-19 14:54:25] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:25.536820054  [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x55d05d6b0f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:25.537005601  [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:25.571962159  [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x55a6c3884000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:25.572266912  [2023-07-19 14:54:25] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:25.572913306  [2023-07-19 14:54:25] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:54:25.573179693  [2023-07-19 14:54:25] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:25.589177460  [2023-07-19 14:54:25] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:25.589423031  [2023-07-19 14:54:25] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:35.603474275  [2023-07-19 14:54:35] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:35.603686937  [2023-07-19 14:54:35] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:35.603920018  [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.detect ERROR   : [rtsp @ 0x55af9e860f00] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:35.604109203  [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Server returned 404 Not Found
2023-07-19 14:54:35.649154771  [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.record ERROR   : [rtsp @ 0x5599dad84000] method DESCRIBE failed: 404 Not Found
2023-07-19 14:54:35.649247565  [2023-07-19 14:54:35] ffmpeg.warehouse_interior_front.record ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front: Server returned 404 Not Found
2023-07-19 14:54:35.649305490  [2023-07-19 14:54:35] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:54:35.649354888  [2023-07-19 14:54:35] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:54:40.764049917  [2023-07-19 14:54:40] frigate.video                  ERROR   : warehouse_interior_front: Unable to read frames from ffmpeg process.
2023-07-19 14:54:40.764573431  [2023-07-19 14:54:40] frigate.video                  ERROR   : warehouse_interior_front: ffmpeg process is not running. exiting capture thread...
2023-07-19 14:54:45.686344141  [2023-07-19 14:54:45] watchdog.warehouse_interior_front ERROR   : Ffmpeg process crashed unexpectedly for warehouse_interior_front.
2023-07-19 14:54:45.686514163  [2023-07-19 14:54:45] watchdog.warehouse_interior_front ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-07-19 14:54:45.686627434  [2023-07-19 14:54:45] ffmpeg.warehouse_interior_front.detect ERROR   : rtsp://127.0.0.1:8554/warehouse_interior_front_sub: Invalid data found when processing input
2023-07-19 14:54:45.717520347  [2023-07-19 14:54:45] watchdog.warehouse_interior_front ERROR   : No new recording segments were created for warehouse_interior_front in the last 120s. restarting the ffmpeg record process...
2023-07-19 14:54:45.717743917  [2023-07-19 14:54:45] watchdog.warehouse_interior_front INFO    : Terminating the existing ffmpeg process...
2023-07-19 14:54:45.717866264  [2023-07-19 14:54:45] watchdog.warehouse_interior_front INFO    : Waiting for ffmpeg to exit gracefully...
2023-07-19 14:59:24.645395062  Exception in thread event_processor:
2023-07-19 14:59:24.645402112  Traceback (most recent call last):
2023-07-19 14:59:24.645568576    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:59:24.678170706      cursor.execute(sql, params or ())
2023-07-19 14:59:24.678371111  sqlite3.OperationalError: database is locked
2023-07-19 14:59:24.678373407  
2023-07-19 14:59:24.678375380  During handling of the above exception, another exception occurred:
2023-07-19 14:59:24.678376756  
2023-07-19 14:59:24.678430985  Traceback (most recent call last):
2023-07-19 14:59:24.678439777    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 14:59:24.693580562      self.run()
2023-07-19 14:59:24.693774376    File "/opt/frigate/frigate/events/maintainer.py", line 93, in run
2023-07-19 14:59:24.699297875      self.handle_object_detection(event_type, camera, event_data)
2023-07-19 14:59:24.699324883    File "/opt/frigate/frigate/events/maintainer.py", line 205, in handle_object_detection
2023-07-19 14:59:24.717302611      Event.insert(event)
2023-07-19 14:59:24.717326310    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 14:59:24.725860641      return method(self, database, *args, **kwargs)
2023-07-19 14:59:24.725894414    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 14:59:24.736404946      return self._execute(database)
2023-07-19 14:59:24.736411984    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 14:59:24.742417231      return super(Insert, self)._execute(database)
2023-07-19 14:59:24.742454397    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 14:59:24.752735931      return self.handle_result(database, cursor)
2023-07-19 14:59:24.757469824    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 14:59:24.800752316      return database.last_insert_id(cursor, self._query_type)
2023-07-19 14:59:24.801096127    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 14:59:24.828738636      return cursor.lastrowid
2023-07-19 14:59:24.828765844    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 14:59:24.845037833      self._wait()
2023-07-19 14:59:24.845068045    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 14:59:24.848293298      raise self._exc
2023-07-19 14:59:24.848564637    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 14:59:24.877475296      cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 14:59:24.877522099    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:59:24.913647512      cursor.execute(sql, params or ())
2023-07-19 14:59:24.913672069    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 14:59:24.942764555      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 14:59:24.942788804    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 14:59:24.944917497      raise value.with_traceback(tb)
2023-07-19 14:59:24.944923921    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 14:59:24.967764664      cursor.execute(sql, params or ())
2023-07-19 14:59:24.967797397  peewee.OperationalError: database is locked
2023-07-19 15:00:37.476707097  [2023-07-19 15:00:37] frigate.object_processing      WARNING : Unable to create jpg because frame 1689793233.618259 is not in the cache
2023-07-19 15:00:37.477011756  [2023-07-19 15:00:37] frigate.object_processing      WARNING : Unable to save snapshot for 1689793232.872475-fj58un.
2023-07-19 15:00:37.477031841  [2023-07-19 15:00:37] frigate.object_processing      WARNING : Unable to create clean png because frame 1689793233.618259 is not in the cache
2023-07-19 15:00:37.477899857  [2023-07-19 15:00:37] frigate.object_processing      WARNING : Unable to save clean snapshot for 1689793232.872475-fj58un.
2023-07-19 15:00:37.478222216  [2023-07-19 15:00:37] frigate.object_processing      WARNING : Unable to create jpg because frame 1689793233.618259 is not in the cache
2023-07-19 15:04:51.087972174  Exception in thread event_cleanup:
2023-07-19 15:04:51.087992548  Traceback (most recent call last):
2023-07-19 15:04:51.087994659    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 15:04:51.137087370      cursor.execute(sql, params or ())
2023-07-19 15:04:51.137681407  sqlite3.OperationalError: database is locked
2023-07-19 15:04:51.137691704  
2023-07-19 15:04:51.137694009  During handling of the above exception, another exception occurred:
2023-07-19 15:04:51.137695615  
2023-07-19 15:04:51.137697497  Traceback (most recent call last):
2023-07-19 15:04:51.137705651    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 15:04:51.149135325      self.run()
2023-07-19 15:04:51.149160687    File "/opt/frigate/frigate/events/cleanup.py", line 164, in run
2023-07-19 15:04:51.150719048      self.expire("clips")
2023-07-19 15:04:51.150726503    File "/opt/frigate/frigate/events/cleanup.py", line 125, in expire
2023-07-19 15:04:51.150888196      update_query.execute()
2023-07-19 15:04:51.150905226    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 15:04:51.158851509      return method(self, database, *args, **kwargs)
2023-07-19 15:04:51.158935061    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 15:04:51.190647638      return self._execute(database)
2023-07-19 15:04:51.190694665    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 15:04:51.208109332      return self.handle_result(database, cursor)
2023-07-19 15:04:51.208134021    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2567, in handle_result
2023-07-19 15:04:51.209236994      return database.rows_affected(cursor)
2023-07-19 15:04:51.209321316    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3759, in rows_affected
2023-07-19 15:04:51.231111916      return cursor.rowcount
2023-07-19 15:04:51.231134510    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 93, in rowcount
2023-07-19 15:04:51.245487015      self._wait()
2023-07-19 15:04:51.245766792    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 15:04:51.253091979      raise self._exc
2023-07-19 15:04:51.253124326    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 15:04:51.253385616      cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 15:04:51.253397615    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 15:04:51.267243391      cursor.execute(sql, params or ())
2023-07-19 15:04:51.267345892    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 15:04:51.270761863      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 15:04:51.270775679    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 15:04:51.271138336      raise value.with_traceback(tb)
2023-07-19 15:04:51.271149300    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 15:04:51.309389243      cursor.execute(sql, params or ())
2023-07-19 15:04:51.310975993  peewee.OperationalError: database is locked
NickM-27 commented 1 year ago

I have a hard time understanding how the DB could be so far behind that it takes 250 seconds and is still locked, but maybe something else is going on. I have made a few improvements in #7227 that should help somewhat

xconverge commented 1 year ago

Note I experienced it with 1 camera, so with the recent commit of lowering the timeout from 60 -> 10 I didn't bother testing it yet. I will try the latest changes once they get merged and report back. It has been so rare for me that I won't know for awhile either though (it hasn't happened in 24 hours)

NickM-27 commented 1 year ago

That just seems weird, 60 seconds with the activity of one camera again seems like an issue with the host / storage that holds the DB from what I can tell. My 6 cameras (so 60 seconds timeout before and after the change) have not had this issue ever.

YBonline commented 1 year ago

I don't believe I mentioned it previously and don't know if it will help, but the last build I ran for a considerable amount of time was the dev build immediately preceding your birdseye view revamp (the last dev build with the grid). On that build, I never once had a DB locked error, and the object recognition was consistently under 1 second. That build was fully rock solid with a large amount of cameras from my experience. Unfortunately I changed a couple cameras around the same time I switched newer dev builds (after you applied the last round of birdseye view fixes) so I wasn't sure if something else I changed caused some funkiness, but thought I would mention that in case it would help.

7227 seems to improve things a bit, 45 min in and no DB lock issues yet, however I started paying a bit more attention to other delays, and I've noticed pretty extreme delays on the object recognition count getting into Home Assistant, so I think something else is in play here. Also, I haven't had this error since switching to the dev build, but it came back, at least for one set of frames (nothing like what it was prior to it being multithreaded):

2023-07-19 17:47:51.442605605  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp636_doorbell. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.442720836  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_bldg_03_side. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.442991148  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443188533  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_back_door. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443389411  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_banquet. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443588370  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_firepit. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.443797216  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_front_door. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444014054  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_1. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444222003  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_2. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444418403  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_parking. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444635019  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.444823123  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool_porch. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445018796  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_entrance_archer. Keeping the 5 most recent segments out of 7 and discarding the rest...
2023-07-19 17:47:51.445347171  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_nw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445533760  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_se. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445722186  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_sw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.445919373  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_interior_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.446102757  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_interior_overview. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:47:51.446332000  [2023-07-19 17:47:51] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_workbench. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-19 17:49:32.299686551  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp636_doorbell. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.300532919  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_bldg_03_side. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.301066652  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_front. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.301604455  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_rear. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-07-19 17:49:32.302353840  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_back_door. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.303078195  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_banquet. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.304277275  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_firepit. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.304986473  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_front_door. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.305632511  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_1. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.306422428  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_2. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.307211452  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_parking. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.307832624  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.308520954  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool_porch. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.309208895  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for gp_entrance_archer. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.309906754  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_nw. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.310921008  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_se. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.311427649  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_sw. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.312631965  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_interior_front. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.313533449  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_interior_overview. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-07-19 17:49:32.314128081  [2023-07-19 17:49:32] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for warehouse_workbench. Keeping the 5 most recent segments out of 10 and discarding the rest...
NickM-27 commented 1 year ago

I don't see any changes that would increase DB activity and cause this issue, seems like it would likely be whatever changes were made to cameras.

however I started paying a bit more attention to other delays, and I've noticed pretty extreme delays on the object recognition count getting into Home Assistant, so I think something else is in play here

I don't believe db delays will affect object count via MQTT at all so it seems unrelated

YBonline commented 1 year ago

The DB locking is definitely causing an effect on the object count that Home Assistant is receiving. Every time the DB locks, while Frigate seems to recover from it eventually and I just have some time of lost recordings, Home Assistant seems to start adding to the object count like crazy, and I end up with Home Assistant reading like 23 people in a view with 0. Every time the DB locks, I have to restart the Home Assistant plugin, it never resets to 0. Even restarting Frigate doesn't reset Home Assistant's object counters back to 0 after the DB locks.

Even after restarting Frigate, have to restart the Home Assistant plugin after the database error for my detections/automations to work correctly)

1.5 hours and no DB errors with #7227 and actually no more of the can't keep up errors since the group of them at the 30 minute point. I have gone 10 hours without the error in the past though

NickM-27 commented 1 year ago

I meant if the DB is slow to respond but not actually having the lock error I don't believe it will affect the mqtt object count.

If frigate actually has a crash from the db locked error then it will absolutely affect the mqtt object count because frigate is crashed and not working correctly.

In any case though, it is good to hear that that change has so far fixed the issue.

YBonline commented 1 year ago

Unfortunately, it did eventually come in multiple times, but it did take significantly longer to appear. I also seemed to have a memory leak that sucked all my 70GB of RAM until Frigate stopped working entirely :-/

2023-07-19 20:57:54.938386870  Exception in thread event_processor:
2023-07-19 20:57:54.938421216  Traceback (most recent call last):
2023-07-19 20:57:54.938424705    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 20:57:54.942297772      cursor.execute(sql, params or ())
2023-07-19 20:57:54.942314561  sqlite3.OperationalError: database is locked
2023-07-19 20:57:54.942317619  
2023-07-19 20:57:54.942320863  During handling of the above exception, another exception occurred:
2023-07-19 20:57:54.942322978  
2023-07-19 20:57:54.942325600  Traceback (most recent call last):
2023-07-19 20:57:54.942328281    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
2023-07-19 20:57:54.942852870      self.run()
2023-07-19 20:57:54.942869188    File "/opt/frigate/frigate/events/maintainer.py", line 93, in run
2023-07-19 20:57:54.943567915      self.handle_object_detection(event_type, camera, event_data)
2023-07-19 20:57:54.943748845    File "/opt/frigate/frigate/events/maintainer.py", line 205, in handle_object_detection
2023-07-19 20:57:54.944025244      Event.insert(event)
2023-07-19 20:57:54.944038745    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner
2023-07-19 20:57:54.944824895      return method(self, database, *args, **kwargs)
2023-07-19 20:57:54.944831462    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute
2023-07-19 20:57:54.945590176      return self._execute(database)
2023-07-19 20:57:54.945597157    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute
2023-07-19 20:57:54.946829045      return super(Insert, self)._execute(database)
2023-07-19 20:57:54.946853511    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute
2023-07-19 20:57:54.948082069      return self.handle_result(database, cursor)
2023-07-19 20:57:54.948103109    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result
2023-07-19 20:57:54.948926720      return database.last_insert_id(cursor, self._query_type)
2023-07-19 20:57:54.948931465    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id
2023-07-19 20:57:54.950009943      return cursor.lastrowid
2023-07-19 20:57:54.950020689    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid
2023-07-19 20:57:54.950124173      self._wait()
2023-07-19 20:57:54.950132095    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait
2023-07-19 20:57:54.950193910      raise self._exc
2023-07-19 20:57:54.950200070    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute
2023-07-19 20:57:54.950278410      cursor = self.database._execute(obj.sql, obj.params)
2023-07-19 20:57:54.950285888    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 20:57:54.951382985      cursor.execute(sql, params or ())
2023-07-19 20:57:54.952043540    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in __exit__
2023-07-19 20:57:54.952806685      reraise(new_type, new_type(exc_value, *exc_args), traceback)
2023-07-19 20:57:54.952813306    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise
2023-07-19 20:57:54.952916305      raise value.with_traceback(tb)
2023-07-19 20:57:54.952922694    File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql
2023-07-19 20:57:54.953647736      cursor.execute(sql, params or ())
2023-07-19 20:57:54.953670615  peewee.OperationalError: database is locked
YBonline commented 1 year ago

I also noticed /tmp/cache seems to both be way fuller then usual, along with the max size seemingly being the full size of my database (????) drive? I am not super familiar with docker, so I don't know if I am onto something or not?

/tmp/cache | 52.13 GiB | 654.96 GiB

My docker-compose has /tmp/cache configured to 2.5GB of memory I believe:

I'm not sure if its actually mounting any tmpfs, I don't see it under mount either inside of the docker container:

root@ff56cc3a3c0d:/opt/frigate# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LVMYR4Z4VTZYUOA4X6KFXVJIPC:/var/lib/docker/overlay2/l/THJDIXRBQIVGDHXOFYBVE5UBT5:/var/lib/docker/overlay2/l/H7T2WJMK4NUFWQOOGA5VDRKOR4:/var/lib/docker/overlay2/l/HAXVSU6FNUVYYNKAWQYO6WHNGY:/var/lib/docker/overlay2/l/TZFLCTHSVGMD5VUKQPDZSKJV4C:/var/lib/docker/overlay2/l/ZWXTBJE535RGOXUUZ4P6GAKUMN:/var/lib/docker/overlay2/l/ADIRBFCCWJL3RVLUQGLGNPGUM5:/var/lib/docker/overlay2/l/N2RXCEC77GFR2G7XH43LR64SXA,upperdir=/var/lib/docker/overlay2/98304f27ab12a452e7edc0c231c9f69e0996d7524707be953d1e84e9f10b964f/diff,workdir=/var/lib/docker/overlay2/98304f27ab12a452e7edc0c231c9f69e0996d7524707be953d1e84e9f10b964f/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1557504k,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on /config type ext4 (rw,relatime)
ubntvar on /db type zfs (rw,xattr,noacl)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/share/zoneinfo/Etc/UTC type ext4 (ro,relatime)
/opt/frigate/frigate on /media/frigate type zfs (rw,xattr,noacl)
ubntvar on /etc/resolv.conf type zfs (rw,xattr,noacl)
ubntvar on /etc/hostname type zfs (rw,xattr,noacl)
ubntvar on /etc/hosts type zfs (rw,xattr,noacl)
tmpfs on /proc/driver/nvidia type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=555,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-smi type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-debugdump type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-persistenced type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-cuda-mps-control type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/bin/nvidia-cuda-mps-server type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libcuda.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libcudadebugger.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/lib/x86_64-linux-gnu/libnvcuvid.so.525.125.06 type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /lib/firmware/nvidia/525.125.06/gsp_ad10x.bin type ext4 (ro,nosuid,nodev,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /lib/firmware/nvidia/525.125.06/gsp_tu10x.bin type ext4 (ro,nosuid,nodev,relatime)
tmpfs on /run/nvidia-persistenced/socket type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6537212k,mode=755,inode64)
udev on /dev/nvidiactl type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
udev on /dev/nvidia-uvm type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
udev on /dev/nvidia-uvm-tools type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
udev on /dev/nvidia0 type devtmpfs (ro,nosuid,noexec,relatime,size=32569268k,nr_inodes=8142317,mode=755,inode64)
proc on /proc/driver/nvidia/gpus/0000:3b:00.0 type proc (ro,nosuid,nodev,noexec,relatime)
root@ff56cc3a3c0d:/opt/frigate# ^C
root@ff56cc3a3c0d:/opt/frigate#

No sign of it on mount in the host either.

NickM-27 commented 1 year ago

Well, what is actually inside of /tmp/cache?

Seems like there's definitely other issues going on with the setup then

YBonline commented 1 year ago

the only thing in there is short clip files, max size 15MB, its just getting backed up. I got the DB is locked error after 1.5 hours this morning

NickM-27 commented 1 year ago

the only thing in there is short clip files, max size 15MB, its just getting backed up

but how many? if frigate's /tmp/cache is 50+ GB then there must be a lot otherwise it seems that is unrelated to frigate

YBonline commented 1 year ago

Actually it seems that the Frigate's storage UI is wrong on the size of /tmp/cache. Its still reading the same 52.13 GB but the actual size of clips is 85MB. It appears to be using the size of the /var/lib/docker merged directory for the container, which seems to be where /tmp/cache is stored. From what I can tell, it doesn't appear to be stored in memory, although my docker-compose indicates it should, correct?

No non-clip files in there, and It does get dumped on each restart. I also got multiple more DB is locked errors overnight

YBonline commented 1 year ago

root@ff56cc3a3c0d:/tmp/cache# dd if=/dev/zero of=test1.img bs=2G count=1 oflag=dsync 0+1 records in 0+1 records out 2147479552 bytes (2.1 GB, 2.0 GiB) copied, 14.0936 s, 152 MB/s

NickM-27 commented 1 year ago

I've not seen that before, it definitely should be stored in memory and it is for me. I don't see anything wrong with the docker compose definition

YBonline commented 1 year ago

I changed the size to 1GB and now it appears to be a tmpfs stored in memory... back to 2.5GB and stored on disk... so not sure how to explain that. I'm leaving it at 1GB to see what happens now... tons of motion activity at the moment and it got to 300MB and dropped a round of frames.

2023-07-20 09:40:41.657642645 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp636_doorbell. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.657752769 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_03_side. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.659271033 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_front. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.660399177 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_bldg_08_rear. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.661322977 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_back_door. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.662558977 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_banquet. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.664081407 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_firepit. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.666221622 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_front_door. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.667167385 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_1. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.669037147 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_gym_2. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.670609249 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_parking. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.671477552 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.672420197 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_clubhouse_pool_porch. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.672907205 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for gp_entrance_archer. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.674062259 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_nw. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.674429415 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_se. Keeping the 5 most recent segments out of 7 and discarding the rest... 2023-07-20 09:40:41.675965486 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_exterior_sw. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.676171432 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_front. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.677865370 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_interior_overview. Keeping the 5 most recent segments out of 6 and discarding the rest... 2023-07-20 09:40:41.678571948 [2023-07-20 09:40:41] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for warehouse_workbench. Keeping the 5 most recent segments out of 7 and discarding the rest...

Then started acting normally for the last 10 minutes... will report back on DB errors with /tmp/cache working and at a smaller size

YBonline commented 1 year ago

2023-07-20 10:09:48.028974729 Exception in thread timeline_processor: 2023-07-20 10:09:48.029013745 Traceback (most recent call last): 2023-07-20 10:09:48.029017379 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql 2023-07-20 10:09:48.030638148 cursor.execute(sql, params or ()) 2023-07-20 10:09:48.030677871 sqlite3.OperationalError: database is locked 2023-07-20 10:09:48.030684321
2023-07-20 10:09:48.030691964 During handling of the above exception, another exception occurred: 2023-07-20 10:09:48.030697874
2023-07-20 10:09:48.030704264 Traceback (most recent call last): 2023-07-20 10:09:48.030740670 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2023-07-20 10:09:48.031225014 self.run() 2023-07-20 10:09:48.031230337 File "/opt/frigate/frigate/timeline.py", line 46, in run 2023-07-20 10:09:48.031365907 self.handle_object_detection( 2023-07-20 10:09:48.031371721 File "/opt/frigate/frigate/timeline.py", line 92, in handle_object_detection 2023-07-20 10:09:48.031474146 Timeline.insert(timeline_entry).execute() 2023-07-20 10:09:48.031497626 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner 2023-07-20 10:09:48.032126401 return method(self, database, *args, *kwargs) 2023-07-20 10:09:48.032132125 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute 2023-07-20 10:09:48.034483710 return self._execute(database) 2023-07-20 10:09:48.034494078 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute 2023-07-20 10:09:48.035138616 return super(Insert, self)._execute(database) 2023-07-20 10:09:48.035145561 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute 2023-07-20 10:09:48.035717367 return self.handle_result(database, cursor) 2023-07-20 10:09:48.035723638 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result 2023-07-20 10:09:48.036345082 return database.last_insert_id(cursor, self._query_type) 2023-07-20 10:09:48.036351020 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id 2023-07-20 10:09:48.037243452 return cursor.lastrowid 2023-07-20 10:09:48.037249780 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid 2023-07-20 10:09:48.037535868 self._wait() 2023-07-20 10:09:48.037542145 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait 2023-07-20 10:09:48.037614664 raise self._exc 2023-07-20 10:09:48.037620536 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute 2023-07-20 10:09:48.037702816 cursor = self.database._execute(obj.sql, obj.params) 2023-07-20 10:09:48.037708534 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql 2023-07-20 10:09:48.038485549 cursor.execute(sql, params or ()) 2023-07-20 10:09:48.038491485 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in exit 2023-07-20 10:09:48.039161180 reraise(new_type, new_type(exc_value, exc_args), traceback) 2023-07-20 10:09:48.039166886 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise 2023-07-20 10:09:48.039252382 raise value.with_traceback(tb) 2023-07-20 10:09:48.039258314 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql 2023-07-20 10:09:48.039965084 cursor.execute(sql, params or ()) 2023-07-20 10:09:48.039989002 peewee.OperationalError: database is locked

:-(

NickM-27 commented 1 year ago

@YBonline you can try the latest commit on that PR, I have made additional improvements to the recordings management that should reduce DB activity

YBonline commented 1 year ago

No luck with the second last commit, trying now with the one with .execute()

2023-07-20 13:36:22.165349158 [2023-07-20 13:36:20] frigate.app INFO : Capture process started for gp_bldg_03_side: 1516 2023-07-20 13:51:18.799544924 [2023-07-20 13:51:18] frigate.object_processing WARNING : Unable to create jpg because frame 1689875475.620079 is not in the cache 2023-07-20 13:52:21.452289462 [2023-07-20 13:52:21] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.741912 is not in the cache 2023-07-20 13:52:21.453911583 [2023-07-20 13:52:21] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.358542 is not in the cache 2023-07-20 13:52:25.947611268 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.358542 is not in the cache 2023-07-20 13:52:25.947702198 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to save snapshot for 1689875486.544059-wv22p2. 2023-07-20 13:52:25.947759549 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to create clean png because frame 1689875540.358542 is not in the cache 2023-07-20 13:52:25.947811091 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to save clean snapshot for 1689875486.544059-wv22p2. 2023-07-20 13:52:25.950705363 [2023-07-20 13:52:25] frigate.object_processing WARNING : Unable to create jpg because frame 1689875540.358542 is not in the cache 2023-07-20 13:53:15.905574432 Exception in thread timeline_processor: 2023-07-20 13:53:15.905985036 Traceback (most recent call last): 2023-07-20 13:53:15.906006217 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql 2023-07-20 13:53:15.908779298 cursor.execute(sql, params or ()) 2023-07-20 13:53:15.908826641 sqlite3.OperationalError: database is locked 2023-07-20 13:53:15.908833678
2023-07-20 13:53:15.908842433 During handling of the above exception, another exception occurred: 2023-07-20 13:53:15.908854992
2023-07-20 13:53:15.908860004 Traceback (most recent call last): 2023-07-20 13:53:15.908881896 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2023-07-20 13:53:15.911555984 self.run() 2023-07-20 13:53:15.911591193 File "/opt/frigate/frigate/timeline.py", line 46, in run 2023-07-20 13:53:15.912064219 self.handle_object_detection( 2023-07-20 13:53:15.912086784 File "/opt/frigate/frigate/timeline.py", line 81, in handle_object_detection 2023-07-20 13:53:15.912238196 Timeline.insert(timeline_entry).execute() 2023-07-20 13:53:15.912256738 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 1966, in inner 2023-07-20 13:53:15.913653870 return method(self, database, *args, *kwargs) 2023-07-20 13:53:15.913690581 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2037, in execute 2023-07-20 13:53:15.917654278 return self._execute(database) 2023-07-20 13:53:15.917661539 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2842, in _execute 2023-07-20 13:53:15.918568999 return super(Insert, self)._execute(database) 2023-07-20 13:53:15.918575514 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2556, in _execute 2023-07-20 13:53:15.919376106 return self.handle_result(database, cursor) 2023-07-20 13:53:15.919398521 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 2851, in handle_result 2023-07-20 13:53:15.920233103 return database.last_insert_id(cursor, self._query_type) 2023-07-20 13:53:15.920240267 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3749, in last_insert_id 2023-07-20 13:53:15.921319357 return cursor.lastrowid 2023-07-20 13:53:15.921329068 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 87, in lastrowid 2023-07-20 13:53:15.921754032 self._wait() 2023-07-20 13:53:15.921760632 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 62, in _wait 2023-07-20 13:53:15.921858956 raise self._exc 2023-07-20 13:53:15.921865309 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 177, in execute 2023-07-20 13:53:15.922238758 cursor = self.database._execute(obj.sql, obj.params) 2023-07-20 13:53:15.922259582 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql 2023-07-20 13:53:15.925387773 cursor.execute(sql, params or ()) 2023-07-20 13:53:15.925411232 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3014, in exit 2023-07-20 13:53:15.926899574 reraise(new_type, new_type(exc_value, exc_args), traceback) 2023-07-20 13:53:15.926918710 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 192, in reraise 2023-07-20 13:53:15.927116317 raise value.with_traceback(tb) 2023-07-20 13:53:15.927134392 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 3246, in execute_sql 2023-07-20 13:53:15.928703866 cursor.execute(sql, params or ()) 2023-07-20 13:53:15.928727669 peewee.OperationalError: database is locked

NickM-27 commented 1 year ago

The next commit won't help because without execute() it wasn't even entering the item into the DB. I think at this point we need debug logs to understand what is going on because it makes absolutely no sense how the DB write is taking longer than 250 seconds

NickM-27 commented 1 year ago 
logger:
  logs:
    peewee: debug

should enable debug logs so we can get better info

NickM-27 commented 1 year ago

nevermind, doesn't seem like that shows the timing of the queries

NickM-27 commented 1 year ago

actually running that would still potentially be helpful because we can see how long there is between queries

YBonline commented 1 year ago

There seems to be delete operations in progress when it crashes which seems to be in line with the other issue I've had of when Frigate is automatically deleting old events, it uses a ton of CPU time.

frigate_sql_locked.txt

NickM-27 commented 1 year ago

That is a lot of single timeline deletes, I think I found a better way to clean these up with bulk deletes that should hopefully help, just updated the PR

YBonline commented 1 year ago

Alright, I'm running that but I'm getting so much stuff in the log file that I can't seem to go through that log now... is it logged to separate files somewhere with the frigate log separated from the debug info for peewee?

NickM-27 commented 1 year ago

The peewee debug info is part of the frigate logs, there is no difference.

YBonline commented 1 year ago

Alright, well I will continue to monitor, but so far 2.5 hours and no database is locked, and I'm more hopeful this time as Frigate seems more performant as well, Birdseye is updating quicker, etc. Probably won't know for sure until the morning if the error is fully gone but this definitely helped a lot so far.

YBonline commented 1 year ago

I think we are good!!!! 13 hours, no database locks, no signs of memory usage spikes, or any other issue for me now. I did shorten up my storage times for motion events since if it was auto expiring based off of free space, the problem and resource usage was far worse, so I'm going to go ahead and re-enable my original setting to see if that causes any issue, but it is looking like #7227 does fix the core issue for me for sure, thank you again! My load average is also now staying around 10 (with little motion) and 15 (with tons of motion) now, this is the best I've ever had running Frigate, by a large margin (nearly 50% reduction)

NickM-27 commented 1 year ago

If the original settings cause issues please create a new issue

YBonline commented 1 year ago

Alright, just wanted to follow up and report that this does seem to be fully fixed. In fact, the could not keep up with the cache errors, and most of my other camera errors seemed to have been related to this as well. Thanks again for all of the help!

xconverge commented 1 year ago

I haven't seen it since I started running the fixes either