search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.96k stars 1.64k forks source link

[Support]: I keep getting "unable to keep up with recording segments" and my footage skips a lot. #7297

Closed Majestic7979 closed 1 year ago

Majestic7979 commented 1 year ago

Describe the problem you are having

Per the title Log says "unable to keep up with recording segments". I don't remember ever getting this on 0.11. I am getting lots of skips in my recordings. The server is more powerful than the one I was using before, but the HDD is the same. The only difference is the Frigate version. What's wrong with my config? I enabled G711 audio in my camera's stream, this is the only difference, but Frigate does not accept it anyway. Even with AAC it doesn't really work and never has. I tried a lot of different ways/presets. Do I have to completely disable audio in the camera to make this work?

Version

0.12.1-367D724

Frigate config file

####################################
########## FRIGATE CONFIG ##########

mqtt:
  enabled: false

detectors:
  tensorrt:
    type: tensorrt
    device: 0

model:
  path: /trt-models/yolov7-tiny-416.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

ui:
  live_mode: jsmpeg
  timezone: Europe/London
  use_experimental: true

logger:
  default: info
  logs:
    frigate.event: info

database:
  path: /db/frigate.db

birdseye:
  enabled: True
  restream: False
  width: 1920
  height: 1080
    # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 15
  mode: continuous

detect:
  width: 2592
  height: 1944
  fps: 5
  enabled: true
  max_disappeared: 60

objects:
  track:
    - person

record:
  enabled: True
  retain:
    days: 10
    mode: all
  events:
    retain:
      default: 10

snapshots:
  enabled: False
  timestamp: true
  bounding_box: False
  crop: False
  retain:
    default: 5

rtmp:
  enabled: False

timestamp_style:
    # "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
  position: "tl"
    # Optional: Format specifier conform to the Python package "datetime" (default: shown below)
    #           Additional Examples:
    #             german: "%d.%m.%Y %H:%M:%S"
  format: "%m/%d/%Y %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2
    # None (No effect),
    # "solid" (solid background in inverse color of font)
    # "shadow" (shadow for font)
  effect: shadow

###################################
########## CAMERA CONFIG ##########

cameras:
  frigate_street_front:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp://10.0.0.100:8554/frigate_street_front
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      enabled: true

  frigate_street_nw:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264      
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp://10.0.0.100:8554/frigate_street_nw
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      enabled: true

  frigate_street_se:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264      
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp://10.0.0.100:8554/frigate_street_se
          input_args: preset-rtsp-generic    
          roles:
            - record
            - detect
    detect:
      enabled: true

  frigate_backyard:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264      
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp://10.0.0.100:8554/frigate_backyard
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      enabled: true

Relevant log output

2023-07-27 05:38:12.267286217  [INFO] Starting Frigate...
2023-07-27 05:38:14.454945761  [2023-07-27 05:38:14] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-27 05:38:14.497220341  [2023-07-27 05:38:14] peewee_migrate                 INFO    : Starting migrations
2023-07-27 05:38:14.509360514  [2023-07-27 05:38:14] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-27 05:38:14.534655211  [2023-07-27 05:38:14] frigate.app                    INFO    : Output process started: 866
2023-07-27 05:38:14.539646275  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_street_front: 869
2023-07-27 05:38:14.545823867  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 870
2023-07-27 05:38:14.553883919  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_street_se: 871
2023-07-27 05:38:14.560161238  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_backyard: 872
2023-07-27 05:38:14.560916236  [2023-07-27 05:38:14] detector.tensorrt              INFO    : Starting detection process: 864
2023-07-27 05:38:14.566928155  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-07-27 05:38:14.573195162  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-07-27 05:38:14.578949344  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_street_se: 881
2023-07-27 05:38:14.585735613  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_backyard: 884
2023-07-27 05:38:15.476234333  [2023-07-27 05:38:15] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 260, GPU 248 (MiB)
2023-07-27 05:38:15.695521137  [2023-07-27 05:38:15] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-07-27 05:38:15.697033215  [2023-07-27 05:38:15] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 05:38:16.626486368  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +1470, now: CPU 515, GPU 1754 (MiB)
2023-07-27 05:38:16.916109890  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +111, GPU +50, now: CPU 626, GPU 1804 (MiB)
2023-07-27 05:38:16.919309849  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 05:38:16.927456762  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 591, GPU 1796 (MiB)
2023-07-27 05:38:16.927461745  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 592, GPU 1804 (MiB)
2023-07-27 05:38:16.927466361  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 05:41:08.885525340  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-07-27 05:41:08.889404405  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-07-27 05:41:08.896003680  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-07-27 05:41:08.900249221  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 9 and discarding the rest...

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1944,"width":2592},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

-> all cameras are the same model and config

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"2.7"},"1":{"cpu":"0.0","mem":"0.0"},"106":{"cpu":"0.0","mem":"0.0"},"112":{"cpu":"1.3","mem":"1.5"},"1154":{"cpu":"0.0","mem":"0.0"},"118":{"cpu":"0.0","mem":"0.0"},"1189":{"cpu":"0.0","mem":"0.0"},"1195":{"cpu":"0.0","mem":"0.1"},"1200":{"cpu":"0.0","mem":"0.0"},"1202":{"cpu":"3.7","mem":"0.1"},"134":{"cpu":"0.0","mem":"0.0"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"149":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"168":{"cpu":"0.0","mem":"0.0"},"187":{"cpu":"0.0","mem":"0.0"},"216":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"241":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"263":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"271":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"280":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"290":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"303":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"326":{"cpu":"0.0","mem":"0.0"},"348":{"cpu":"0.0","mem":"0.0"},"384":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"425":{"cpu":"0.0","mem":"0.0"},"450":{"cpu":"0.0","mem":"0.0"},"475":{"cpu":"0.0","mem":"0.0"},"499":{"cpu":"0.0","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"858":{"cpu":"0.0","mem":"0.5"},"863":{"cpu":"0.3","mem":"0.0"},"864":{"cpu":"12.7","mem":"3.4"},"866":{"cpu":"2.0","mem":"0.6"},"869":{"cpu":"13.0","mem":"0.6"},"872":{"cpu":"52.3","mem":"0.7"},"873":{"cpu":"26.3","mem":"0.7"},"874":{"cpu":"0.7","mem":"0.6"},"875":{"cpu":"5.0","mem":"0.6"},"877":{"cpu":"5.3","mem":"0.6"},"879":{"cpu":"0.0","mem":"0.1"},"880":{"cpu":"5.3","mem":"0.6"},"885":{"cpu":"11.3","mem":"0.8"},"886":{"cpu":"11.3","mem":"0.8"},"887":{"cpu":"5.3","mem":"0.6"},"888":{"cpu":"0.0","mem":"0.1"},"889":{"cpu":"10.7","mem":"0.9"},"89":{"cpu":"0.0","mem":"0.1"},"895":{"cpu":"0.0","mem":"0.1"},"896":{"cpu":"12.3","mem":"0.8"},"898":{"cpu":"0.0","mem":"0.1"},"900":{"cpu":"0.0","mem":"0.1"},"MiB":{"cpu":"12239.3","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"average:","mem":"24.76,"}},"detection_fps":22.7,"detectors":{"tensorrt":{"detection_start":0.0,"inference_speed":4.66,"pid":864}},"frigate_backyard":{"camera_fps":5.0,"capture_pid":887,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":896,"pid":874,"process_fps":5.0,"skipped_fps":0.0},"frigate_street_front":{"camera_fps":5.0,"capture_pid":875,"detection_enabled":1,"detection_fps":5.2,"ffmpeg_pid":885,"pid":869,"process_fps":5.0,"skipped_fps":0.0},"frigate_street_nw":{"camera_fps":5.1,"capture_pid":877,"detection_enabled":1,"detection_fps":12.1,"ffmpeg_pid":886,"pid":872,"process_fps":5.1,"skipped_fps":0.0},"frigate_street_se":{"camera_fps":5.1,"capture_pid":880,"detection_enabled":1,"detection_fps":5.4,"ffmpeg_pid":889,"pid":873,"process_fps":5.1,"skipped_fps":0.0},"gpu_usages":{"NVIDIA GeForce GTX 1080":{"gpu":"6 %","mem":"22.2 %"}},"service":{"last_updated":1690430774,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":5335.6,"mount_type":"tmpfs","total":5368.7,"used":33.1},"/media/frigate/clips":{"free":458769.1,"mount_type":"ext4","total":2998506.7,"used":2416951.1},"/media/frigate/recordings":{"free":458769.1,"mount_type":"ext4","total":2998506.7,"used":2416951.1},"/tmp/cache":{"free":1915.6,"mount_type":"tmpfs","total":2000.0,"used":84.4}},"temperatures":{},"uptime":54,"version":"0.12.1-367d724"}}

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Anpviz 5MP

Any other information that may be helpful

No response

Majestic7979 commented 1 year ago

This is the camera details - my camera is basically the same one, but mine has 4x optical zoom. They don't sell it anymore, but it's pointless anyway since I never use the zoom... https://www.anpvizsecurity.com/products-detail-149850

Majestic7979 commented 1 year ago

image

Majestic7979 commented 1 year ago

image

Majestic7979 commented 1 year ago

image

Majestic7979 commented 1 year ago

image

Majestic7979 commented 1 year ago

image

Majestic7979 commented 1 year ago

2023-07-27 05:38:12.248355587 [INFO] Preparing go2rtc config... 2023-07-27 05:38:12.840348777 [INFO] Starting go2rtc... 2023-07-27 05:38:12.927546476 05:38:12.927 INF go2rtc version 1.2.0 linux/amd64 2023-07-27 05:38:12.927863464 05:38:12.927 INF [api] listen addr=:1984 2023-07-27 05:38:12.928059559 05:38:12.928 INF [rtsp] listen addr=:8554 2023-07-27 05:38:12.928251082 05:38:12.928 INF [srtp] listen addr=:8443 2023-07-27 05:38:12.928434646 05:38:12.928 INF [webrtc] listen addr=:8555 2023-07-27 05:38:22.250947703 [INFO] Starting go2rtc healthcheck service...

Majestic7979 commented 1 year ago 
2023-07-27 05:38:12.252644235  [INFO] Starting NGINX...
2023-07-27 05:38:12.787907773  2023/07/27 05:38:12 [error] 133#133: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.211, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001/stats", host: "10.0.0.100:5000"
2023-07-27 05:38:12.787913838  10.0.0.211 - - [27/Jul/2023:05:38:12 +0100] "GET /api/stats HTTP/1.1" 502 157 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.431646851  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.436619483  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.438426169  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.449435910  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.231424595  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3213 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.297277909  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3662 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.308432389  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.329202143  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3741 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:21.036429727  10.0.0.211 - - [27/Jul/2023:05:38:21 +0100] "GET /api/stats HTTP/1.1" 200 3557 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.889390774  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8561 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.890571543  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7279 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.891752588  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9158 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.915748783  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8396 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:29.059318788  10.0.0.211 - - [27/Jul/2023:05:38:29 +0100] "GET /api/stats HTTP/1.1" 200 3550 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.291292849  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3223 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.307715284  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3544 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.311573641  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3740 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.323592817  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3664 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.807765027  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.831799781  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9138 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.837743184  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7293 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.857281786  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8424 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:37.043264858  10.0.0.211 - - [27/Jul/2023:05:38:37 +0100] "GET /api/stats HTTP/1.1" 200 3593 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.206203918  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3739 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.222862769  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3556 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.263576095  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.264607948  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3660 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.043740305  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/stats HTTP/1.1" 200 3627 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.867897063  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7303 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.879043706  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8557 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.901132245  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8455 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.903781198  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9163 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.201393673  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3558 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.212774584  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3745 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.230717375  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3649 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.230722470  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3238 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:53.119573612  10.0.0.211 - - [27/Jul/2023:05:38:53 +0100] "GET /api/stats HTTP/1.1" 200 3647 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.852768669  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.874669846  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7308 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.889875630  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9188 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.895579911  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8459 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.299693496  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3235 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.307231210  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3534 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.340042518  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3652 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.340241709  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3755 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:01.051734661  10.0.0.211 - - [27/Jul/2023:05:39:01 +0100] "GET /api/stats HTTP/1.1" 200 3615 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.856406955  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8570 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.916682313  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7315 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.935457308  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9180 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.939538823  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8422 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:09.079743866  10.0.0.211 - - [27/Jul/2023:05:39:09 +0100] "GET /api/stats HTTP/1.1" 200 3583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.249082932  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.256962140  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3542 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.261053412  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3652 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.265379335  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3758 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.831576485  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8581 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.841112385  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9177 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.861353707  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8416 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.862452710  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7302 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:17.027732752  10.0.0.211 - - [27/Jul/2023:05:39:17 +0100] "GET /api/stats HTTP/1.1" 200 3584 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.199703053  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3250 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.223576289  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3540 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.243577429  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3755 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.259714764  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3665 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.056271979  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/stats HTTP/1.1" 200 3664 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.851576288  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9233 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.855568354  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7314 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.859734345  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8529 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.876347884  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8449 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.302785553  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.310445539  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.335745205  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3768 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.365298962  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3651 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:33.064522539  10.0.0.211 - - [27/Jul/2023:05:39:33 +0100] "GET /api/stats HTTP/1.1" 200 3598 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.834617599  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8540 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.857656229  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7300 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.862026021  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8440 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.883778149  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9237 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.210843173  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3244 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.218080399  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3563 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.246018290  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3782 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.267585193  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3661 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:41.033828433  10.0.0.211 - - [27/Jul/2023:05:39:41 +0100] "GET /api/stats HTTP/1.1" 200 3585 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.875895779  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8591 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.892384057  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7415 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.900027351  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9247 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.906613740  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8462 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:49.047173785  10.0.0.211 - - [27/Jul/2023:05:39:49 +0100] "GET /api/stats HTTP/1.1" 200 3628 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.237548382  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3535 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.251151712  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3667 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.254006903  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3772 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.263504692  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3183 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.881816093  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7229 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.883126887  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8532 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.891740992  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8369 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.896643805  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9247 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:57.060453696  10.0.0.211 - - [27/Jul/2023:05:39:57 +0100] "GET /api/stats HTTP/1.1" 200 3617 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.191185760  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3193 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.195884250  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3530 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.201606413  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3784 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.228000964  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3648 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:05.056632599  10.0.0.211 - - [27/Jul/2023:05:40:05 +0100] "GET /api/stats HTTP/1.1" 200 3598 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.196302668  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3208 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.219712275  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3531 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.222178318  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3774 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.238280609  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3637 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:13.044757436  10.0.0.211 - - [27/Jul/2023:05:40:13 +0100] "GET /api/stats HTTP/1.1" 200 3583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.838428741  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8548 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.887592170  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7232 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.899749130  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8387 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.907584023  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9227 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:17.832151443  10.0.0.211 - - [27/Jul/2023:05:40:17 +0100] "GET /config HTTP/1.1" 200 6641 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:17.879712565  10.0.0.211 - - [27/Jul/2023:05:40:17 +0100] "GET /api/config HTTP/1.1" 200 12623 "https://mydomain.example/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:18.163871560  10.0.0.211 - - [27/Jul/2023:05:40:18 +0100] "GET /api/config/raw HTTP/1.1" 200 1002 "https://mydomain.example/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:18.431577478  10.0.0.211 - - [27/Jul/2023:05:40:18 +0100] "GET //api/config/schema.json HTTP/1.1" 200 25729 "https://mydomain.example/assets/yaml.worker.bundle.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:20.242401440  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3195 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.262206588  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3788 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.262212030  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3522 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.278525963  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3653 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:21.043582568  10.0.0.211 - - [27/Jul/2023:05:40:21 +0100] "GET /api/stats HTTP/1.1" 200 3650 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:22.710410903  10.0.0.211 - - [27/Jul/2023:05:40:22 +0100] "GET /api/stats HTTP/1.1" 200 3648 "https://mydomain.example/system" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:25.879279294  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8542 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.899580698  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7242 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.909508582  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9258 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.929518059  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8381 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:26.761249705  10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/nginx HTTP/1.1" 200 2532 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:26.777079919  10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/go2rtc HTTP/1.1" 200 258 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:26.800516415  10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/frigate HTTP/1.1" 200 821 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:28.293681829  10.0.0.211 - - [27/Jul/2023:05:40:28 +0100] "GET /api/stats HTTP/1.1" 200 3683 "https://mydomain.example/storage" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:28.509471747  10.0.0.211 - - [27/Jul/2023:05:40:28 +0100] "GET /api/recordings/storage HTTP/1.1" 200 390 "https://mydomain.example/storage" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:29.052124839  10.0.0.211 - - [27/Jul/2023:05:40:29 +0100] "GET /api/stats HTTP/1.1" 200 3683 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.178105819  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3218 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.238486528  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3541 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.247574161  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3781 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.299575305  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3650 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:35.887575575  10.0.0.211 - - [27/Jul/2023:05:40:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8571 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:35.923575548  10.0.0.211 - - [27/Jul/2023:05:40:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7263 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:36.799577193  10.0.0.211 - - [27/Jul/2023:05:40:36 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:36.847569285  10.0.0.211 - - [27/Jul/2023:05:40:36 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8423 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:37.043577178  10.0.0.211 - - [27/Jul/2023:05:40:37 +0100] "GET /api/stats HTTP/1.1" 200 3618 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:39.968607191  10.0.0.211 - - [27/Jul/2023:05:40:39 +0100] "GET /api/config HTTP/1.1" 200 12623 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:39.970618926  10.0.0.211 - - [27/Jul/2023:05:40:39 +0100] "GET /api/logs/go2rtc HTTP/1.1" 200 258 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
Majestic7979 commented 1 year ago

Host:



server.local system information |SYSTEM INFORMATION     | | |---|---| | System hostname | server.local (127.0.1.1) | | Operating system | Debian Linux 12 | | Webmin version | 2.100 | | Authentic theme version | 21.01-2 | | Time on system | Thursday, July 27, 2023 5:48 AM | | Kernel and CPU | Linux 6.1.0-10-amd64 on x86_64 | | Processor information | 13th Gen Intel(R) Core(TM) i5-13500, 20 cores | | CPU temperatures | Core 1: 97°CCore 5: 96°CCore 9: 99°CCore 13: 100°CCore 17: 98°CCore 21: 100°CCore 25: 92°CCore 26: 92°CCore 27: 92°CCore 28: 92°CCore 29: 89°CCore 30: 89°CCore 31: 89°CCore 32: 89°C | | CPU fans | Fan 1: 0 RPMFan 2: 3125 RPMFan 6: 0 RPMFan 7: 0 RPM | | Drive temperatures | nvme0n1: 67°Csda: 44°Csdb: 43°Csdc: 41°C | | System uptime | 4 hours, 52 minutes | | Running processes | 936 | | CPU load averages | 32.26 (1 min) 34.29 (5 mins) 34.86 (15 mins) | | Real memory | 18.58 GiB used / 12.22 GiB cached / 31.08 GiB total | | Virtual memory | 7.35 GiB used / 63.99 GiB total | | Local disk space | 6.54 TiB used / 11.55 TiB free / 18.09 TiB total | | Package updates | All installed packages are up to date |


Majestic7979 commented 1 year ago

I want you to ignore the CPU temps. It's high because ffmpeg is running on another process (immich) via docker to transcode some videos. The issue I have with Frigate not keeping up with frames happens even when the CPU is not stressed. Regardless, if you will insist that this is the issue let me know how to force the frigate container to run as a priority because my security system must always be prioritized above all else.

blakeblackshear commented 1 year ago

Is the disk where recordings are stored on the local machine or are you storing them on a network share?

You can turn on debug logging to see the copy times for moving recordings:

logger:
  logs:
    frigate.record: debug

If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up.

I do not know of a way to specify priority in HassOS for an addon.

Alternatively, you can use motion mode instead of all mode for record so fewer segments are copied over. This has helped several users.

Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps.

Majestic7979 commented 1 year ago

Is the disk where recordings are stored on the local machine or are you storing them on a network share?

You can turn on debug logging to see the copy times for moving recordings:

logger:
  logs:
    frigate.record: debug

If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up.

I do not know of a way to specify priority in HassOS for an addon.

Alternatively, you can use motion mode instead of all mode for record so fewer segments are copied over. This has helped several users.

Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps.

It's a SATA disk in the host device.

I get this issue even when the CPU is in the low percent of usage. I knew this was going to come up... It may be that, I'm not going to say "it's not that", without data neither of us don't know, so it might be. But it happens even when there's no high CPU utilization. Nick told me to increase shm, he said that was the issue. But it didn't fix, you can see my shm is quite high. As per above form answer I don't use HassOS, I run directly on Docker on a Debian 12 system.

I cannot use motion for recording, It's simply not reliable enough and the police said they need an hour before and an hour after any incident and if there is any skipping they cannot accept the evidence.

The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res.

Is the issue related to motion detection? Or is the issue related to recording?

I can try motion masks but the whole point of this project is smart detection isn't it... I mean nobody can say my system is slow, it's better than most people's main personal computers!!

I used to run on a 8GB DDR3 Intel Core i7 3770T processor without GPU (apart from built-in) Now I run on a 32GB DDR5 Intel Core i5 13500 which is latest gen with many more cores and a relatively beefy GTX1080 GPU and it can't keep up.... Something is not right. This is what I said in my "opinion" - Frigate is focusing a lot on delivering features and skipping on stability, meaning the footage is unusable when it's most required to prosecute a criminal. Honestly, please rethink the approach here. I've seen go2rtc get added with added complexity and added bugs... meanwhile we're seeing a system of this level struggle!? It's not a criticism it's just a fact that's happening right now. Why not focus on multiprocessing/GPU processing? Apparently even with a GPU the CPU is still used quite a lot for some tasks like motion. That should not be happening if there is such a beefy GPU in the host.

But that was digressing. I just need support for the issue right now.

In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)?

Oh and I know I shouldn't expect anyone to assume but it goes without saying I appreciate your time and effort in building this application and supporting others and myself. Big thank you. Never think I am criticizing you.

NickM-27 commented 1 year ago

The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res.

This is no longer true, if you use ffmpeg presets for hwaccel (like you are) then the scaling will be done on the GPU instead so it is not a problem.

Frigate is focusing a lot on delivering features and skipping on stability, meaning the footage is unusable when it's most required to prosecute a criminal. Honestly, please rethink the approach here. I've seen go2rtc get added with added complexity and added bugs... meanwhile we're seeing a system of this level struggle!? It's not a criticism it's just a fact that's happening right now. Why not focus on multiprocessing/GPU processing?

That is objectively false. You can look at the commits for the upcoming 0.13 release, there have been many multiprocessing improvements for recording management and otherwise. But for now the recommendations are made for the current version, which is mainly to reduce the amount of segments that are kept so the recordings maintainer has less work to do.

In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)?

The record debug logging will show the amount of time it takes to move the recording segments from cache to the storage drive, which is crucial to understanding where the slow down is.

NickM-27 commented 1 year ago

Also just wanted to confirm, the /db/ volume is not on the Sata SSD correct?

blakeblackshear commented 1 year ago

Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you.

Majestic7979 commented 1 year ago

Hey Nick.

What resolution do you suggest me to use then?

Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability.

I will enable the debug logging now, thank you.

The /db/ is on the SATA drive because I run my container with a bind mount, and the mount is stored in the /home partition which is in a 14TB mirroring RAID.

This is my docker CLI:

docker create --name frigate --restart=always -p 5000:5000 --gpus=all --cap-add=CAP_PERFMON --mount type=tmpfs,target=/tmp/cache,tmpfs-size=2000000000 --shm-size=5g --device /dev/bus/usb:/dev/bus/usb --device /dev/dri/renderD128 -v /cctv:/media/frigate -v /home/user1/dockerbindmounts/frigate/config.yml:/config/config.yml -v /home/user1/dockerbindmounts/frigate/db/:/db/ -v /etc/localtime:/etc/localtime:ro -v /home/user1/trt-models:/trt-models -e FRIGATE_RTSP_PASSWORD='removedforprivacy' -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,video,utility -e YOLO_MODELS=yolov7-tiny-416 ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt &&  docker start frigate
Majestic7979 commented 1 year ago

Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you.

Right and that's what is making me scratch my head.

Majestic7979 commented 1 year ago

For info I run a 3TB WD Red Pro drive for my footage, it's mounted on /cctv

Majestic7979 commented 1 year ago

To make absolutely clear I store my footage in a separate SATA drive (3TB WD Red Pro) and my frigate db is in the main RAID array (14TB). The DB and the frigate config/db are in separate storage drives therefore read/write are separate...

NickM-27 commented 1 year ago

What resolution do you suggest me to use then?

Looks like your cameras are 4:3 so you could try 

detect:
  width: 1280
  height: 960

and see if the objects are detected as you would like. Without seeing the camera images it is difficult to suggest fully because smaller objects that are farther away of course require more resolution to be detected reliably. I run all of my cameras with a height of 720 (and the 4:3 cameras with a height of 960) and have not had any issues detection people far away from the camera.

Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability.

I would suggest that there were a number of stability improvements in 0.12 as well, but I understand where you're coming from and of course you wouldn't know about the commits in the unreleased version.

The to-do list for 0.13 is getting closer so a beta should not be too far away.

Another thing that could be tried here given that you require all recording instead of motion recording is reducing your cameras bit rate so there is less data for the recordings maintainer to move.

I will enable the debug logging now, thank you.

Great, the values in general will be helpful but it will be mostly helpful to see what the copy times are after the segment cache errors start occurring.

Majestic7979 commented 1 year ago

Cam image: [removed for privacy]

Thanks, that's an empathetic response! I really appreciate it!!! I am doing all I can to try to help you help me, and I know you're a stellar person here with your support. It's really nice when someone recognizes comments and takes it on board, understanding that the intentions are always good.

The problem with reducing the bitrate is that it will likely pixellate... this will greatly affect number plate in vehicles especially at night. But the cameras are all feeding directly into a Ubiquiti USW-24-PoE switch which isn't even fully utilized, each camera obviously using their own cable and going into their specific switch port. There is no "in between", it connects directly to my main switch, which the server is also directly connected into.

My cameras are outputting to MediaMTX (rtsp-simple-server) and I input the proxied stream into Frigate. If I input the MediaMTX proxied stream for any of the cams into VLC I get a "perfect" stream. I put perfect into quotes because the audio, for some reason, is slow-mo. You know when you record someone speaking and then you slow-mo and the person sounds like their voice is deeper and slower? This is happening and I don't know why. The video itself is real-time, there is a timestamp as you can see above and it's playing back second to second as expected. I'll now post a new log with debug in a separate comment

Majestic7979 commented 1 year ago 
2023-07-27 14:51:24.068200519  [INFO] Starting Frigate...
2023-07-27 14:51:25.991688431  [2023-07-27 14:51:25] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-27 14:51:26.035304264  [2023-07-27 14:51:26] peewee_migrate                 INFO    : Starting migrations
2023-07-27 14:51:26.071227368  [2023-07-27 14:51:26] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-27 14:51:26.101592142  [2023-07-27 14:51:26] frigate.app                    INFO    : Output process started: 865
2023-07-27 14:51:26.103897208  [2023-07-27 14:51:26] detector.tensorrt              INFO    : Starting detection process: 864
2023-07-27 14:51:26.128042669  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_street_front: 869
2023-07-27 14:51:26.128046642  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 870
2023-07-27 14:51:26.128049471  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_street_se: 871
2023-07-27 14:51:26.128051525  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_backyard: 872
2023-07-27 14:51:26.141239340  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-07-27 14:51:26.146591593  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-07-27 14:51:26.153775457  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_street_se: 882
2023-07-27 14:51:26.159047702  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_backyard: 885
2023-07-27 14:51:26.813366819  [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 248 (MiB)
2023-07-27 14:51:26.978411519  [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-07-27 14:51:26.979692337  [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 14:51:27.364548466  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 358 (MiB)
2023-07-27 14:51:27.557194923  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 404 (MiB)
2023-07-27 14:51:27.559115628  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 14:51:27.565314022  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 396 (MiB)
2023-07-27 14:51:27.565484698  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 404 (MiB)
2023-07-27 14:51:27.565488372  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 14:51:44.271330251  [2023-07-27 14:51:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.27.mp4 in 2.8024659156799316 seconds.
2023-07-27 14:51:47.419609909  [2023-07-27 14:51:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.27.mp4 in 2.8160011768341064 seconds.
2023-07-27 14:51:50.261774582  [2023-07-27 14:51:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.27.mp4 in 2.7169740200042725 seconds.
2023-07-27 14:51:53.223402149  [2023-07-27 14:51:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.27.mp4 in 2.8212969303131104 seconds.
2023-07-27 14:51:56.121896583  [2023-07-27 14:51:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.36.mp4 in 2.7229220867156982 seconds.
2023-07-27 14:51:59.070124450  [2023-07-27 14:51:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.36.mp4 in 2.8004510402679443 seconds.
2023-07-27 14:52:01.971288144  [2023-07-27 14:52:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.36.mp4 in 2.7553539276123047 seconds.
2023-07-27 14:52:05.733478023  [2023-07-27 14:52:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.37.mp4 in 3.5945658683776855 seconds.
2023-07-27 14:52:08.670631442  [2023-07-27 14:52:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.46.mp4 in 2.7888638973236084 seconds.
2023-07-27 14:52:11.538703695  [2023-07-27 14:52:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.46.mp4 in 2.696091890335083 seconds.
2023-07-27 14:52:14.378159551  [2023-07-27 14:52:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.46.mp4 in 2.714629888534546 seconds.
2023-07-27 14:52:17.266218893  [2023-07-27 14:52:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.47.mp4 in 2.741323947906494 seconds.
2023-07-27 14:52:20.299887659  [2023-07-27 14:52:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.56.mp4 in 2.8629918098449707 seconds.
2023-07-27 14:52:23.280067169  [2023-07-27 14:52:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.06.mp4 in 2.8107500076293945 seconds.
2023-07-27 14:52:26.167934835  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.56.mp4 in 2.734575033187866 seconds.
2023-07-27 14:52:26.336420434  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start expire recordings (new).
2023-07-27 14:52:26.336423851  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start deleted cameras.
2023-07-27 14:52:26.919573876  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Expiring 0 recordings
2023-07-27 14:52:26.923573589  [2023-07-27 14:52:26] frigate.record                 DEBUG   : End deleted cameras.
2023-07-27 14:52:26.923578160  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start all cameras.
2023-07-27 14:52:26.923580102  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start camera: frigate_street_front.
2023-07-27 14:52:29.056176497  [2023-07-27 14:52:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.06.mp4 in 2.7186970710754395 seconds.
2023-07-27 14:52:32.019877456  [2023-07-27 14:52:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.56.mp4 in 2.780247926712036 seconds.
2023-07-27 14:52:35.067947513  [2023-07-27 14:52:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.06.mp4 in 2.833055019378662 seconds.
2023-07-27 14:52:38.012625476  [2023-07-27 14:52:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.57.mp4 in 2.7677149772644043 seconds.
2023-07-27 14:52:40.893284633  [2023-07-27 14:52:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.07.mp4 in 2.7566659450531006 seconds.
2023-07-27 14:52:43.747509775  [2023-07-27 14:52:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.16.mp4 in 2.7001211643218994 seconds.
2023-07-27 14:52:46.693283952  [2023-07-27 14:52:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.27.mp4 in 2.7567219734191895 seconds.
2023-07-27 14:52:49.649205452  [2023-07-27 14:52:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.16.mp4 in 2.7733681201934814 seconds.
2023-07-27 14:52:52.815865164  [2023-07-27 14:52:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.26.mp4 in 2.766500949859619 seconds.
2023-07-27 14:52:55.708985391  [2023-07-27 14:52:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.16.mp4 in 2.753082036972046 seconds.
2023-07-27 14:52:58.697371241  [2023-07-27 14:52:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.26.mp4 in 2.7924540042877197 seconds.
2023-07-27 14:53:01.773608922  [2023-07-27 14:53:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.17.mp4 in 2.83032488822937 seconds.
2023-07-27 14:53:04.880077904  [2023-07-27 14:53:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.27.mp4 in 2.8705949783325195 seconds.
2023-07-27 14:53:07.837504876  [2023-07-27 14:53:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.37.mp4 in 2.772428035736084 seconds.
2023-07-27 14:53:10.835078066  [2023-07-27 14:53:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.47.mp4 in 2.780242919921875 seconds.
2023-07-27 14:53:13.794528070  [2023-07-27 14:53:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.36.mp4 in 2.7508840560913086 seconds.
2023-07-27 14:53:16.815387919  [2023-07-27 14:53:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.46.mp4 in 2.7242228984832764 seconds.
2023-07-27 14:53:20.046110795  [2023-07-27 14:53:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.36.mp4 in 3.063145160675049 seconds.
2023-07-27 14:53:23.023597905  [2023-07-27 14:53:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.46.mp4 in 2.789849042892456 seconds.
2023-07-27 14:53:25.665285688  [2023-07-27 14:53:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.37.mp4 in 2.4909911155700684 seconds.
2023-07-27 14:53:28.683837655  [2023-07-27 14:53:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.46.mp4 in 2.7974019050598145 seconds.
2023-07-27 14:53:31.696797664  [2023-07-27 14:53:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.57.mp4 in 2.8094191551208496 seconds.
2023-07-27 14:53:34.678908144  [2023-07-27 14:53:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.07.mp4 in 2.807781934738159 seconds.
2023-07-27 14:53:37.700928489  [2023-07-27 14:53:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.17.mp4 in 2.8578808307647705 seconds.
2023-07-27 14:53:40.810218748  [2023-07-27 14:53:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.56.mp4 in 2.919498920440674 seconds.
2023-07-27 14:53:43.785736486  [2023-07-27 14:53:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.06.mp4 in 2.787822961807251 seconds.
2023-07-27 14:53:46.896286987  [2023-07-27 14:53:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.16.mp4 in 2.8843400478363037 seconds.
2023-07-27 14:53:49.849318334  [2023-07-27 14:53:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.57.mp4 in 2.8049111366271973 seconds.
2023-07-27 14:53:54.355680230  [2023-07-27 14:53:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.07.mp4 in 4.337353944778442 seconds.
2023-07-27 14:53:59.343875193  [2023-07-27 14:53:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.17.mp4 in 4.821834087371826 seconds.
2023-07-27 14:54:04.400656221  [2023-07-27 14:54:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.56.mp4 in 4.908252954483032 seconds.
2023-07-27 14:54:09.597482962  [2023-07-27 14:54:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.06.mp4 in 5.061578989028931 seconds.
2023-07-27 14:54:14.727572184  [2023-07-27 14:54:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.16.mp4 in 4.95068883895874 seconds.
2023-07-27 14:54:18.858567075  [2023-07-27 14:54:18] frigate.record                 DEBUG   : Expiring 192 recordings
2023-07-27 14:54:19.030255548  [2023-07-27 14:54:19] frigate.record                 DEBUG   : End camera: frigate_street_front.
2023-07-27 14:54:19.030336823  [2023-07-27 14:54:19] frigate.record                 DEBUG   : Start camera: frigate_street_nw.
2023-07-27 14:54:19.209223005  [2023-07-27 14:54:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.27.mp4 in 4.3155388832092285 seconds.
2023-07-27 14:54:22.139087749  [2023-07-27 14:54:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.37.mp4 in 2.776745080947876 seconds.
2023-07-27 14:54:27.209055086  [2023-07-27 14:54:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.47.mp4 in 4.884838819503784 seconds.
2023-07-27 14:54:32.727577331  [2023-07-27 14:54:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.57.mp4 in 4.9430060386657715 seconds.
2023-07-27 14:54:37.911973178  [2023-07-27 14:54:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.26.mp4 in 5.032839059829712 seconds.
2023-07-27 14:54:43.181262654  [2023-07-27 14:54:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.36.mp4 in 5.1069581508636475 seconds.
2023-07-27 14:54:48.060652981  [2023-07-27 14:54:48] frigate.record                 DEBUG   : Expiring 190 recordings
2023-07-27 14:54:48.115701657  [2023-07-27 14:54:48] frigate.record                 DEBUG   : End camera: frigate_street_nw.
2023-07-27 14:54:48.115706041  [2023-07-27 14:54:48] frigate.record                 DEBUG   : Start camera: frigate_street_se.
2023-07-27 14:54:48.141978684  [2023-07-27 14:54:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.46.mp4 in 4.81535005569458 seconds.
2023-07-27 14:54:51.013401169  [2023-07-27 14:54:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.56.mp4 in 2.715502977371216 seconds.
2023-07-27 14:54:55.920729949  [2023-07-27 14:54:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.27.mp4 in 4.715717077255249 seconds.
2023-07-27 14:55:01.098977707  [2023-07-27 14:55:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.37.mp4 in 4.994410991668701 seconds.
2023-07-27 14:55:06.160359378  [2023-07-27 14:55:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.47.mp4 in 4.908405065536499 seconds.
2023-07-27 14:55:11.219192641  [2023-07-27 14:55:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.57.mp4 in 4.902930021286011 seconds.
2023-07-27 14:55:15.146357792  [2023-07-27 14:55:15] frigate.record                 DEBUG   : Expiring 173 recordings
2023-07-27 14:55:15.222282726  [2023-07-27 14:55:15] frigate.record                 DEBUG   : End camera: frigate_street_se.
2023-07-27 14:55:15.222387908  [2023-07-27 14:55:15] frigate.record                 DEBUG   : Start camera: frigate_backyard.
2023-07-27 14:55:15.498984302  [2023-07-27 14:55:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.26.mp4 in 4.104842185974121 seconds.
2023-07-27 14:55:18.325911082  [2023-07-27 14:55:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.36.mp4 in 2.706555128097534 seconds.
2023-07-27 14:55:23.467614561  [2023-07-27 14:55:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.46.mp4 in 4.991831064224243 seconds.
2023-07-27 14:55:28.530655564  [2023-07-27 14:55:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.56.mp4 in 4.9489710330963135 seconds.
2023-07-27 14:55:28.561354799  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.565064279  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.567689923  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.570692819  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:33.305559493  [2023-07-27 14:55:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.36.mp4 in 4.606783151626587 seconds.
2023-07-27 14:55:38.091694580  [2023-07-27 14:55:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.46.mp4 in 4.640094995498657 seconds.
2023-07-27 14:55:43.146840524  [2023-07-27 14:55:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.56.mp4 in 4.912054061889648 seconds.
2023-07-27 14:55:44.605703963  [2023-07-27 14:55:44] frigate.record                 DEBUG   : Expiring 192 recordings
2023-07-27 14:55:44.705574828  [2023-07-27 14:55:44] frigate.record                 DEBUG   : End camera: frigate_backyard.
2023-07-27 14:55:44.705579297  [2023-07-27 14:55:44] frigate.record                 DEBUG   : End all cameras.
2023-07-27 14:55:44.705581368  [2023-07-27 14:55:44] frigate.record                 DEBUG   : End expire recordings (new).
2023-07-27 14:55:44.705582996  [2023-07-27 14:55:44] frigate.record                 DEBUG   : Start expire files (legacy).
2023-07-27 14:55:44.751749462  [2023-07-27 14:55:44] frigate.record                 DEBUG   : Oldest recording in the db: 1689601642.515418
2023-07-27 14:55:48.178487178  [2023-07-27 14:55:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/55.06.mp4 in 4.88518500328064 seconds.
2023-07-27 14:55:51.890108621  [2023-07-27 14:55:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/55.16.mp4 in 3.5519468784332275 seconds.
2023-07-27 14:55:56.885556220  [2023-07-27 14:55:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.36.mp4 in 4.819516897201538 seconds.
2023-07-27 14:56:01.939756131  [2023-07-27 14:56:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.46.mp4 in 4.904176950454712 seconds.
2023-07-27 14:56:06.808378756  [2023-07-27 14:56:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.56.mp4 in 4.73555588722229 seconds.
2023-07-27 14:56:12.329512784  [2023-07-27 14:56:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/55.07.mp4 in 5.364140033721924 seconds.
2023-07-27 14:56:18.414092681  [2023-07-27 14:56:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/55.17.mp4 in 5.933212041854858 seconds.
2023-07-27 14:56:24.186002145  [2023-07-27 14:56:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.36.mp4 in 5.628190040588379 seconds.
2023-07-27 14:56:30.308024214  [2023-07-27 14:56:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.46.mp4 in 5.9718639850616455 seconds.
2023-07-27 14:56:35.452113307  [2023-07-27 14:56:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.56.mp4 in 4.97012996673584 seconds.
2023-07-27 14:56:40.990185255  [2023-07-27 14:56:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/55.06.mp4 in 5.417521953582764 seconds.
2023-07-27 14:56:46.175705449  [2023-07-27 14:56:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/55.16.mp4 in 4.98787784576416 seconds.
2023-07-27 14:56:51.373647527  [2023-07-27 14:56:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.36.mp4 in 5.038317918777466 seconds.
2023-07-27 14:56:54.940395663  [2023-07-27 14:56:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.46.mp4 in 3.3942999839782715 seconds.
2023-07-27 14:56:58.543646121  [2023-07-27 14:56:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.56.mp4 in 3.4380130767822266 seconds.
2023-07-27 14:57:02.081852655  [2023-07-27 14:57:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/55.06.mp4 in 3.396026134490967 seconds.
NickM-27 commented 1 year ago

Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view.

As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written.

I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now.

Majestic7979 commented 1 year ago

Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view.

As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written.

I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now.

Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.

I suggest... How about we look into my camera output in more detail and see if we can debug what's being sent to Frigate? Do you want a raw output stream? I can send you a link from cloud storage for debug purposes. What do you propose other than the storage? It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.

SMART output follows in next comment

Majestic7979 commented 1 year ago 
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-10-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-xxxxxxxxxx
LU WWN Device Id: 5 0014ee xxxxxx
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jul 27 16:53:55 2023 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (40500) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 406) minutes.
Conveyance self-test routine
recommended polling time:    (   5) minutes.
SCT capabilities:          (0x703d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   191   175   021    Pre-fail  Always       -       5450
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       852
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   001   001   000    Old_age   Always       -       78776
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       775
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       226
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       625
194 Temperature_Celsius     0x0022   110   103   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     52260         -
# 2  Short offline       Completed without error       00%     52236         -
# 3  Short offline       Completed without error       00%     46009         -
# 4  Short offline       Completed without error       00%     45985         -
# 5  Short offline       Completed without error       00%     45961         -
# 6  Short offline       Completed without error       00%     45937         -
# 7  Short offline       Completed without error       00%     45913         -
# 8  Short offline       Completed without error       00%     45889         -
# 9  Short offline       Completed without error       00%     45865         -
#10  Short offline       Completed without error       00%     45841         -
#11  Short offline       Completed without error       00%     45817         -
#12  Short offline       Completed without error       00%     45793         -
#13  Short offline       Completed without error       00%     45769         -
#14  Short offline       Completed without error       00%     45745         -
#15  Short offline       Completed without error       00%     45721         -
#16  Short offline       Completed without error       00%     45697         -
#17  Short offline       Completed without error       00%     45673         -
#18  Short offline       Completed without error       00%     45649         -
#19  Short offline       Completed without error       00%     45625         -
#20  Extended offline    Completed without error       00%     43179         -
#21  Short offline       Completed without error       00%      8992         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Majestic7979 commented 1 year ago

image

NickM-27 commented 1 year ago

Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.

Frigate 0.11 did not have these warning logs so there would be no way to know except notice the recording segment wasn't there.

It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.

For one it's not constant write it's separate writes for each segment. The timing logs are very simple around the ffmpeg call to optimize the segment metadata and write to storage so there's no room for error in that calculation.

Can you show your docker compose? One explanation would be if /tmp/cache isn't using RAM but is using disk instead.

Majestic7979 commented 1 year ago

image I cannot do write test because I'm on the live system

Majestic7979 commented 1 year ago

Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.

Frigate 0.11 did not have these warning logs so there would be no way to know except notice the recording segment wasn't there.

It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.

For one it's not constant write it's separate writes for each segment. The timing logs are very simple around the ffmpeg call to optimize the segment metadata and write to storage so there's no room for error in that calculation.

Can you show your docker compose? One explanation would be if /tmp/cache isn't using RAM but is using disk instead.

I don't use compose. Per previous reply I posted by CLI command. Copying again for your convenience:

docker create --name frigate --restart=always -p 5000:5000 --gpus=all --cap-add=CAP_PERFMON --mount type=tmpfs,target=/tmp/cache,tmpfs-size=2000000000 --shm-size=5g --device /dev/bus/usb:/dev/bus/usb --device /dev/dri/renderD128 -v /cctv:/media/frigate -v /home/user1/dockerbindmounts/frigate/config.yml:/config/config.yml -v /home/user1/dockerbindmounts/frigate/db/:/db/ -v /etc/localtime:/etc/localtime:ro -v /home/user1/trt-models:/trt-models -e FRIGATE_RTSP_PASSWORD='removedforprivacy' -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,video,utility -e YOLO_MODELS=yolov7-tiny-416 ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt &&  docker start frigate

"/home/user1" is in a mirrored RAID of 14TB. "/cctv" is the 3TB disk I posted above where Frigate stores the recordings.

NickM-27 commented 1 year ago

Right. For one I would suggest setting shm size to 1g. I can't remember the specifics of my previous recommendation but for 4 cameras 5g is excessive.

We saw another case with a user seeing this issue where for some reason when they specified 2GB for /tmp/cache it would use the storage but 1GB would use RAM. If you run df inside the frigate container it should give more info on the type of mount that is being used for /tmp/cache

Majestic7979 commented 1 year ago

docker exec -it frigate bash root@4ea5bdb42c53:/opt/frigate# df Filesystem 1K-blocks Used Available Use% Mounted on overlay 256920004 29829712 213966708 13% / tmpfs 65536 0 65536 0% /dev shm 5242880 11752 5231128 1% /dev/shm /dev/mapper/RAID-Home 15625744384 4463767664 11159045040 29% /trt-models /dev/sdc1 2928229216 2354485980 453834536 84% /media/frigate /dev/nvme0n1p3 256920004 12746140 231050280 6% /usr/share/zoneinfo/Etc/UTC /dev/nvme0n1p4 256920004 29829712 213966708 13% /etc/hosts tmpfs 1953128 23568 1929560 2% /tmp/cache tmpfs 16297808 12 16297796 1% /proc/driver/nvidia tmpfs 3259564 6276 3253288 1% /run/nvidia-persistenced/socket udev 16251384 0 16251384 0% /dev/nvidia0 tmpfs 16297808 0 16297808 0% /proc/asound tmpfs 16297808 0 16297808 0% /proc/acpi tmpfs 16297808 0 16297808 0% /sys/firmware root@4ea5bdb42c53:/opt/frigate#

I increased the shm considerably just so there was absolutely zero doubt that it's not that.

Majestic7979 commented 1 year ago

shm now 1g.

NickM-27 commented 1 year ago

What do copy times look like currently?

Majestic7979 commented 1 year ago

Do you mean you want the logs? or benchmark from gnome disk?

NickM-27 commented 1 year ago

logs

Majestic7979 commented 1 year ago

I just spun a new container because of changing the cli paramenter for the shm so frigate has not been running for more than 5 mins. Here's what it's showing so far and don't worry it will show the same issue we're discussing in a few mins.

2023-07-27 17:45:38.870272750  [INFO] Starting Frigate...
2023-07-27 17:45:39.577489775  [2023-07-27 17:45:39] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-27 17:45:39.587169227  [2023-07-27 17:45:39] peewee_migrate                 INFO    : Starting migrations
2023-07-27 17:45:39.658487787  [2023-07-27 17:45:39] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-27 17:45:39.681029022  [2023-07-27 17:45:39] detector.tensorrt              INFO    : Starting detection process: 864
2023-07-27 17:45:39.681904459  [2023-07-27 17:45:39] frigate.app                    INFO    : Output process started: 866
2023-07-27 17:45:39.687633131  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_street_front: 869
2023-07-27 17:45:39.687687773  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 870
2023-07-27 17:45:39.693073311  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_street_se: 873
2023-07-27 17:45:39.693119466  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_backyard: 874
2023-07-27 17:45:39.696113238  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-07-27 17:45:39.698927529  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-07-27 17:45:39.700931817  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_street_se: 882
2023-07-27 17:45:39.703259426  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_backyard: 887
2023-07-27 17:45:39.927048454  [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 403 (MiB)
2023-07-27 17:45:39.937806476  [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-07-27 17:45:39.938086135  [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 17:45:40.087250178  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 513 (MiB)
2023-07-27 17:45:40.165109271  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 559 (MiB)
2023-07-27 17:45:40.166873912  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 17:45:40.169431761  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 551 (MiB)
2023-07-27 17:45:40.169435428  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 559 (MiB)
2023-07-27 17:45:40.169448689  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 17:45:58.288109604  [2023-07-27 17:45:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.41.mp4 in 3.342507839202881 seconds.
2023-07-27 17:46:01.760975016  [2023-07-27 17:46:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.41.mp4 in 3.342647075653076 seconds.
2023-07-27 17:46:05.129348800  [2023-07-27 17:46:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.41.mp4 in 3.2932169437408447 seconds.
2023-07-27 17:46:08.550982720  [2023-07-27 17:46:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.41.mp4 in 3.347458839416504 seconds.
2023-07-27 17:46:11.924035339  [2023-07-27 17:46:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.50.mp4 in 3.2893340587615967 seconds.
2023-07-27 17:46:15.317924284  [2023-07-27 17:46:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.50.mp4 in 3.279114007949829 seconds.
2023-07-27 17:46:18.629513704  [2023-07-27 17:46:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.50.mp4 in 3.2351670265197754 seconds.
2023-07-27 17:46:21.979261148  [2023-07-27 17:46:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.50.mp4 in 3.268557071685791 seconds.
2023-07-27 17:46:25.345868751  [2023-07-27 17:46:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.00.mp4 in 3.2627789974212646 seconds.
2023-07-27 17:46:28.706942690  [2023-07-27 17:46:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.10.mp4 in 3.2569751739501953 seconds.
2023-07-27 17:46:32.092718168  [2023-07-27 17:46:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.00.mp4 in 3.310637950897217 seconds.
2023-07-27 17:46:35.474481726  [2023-07-27 17:46:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.10.mp4 in 3.2653961181640625 seconds.
2023-07-27 17:46:38.891189180  [2023-07-27 17:46:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.00.mp4 in 3.295841932296753 seconds.
2023-07-27 17:46:39.860844442  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start expire recordings (new).
2023-07-27 17:46:39.861027446  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start deleted cameras.
2023-07-27 17:46:39.864309059  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Expiring 0 recordings
2023-07-27 17:46:39.865500156  [2023-07-27 17:46:39] frigate.record                 DEBUG   : End deleted cameras.
2023-07-27 17:46:39.865556063  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start all cameras.
2023-07-27 17:46:39.865609077  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start camera: frigate_street_front.
2023-07-27 17:46:42.292586308  [2023-07-27 17:46:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.10.mp4 in 3.279989004135132 seconds.
2023-07-27 17:46:45.706613798  [2023-07-27 17:46:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.00.mp4 in 3.264413833618164 seconds.
2023-07-27 17:46:51.528580781  [2023-07-27 17:46:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.10.mp4 in 5.636636018753052 seconds.
2023-07-27 17:46:56.901660140  [2023-07-27 17:46:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.20.mp4 in 5.226005792617798 seconds.
2023-07-27 17:47:02.976255345  [2023-07-27 17:47:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.29.mp4 in 5.962038993835449 seconds.
2023-07-27 17:47:08.963150298  [2023-07-27 17:47:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.39.mp4 in 5.863729953765869 seconds.
2023-07-27 17:47:15.009231881  [2023-07-27 17:47:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.20.mp4 in 5.9348108768463135 seconds.
2023-07-27 17:47:20.818260089  [2023-07-27 17:47:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.30.mp4 in 5.67697286605835 seconds.
Majestic7979 commented 1 year ago

Annnd it happened again:

2023-07-27 17:45:38.870272750  [INFO] Starting Frigate...
2023-07-27 17:45:39.577489775  [2023-07-27 17:45:39] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-27 17:45:39.587169227  [2023-07-27 17:45:39] peewee_migrate                 INFO    : Starting migrations
2023-07-27 17:45:39.658487787  [2023-07-27 17:45:39] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-27 17:45:39.681029022  [2023-07-27 17:45:39] detector.tensorrt              INFO    : Starting detection process: 864
2023-07-27 17:45:39.681904459  [2023-07-27 17:45:39] frigate.app                    INFO    : Output process started: 866
2023-07-27 17:45:39.687633131  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_street_front: 869
2023-07-27 17:45:39.687687773  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 870
2023-07-27 17:45:39.693073311  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_street_se: 873
2023-07-27 17:45:39.693119466  [2023-07-27 17:45:39] frigate.app                    INFO    : Camera processor started for frigate_backyard: 874
2023-07-27 17:45:39.696113238  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-07-27 17:45:39.698927529  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-07-27 17:45:39.700931817  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_street_se: 882
2023-07-27 17:45:39.703259426  [2023-07-27 17:45:39] frigate.app                    INFO    : Capture process started for frigate_backyard: 887
2023-07-27 17:45:39.927048454  [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 403 (MiB)
2023-07-27 17:45:39.937806476  [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-07-27 17:45:39.938086135  [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 17:45:40.087250178  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 513 (MiB)
2023-07-27 17:45:40.165109271  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 559 (MiB)
2023-07-27 17:45:40.166873912  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 17:45:40.169431761  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 551 (MiB)
2023-07-27 17:45:40.169435428  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 559 (MiB)
2023-07-27 17:45:40.169448689  [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 17:45:58.288109604  [2023-07-27 17:45:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.41.mp4 in 3.342507839202881 seconds.
2023-07-27 17:46:01.760975016  [2023-07-27 17:46:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.41.mp4 in 3.342647075653076 seconds.
2023-07-27 17:46:05.129348800  [2023-07-27 17:46:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.41.mp4 in 3.2932169437408447 seconds.
2023-07-27 17:46:08.550982720  [2023-07-27 17:46:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.41.mp4 in 3.347458839416504 seconds.
2023-07-27 17:46:11.924035339  [2023-07-27 17:46:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.50.mp4 in 3.2893340587615967 seconds.
2023-07-27 17:46:15.317924284  [2023-07-27 17:46:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.50.mp4 in 3.279114007949829 seconds.
2023-07-27 17:46:18.629513704  [2023-07-27 17:46:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.50.mp4 in 3.2351670265197754 seconds.
2023-07-27 17:46:21.979261148  [2023-07-27 17:46:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.50.mp4 in 3.268557071685791 seconds.
2023-07-27 17:46:25.345868751  [2023-07-27 17:46:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.00.mp4 in 3.2627789974212646 seconds.
2023-07-27 17:46:28.706942690  [2023-07-27 17:46:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.10.mp4 in 3.2569751739501953 seconds.
2023-07-27 17:46:32.092718168  [2023-07-27 17:46:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.00.mp4 in 3.310637950897217 seconds.
2023-07-27 17:46:35.474481726  [2023-07-27 17:46:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.10.mp4 in 3.2653961181640625 seconds.
2023-07-27 17:46:38.891189180  [2023-07-27 17:46:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.00.mp4 in 3.295841932296753 seconds.
2023-07-27 17:46:39.860844442  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start expire recordings (new).
2023-07-27 17:46:39.861027446  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start deleted cameras.
2023-07-27 17:46:39.864309059  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Expiring 0 recordings
2023-07-27 17:46:39.865500156  [2023-07-27 17:46:39] frigate.record                 DEBUG   : End deleted cameras.
2023-07-27 17:46:39.865556063  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start all cameras.
2023-07-27 17:46:39.865609077  [2023-07-27 17:46:39] frigate.record                 DEBUG   : Start camera: frigate_street_front.
2023-07-27 17:46:42.292586308  [2023-07-27 17:46:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.10.mp4 in 3.279989004135132 seconds.
2023-07-27 17:46:45.706613798  [2023-07-27 17:46:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.00.mp4 in 3.264413833618164 seconds.
2023-07-27 17:46:51.528580781  [2023-07-27 17:46:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.10.mp4 in 5.636636018753052 seconds.
2023-07-27 17:46:56.901660140  [2023-07-27 17:46:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.20.mp4 in 5.226005792617798 seconds.
2023-07-27 17:47:02.976255345  [2023-07-27 17:47:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.29.mp4 in 5.962038993835449 seconds.
2023-07-27 17:47:08.963150298  [2023-07-27 17:47:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.39.mp4 in 5.863729953765869 seconds.
2023-07-27 17:47:15.009231881  [2023-07-27 17:47:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.20.mp4 in 5.9348108768463135 seconds.
2023-07-27 17:47:20.818260089  [2023-07-27 17:47:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.30.mp4 in 5.67697286605835 seconds.
2023-07-27 17:47:26.696242611  [2023-07-27 17:47:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.40.mp4 in 5.780220031738281 seconds.
2023-07-27 17:47:30.947251243  [2023-07-27 17:47:30] frigate.record                 DEBUG   : Expiring 254 recordings
2023-07-27 17:47:31.160119645  [2023-07-27 17:47:31] frigate.record                 DEBUG   : End camera: frigate_street_front.
2023-07-27 17:47:31.160155044  [2023-07-27 17:47:31] frigate.record                 DEBUG   : Start camera: frigate_street_nw.
2023-07-27 17:47:31.374643984  [2023-07-27 17:47:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.20.mp4 in 4.5776848793029785 seconds.
2023-07-27 17:47:35.087833908  [2023-07-27 17:47:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.30.mp4 in 3.580332040786743 seconds.
2023-07-27 17:47:40.649230966  [2023-07-27 17:47:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.40.mp4 in 5.437561988830566 seconds.
2023-07-27 17:47:46.234215616  [2023-07-27 17:47:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.20.mp4 in 5.468185186386108 seconds.
2023-07-27 17:47:51.996834563  [2023-07-27 17:47:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.30.mp4 in 5.669494152069092 seconds.
2023-07-27 17:47:57.703996357  [2023-07-27 17:47:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.40.mp4 in 5.624220132827759 seconds.
2023-07-27 17:47:57.740316252  [2023-07-27 17:47:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:47:57.741450451  [2023-07-27 17:47:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:47:57.741472674  [2023-07-27 17:47:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:47:57.741709245  [2023-07-27 17:47:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:48:03.737077274  [2023-07-27 17:48:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.00.mp4 in 5.928970098495483 seconds.
2023-07-27 17:48:04.878764998  [2023-07-27 17:48:04] frigate.record                 DEBUG   : Expiring 175 recordings
2023-07-27 17:48:04.988821564  [2023-07-27 17:48:04] frigate.record                 DEBUG   : End camera: frigate_street_nw.
2023-07-27 17:48:04.989021816  [2023-07-27 17:48:04] frigate.record                 DEBUG   : Start camera: frigate_street_se.
2023-07-27 17:48:07.479430642  [2023-07-27 17:48:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.10.mp4 in 3.604843854904175 seconds.
2023-07-27 17:48:13.026042592  [2023-07-27 17:48:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.20.mp4 in 5.439263105392456 seconds.
2023-07-27 17:48:19.169508223  [2023-07-27 17:48:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.30.mp4 in 6.029353141784668 seconds.
2023-07-27 17:48:25.108971065  [2023-07-27 17:48:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.40.mp4 in 5.83636999130249 seconds.
2023-07-27 17:48:31.183972145  [2023-07-27 17:48:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.00.mp4 in 5.951390027999878 seconds.
2023-07-27 17:48:37.581601426  [2023-07-27 17:48:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.10.mp4 in 6.262295961380005 seconds.
2023-07-27 17:48:43.592764277  [2023-07-27 17:48:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.20.mp4 in 5.860684156417847 seconds.
2023-07-27 17:48:49.617885615  [2023-07-27 17:48:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.30.mp4 in 5.906958103179932 seconds.
Majestic7979 commented 1 year ago

By the way, Immich finished transcoding my videos so now the CPU is idling at 20% tops with CPU temps mid 50C on average..It is not system load.



server.local system information |SYSTEM INFORMATION     | | |---|---| | System hostname | server.local (127.0.1.1) | | Operating system | Debian Linux 12 | | Webmin version | 2.100 | | Authentic theme version | 21.01-2 | | Time on system | Thursday, July 27, 2023 5:53 PM | | Kernel and CPU | Linux 6.1.0-10-amd64 on x86_64 | | Processor information | 13th Gen Intel(R) Core(TM) i5-13500, 20 cores | | CPU temperatures | Core 1: 55°CCore 5: 55°CCore 9: 56°CCore 13: 56°CCore 17: 68°CCore 21: 52°CCore 25: 55°CCore 26: 55°CCore 27: 55°CCore 28: 55°CCore 29: 53°CCore 30: 53°CCore 31: 53°CCore 32: 53°C | | CPU fans | Fan 1: 0 RPMFan 2: 1679 RPMFan 6: 0 RPMFan 7: 0 RPM | | Drive temperatures | nvme0n1: 54°Csda: 41°Csdb: 40°Csdc: 37°C | | System uptime | 16 hours, 57 minutes | | Running processes | 1014 | | CPU load averages | 7.41 (1 min) 5.64 (5 mins) 3.68 (15 mins) | | Real memory | 8.89 GiB used / 21.93 GiB cached / 31.08 GiB total | | Virtual memory | 5.6 GiB used / 63.99 GiB total | | Local disk space | 6.53 TiB used / 11.55 TiB free / 18.09 TiB total | | Package updates | All installed packages are up to date |


Majestic7979 commented 1 year ago

This is currently how my CPU is being utilized: image

NickM-27 commented 1 year ago

Yeah, these copy times are very high:

2023-07-27 17:48:31.183972145  [2023-07-27 17:48:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.00.mp4 in 5.951390027999878 seconds.
2023-07-27 17:48:37.581601426  [2023-07-27 17:48:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.10.mp4 in 6.262295961380005 seconds.
2023-07-27 17:48:43.592764277  [2023-07-27 17:48:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.20.mp4 in 5.860684156417847 seconds.
2023-07-27 17:48:49.617885615  [2023-07-27 17:48:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.30.mp4 in 5.906958103179932 seconds.

There's nothing frigate can do if the I/O is unable to keep up. For context, the users who were CPU bound had copy times of ~0.1 seconds per segment with segments at an average of 5MB. Personally, my highest bandwidth camera that is CBR 8192 kbps and has segments 10MB has a move time of ~0.5 seconds per segment.

Majestic7979 commented 1 year ago

But I don't understand, you're saying it's the IO, however I provided all sorts of data and even a disk benchmark... surely there's a way to debug deeper to understand what is going on?

Memory usage: image

Majestic7979 commented 1 year ago

Benchmark with docker service stopped and zero containers running. I will shortly do the same with all containers running to compare.

user@server:~$ sudo systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
user@server:~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=TGS --filename=TGS --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
TGS: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
TGS: Laying out IO file (1 file / 4096MiB)
^Cbs: 1 (f=1): [m(1)][14.9%][r=4092KiB/s,w=1268KiB/s][r=1023,w=317 IOPS][eta 09m:35s]
fio: terminating on signal 2
Jobs: 1 (f=1): [m(1)][15.1%][r=4480KiB/s,w=1533KiB/s][r=1120,w=383 IOPS][eta 09m:34s]
TGS: (groupid=0, jobs=1): err= 0: pid=561308: Thu Jul 27 18:11:42 2023
  read: IOPS=1173, BW=4693KiB/s (4806kB/s)(463MiB/101086msec)
   bw (  KiB/s): min=  816, max= 5368, per=100.00%, avg=4695.14, stdev=689.26, samples=202
   iops        : min=  204, max= 1342, avg=1173.78, stdev=172.32, samples=202
  write: IOPS=390, BW=1563KiB/s (1601kB/s)(154MiB/101086msec); 0 zone resets
   bw (  KiB/s): min=  320, max= 1984, per=100.00%, avg=1564.15, stdev=256.99, samples=202
   iops        : min=   80, max=  496, avg=391.02, stdev=64.24, samples=202
  cpu          : usr=0.57%, sys=5.14%, ctx=126373, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=118602,39501,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=4693KiB/s (4806kB/s), 4693KiB/s-4693KiB/s (4806kB/s-4806kB/s), io=463MiB (486MB), run=101086-101086msec
  WRITE: bw=1563KiB/s (1601kB/s), 1563KiB/s-1563KiB/s (1601kB/s-1601kB/s), io=154MiB (162MB), run=101086-101086msec
NickM-27 commented 1 year ago

I can run a benchmark on my Samsung 980 Pro and get a write speed of 5 GB/s, that does not mean that a 10MB segment will be written in 0.002 seconds.

Benchmark with docker service stopped and zero containers running. I will shortly do the same with all containers running to compare.

user@server:~$ sudo systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
user@server:~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=TGS --filename=TGS --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
TGS: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
TGS: Laying out IO file (1 file / 4096MiB)
^Cbs: 1 (f=1): [m(1)][14.9%][r=4092KiB/s,w=1268KiB/s][r=1023,w=317 IOPS][eta 09m:35s]
fio: terminating on signal 2
Jobs: 1 (f=1): [m(1)][15.1%][r=4480KiB/s,w=1533KiB/s][r=1120,w=383 IOPS][eta 09m:34s]
TGS: (groupid=0, jobs=1): err= 0: pid=561308: Thu Jul 27 18:11:42 2023
  read: IOPS=1173, BW=4693KiB/s (4806kB/s)(463MiB/101086msec)
   bw (  KiB/s): min=  816, max= 5368, per=100.00%, avg=4695.14, stdev=689.26, samples=202
   iops        : min=  204, max= 1342, avg=1173.78, stdev=172.32, samples=202
  write: IOPS=390, BW=1563KiB/s (1601kB/s)(154MiB/101086msec); 0 zone resets
   bw (  KiB/s): min=  320, max= 1984, per=100.00%, avg=1564.15, stdev=256.99, samples=202
   iops        : min=   80, max=  496, avg=391.02, stdev=64.24, samples=202
  cpu          : usr=0.57%, sys=5.14%, ctx=126373, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=118602,39501,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=4693KiB/s (4806kB/s), 4693KiB/s-4693KiB/s (4806kB/s-4806kB/s), io=463MiB (486MB), run=101086-101086msec
  WRITE: bw=1563KiB/s (1601kB/s), 1563KiB/s-1563KiB/s (1601kB/s-1601kB/s), io=154MiB (162MB), run=101086-101086msec

You are getting a write speed of 1563KiB/s which if you compare that to your 8MB segment size is 5.3 seconds, which again shows this to be the problem.

Majestic7979 commented 1 year ago

Please wait, I will finish the benchmarks and once I post all the results we can continue to discuss. I'm not done yet. I am not trying to prove anything. If the problem is my system, the problem is my system, I will accept that and ask for suggestions. I have not done the comparison and another script bench. Bear with.

Majestic7979 commented 1 year ago

This is with all docker containers running:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=TGS --filename=TGS --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
TGS: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
^Cbs: 1 (f=1): [m(1)][18.0%][r=1933KiB/s,w=760KiB/s][r=483,w=190 IOPS][eta 18m:38s]
fio: terminating on signal 2

TGS: (groupid=0, jobs=1): err= 0: pid=575208: Thu Jul 27 18:17:53 2023
  read: IOPS=576, BW=2305KiB/s (2360kB/s)(555MiB/246583msec)
   bw (  KiB/s): min=   16, max= 5120, per=100.00%, avg=2312.20, stdev=801.47, samples=491
   iops        : min=    4, max= 1280, avg=578.05, stdev=200.37, samples=491
  write: IOPS=191, BW=767KiB/s (785kB/s)(185MiB/246583msec); 0 zone resets
   bw (  KiB/s): min=   56, max= 1840, per=100.00%, avg=767.74, stdev=294.21, samples=492
   iops        : min=   14, max=  460, avg=191.93, stdev=73.55, samples=492
  cpu          : usr=0.33%, sys=2.82%, ctx=139743, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=142064,47276,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=2305KiB/s (2360kB/s), 2305KiB/s-2305KiB/s (2360kB/s-2360kB/s), io=555MiB (582MB), run=246583-246583msec
  WRITE: bw=767KiB/s (785kB/s), 767KiB/s-767KiB/s (785kB/s-785kB/s), io=185MiB (194MB), run=246583-246583msec
Majestic7979 commented 1 year ago

YABS on the /home volume (16TB RAID) where the /db is stored:

curl -sL yabs.sh | bash
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
#              Yet-Another-Bench-Script              #
#                     v2023-04-23                    #
# https://github.com/masonr/yet-another-bench-script #
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #

Thu 27 Jul 18:18:28 BST 2023

Basic System Information:
---------------------------------
Uptime     : 0 days, 17 hours, 22 minutes
Processor  : 13th Gen Intel(R) Core(TM) i5-13500
CPU cores  : 20 @ 1430.338 MHz
AES-NI     : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM        : 31.1 GiB
Swap       : 64.0 GiB
Disk       : 18.1 TiB
Distro     : Debian GNU/Linux 12 (bookworm)
Kernel     : 6.1.0-10-amd64
VM Type    : NONE
IPv4/IPv6  : ✔ Online / ❌ Offline

IPv4 Network Information:
---------------------------------
ISP        : Virgin Media
ASN        : AS5089 Virgin Media Limited
Location   : City, England (ENG)
Country    : United Kingdom

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 2.67 MB/s      (669) | 19.02 MB/s     (297)
Write      | 2.68 MB/s      (670) | 19.64 MB/s     (306)
Total      | 5.35 MB/s     (1.3k) | 38.67 MB/s     (603)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 18.72 MB/s      (36) | 23.42 MB/s      (22)
Write      | 20.20 MB/s      (39) | 26.42 MB/s      (25)
Total      | 38.92 MB/s      (75) | 49.84 MB/s      (47)

iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed      | Ping
-----           | -----                     | ----            | ----            | ----
Clouvider       | London, UK (10G)          | 35.1 Mbits/sec  | 729 Mbits/sec   | 21.2 ms
Scaleway        | Paris, FR (10G)           | 35.8 Mbits/sec  | 718 Mbits/sec   | 29.8 ms
NovoServe       | North Holland, NL (40G)   | 36.2 Mbits/sec  | 686 Mbits/sec   | 19.9 ms
Uztelecom       | Tashkent, UZ (10G)        | 21.8 Mbits/sec  | 612 Mbits/sec   | 120 ms
Clouvider       | NYC, NY, US (10G)         | 20.9 Mbits/sec  | 640 Mbits/sec   | 110 ms
Clouvider       | Dallas, TX, US (10G)      | 21.7 Mbits/sec  | 625 Mbits/sec   | 152 ms
Clouvider       | Los Angeles, CA, US (10G) | 19.5 Mbits/sec  | 513 Mbits/sec   | 186 ms

Geekbench 6 Benchmark Test:
---------------------------------
Test            | Value
                |
Single Core     | 2513
Multi Core      | 13865
Full Test       | https://browser.geekbench.com/v6/cpu/2058641

YABS completed in 10 min 49 sec

Next will be YABS in the /cctv where the recordings are stored.

Majestic7979 commented 1 year ago

I am struggling to get the volume tested. I think it's being used so the system is not releasing it. Any idea? I am really tired and my head isn't thinking very well because of that, could use with some suggestions. I really want to benchmark this drive and see if it is the drive. Not sure how to release the mount so it can run.

Majestic7979 commented 1 year ago

I'm now running sudo e2fsck -cfpv /dev/sdc1 - sdc1 being the /cctv drive. I managed to unmount using sudo fuser -km /dev/sdc1 and sudo fuser -km /cctv. I don't want to waste your time, I want to make sure the drive isn't the problem. Because the server definitely isn't... the drive has been running for 9 years mostly 24/7... WD drives are freaking impressive. SMART says it's fine, but who knows. It's checking for bad blocks now.

NickM-27 commented 1 year ago

I'm sure you know, but worth saying that a drive can pass SMART, have no bad sectors, but still be slow, and of course speed can vary by the workload. The first benchmark shows a write speed that matches the copy segment times.

I don't know what it would be if not the drive, or something related to the drive from the host or the mount.

Majestic7979 commented 1 year ago

The command is still running... it's a large drive. But I opened iostat and it seems like it's reading at a healthy 135MBps image

When you say "... but still be slow...", well, if it's not a hardware issue then it is fixable. I really want to find out. Because if I can't, it means I just can't use your solution as my NVR and that would be a huge pity for me and anyone with the same type of issue. Maybe people don't have this problem now because they potentially run 1 or 2 cams... what happens when they scale, their hardware is totally fine but there is a bug or whatever? I mean, it's your time and effort, you decide, but I hope you will take the route of walking me through and assisting. It will increase both our knowledge and ability to resolve this type of issue in the future as well as help you diagnose similar problems from others. As soon as the badblock scan is done, I will see about benchmarking properly. I was running docker - I thought I stopped it but I forgot all my containers have "restart=always" so they were getting spun again and affecting the benchmarks. We are currently looking at what the disk itself is doing. Let's rule out a hardware problem and hopefully go from there.