Closed Majestic7979 closed 1 year ago
This is the camera details - my camera is basically the same one, but mine has 4x optical zoom. They don't sell it anymore, but it's pointless anyway since I never use the zoom... https://www.anpvizsecurity.com/products-detail-149850
2023-07-27 05:38:12.248355587 [INFO] Preparing go2rtc config... 2023-07-27 05:38:12.840348777 [INFO] Starting go2rtc... 2023-07-27 05:38:12.927546476 05:38:12.927 INF go2rtc version 1.2.0 linux/amd64 2023-07-27 05:38:12.927863464 05:38:12.927 INF [api] listen addr=:1984 2023-07-27 05:38:12.928059559 05:38:12.928 INF [rtsp] listen addr=:8554 2023-07-27 05:38:12.928251082 05:38:12.928 INF [srtp] listen addr=:8443 2023-07-27 05:38:12.928434646 05:38:12.928 INF [webrtc] listen addr=:8555 2023-07-27 05:38:22.250947703 [INFO] Starting go2rtc healthcheck service...
2023-07-27 05:38:12.252644235 [INFO] Starting NGINX...
2023-07-27 05:38:12.787907773 2023/07/27 05:38:12 [error] 133#133: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.211, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001/stats", host: "10.0.0.100:5000"
2023-07-27 05:38:12.787913838 10.0.0.211 - - [27/Jul/2023:05:38:12 +0100] "GET /api/stats HTTP/1.1" 502 157 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.431646851 10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.436619483 10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.438426169 10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.449435910 10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.231424595 10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3213 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.297277909 10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3662 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.308432389 10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.329202143 10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3741 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:21.036429727 10.0.0.211 - - [27/Jul/2023:05:38:21 +0100] "GET /api/stats HTTP/1.1" 200 3557 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.889390774 10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8561 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.890571543 10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7279 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.891752588 10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9158 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.915748783 10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8396 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:29.059318788 10.0.0.211 - - [27/Jul/2023:05:38:29 +0100] "GET /api/stats HTTP/1.1" 200 3550 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.291292849 10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3223 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.307715284 10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3544 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.311573641 10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3740 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.323592817 10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3664 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.807765027 10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.831799781 10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9138 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.837743184 10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7293 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.857281786 10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8424 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:37.043264858 10.0.0.211 - - [27/Jul/2023:05:38:37 +0100] "GET /api/stats HTTP/1.1" 200 3593 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.206203918 10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3739 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.222862769 10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3556 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.263576095 10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.264607948 10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3660 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.043740305 10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/stats HTTP/1.1" 200 3627 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.867897063 10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7303 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.879043706 10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8557 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.901132245 10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8455 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.903781198 10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9163 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.201393673 10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3558 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.212774584 10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3745 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.230717375 10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3649 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.230722470 10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3238 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:53.119573612 10.0.0.211 - - [27/Jul/2023:05:38:53 +0100] "GET /api/stats HTTP/1.1" 200 3647 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.852768669 10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.874669846 10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7308 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.889875630 10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9188 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.895579911 10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8459 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.299693496 10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3235 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.307231210 10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3534 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.340042518 10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3652 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.340241709 10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3755 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:01.051734661 10.0.0.211 - - [27/Jul/2023:05:39:01 +0100] "GET /api/stats HTTP/1.1" 200 3615 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.856406955 10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8570 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.916682313 10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7315 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.935457308 10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9180 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.939538823 10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8422 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:09.079743866 10.0.0.211 - - [27/Jul/2023:05:39:09 +0100] "GET /api/stats HTTP/1.1" 200 3583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.249082932 10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.256962140 10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3542 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.261053412 10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3652 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.265379335 10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3758 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.831576485 10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8581 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.841112385 10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9177 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.861353707 10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8416 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.862452710 10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7302 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:17.027732752 10.0.0.211 - - [27/Jul/2023:05:39:17 +0100] "GET /api/stats HTTP/1.1" 200 3584 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.199703053 10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3250 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.223576289 10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3540 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.243577429 10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3755 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.259714764 10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3665 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.056271979 10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/stats HTTP/1.1" 200 3664 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.851576288 10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9233 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.855568354 10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7314 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.859734345 10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8529 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.876347884 10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8449 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.302785553 10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.310445539 10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.335745205 10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3768 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.365298962 10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3651 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:33.064522539 10.0.0.211 - - [27/Jul/2023:05:39:33 +0100] "GET /api/stats HTTP/1.1" 200 3598 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.834617599 10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8540 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.857656229 10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7300 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.862026021 10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8440 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.883778149 10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9237 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.210843173 10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3244 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.218080399 10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3563 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.246018290 10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3782 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.267585193 10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3661 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:41.033828433 10.0.0.211 - - [27/Jul/2023:05:39:41 +0100] "GET /api/stats HTTP/1.1" 200 3585 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.875895779 10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8591 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.892384057 10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7415 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.900027351 10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9247 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.906613740 10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8462 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:49.047173785 10.0.0.211 - - [27/Jul/2023:05:39:49 +0100] "GET /api/stats HTTP/1.1" 200 3628 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.237548382 10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3535 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.251151712 10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3667 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.254006903 10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3772 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.263504692 10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3183 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.881816093 10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7229 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.883126887 10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8532 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.891740992 10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8369 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.896643805 10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9247 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:57.060453696 10.0.0.211 - - [27/Jul/2023:05:39:57 +0100] "GET /api/stats HTTP/1.1" 200 3617 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.191185760 10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3193 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.195884250 10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3530 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.201606413 10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3784 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.228000964 10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3648 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:05.056632599 10.0.0.211 - - [27/Jul/2023:05:40:05 +0100] "GET /api/stats HTTP/1.1" 200 3598 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.196302668 10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3208 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.219712275 10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3531 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.222178318 10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3774 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.238280609 10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3637 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:13.044757436 10.0.0.211 - - [27/Jul/2023:05:40:13 +0100] "GET /api/stats HTTP/1.1" 200 3583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.838428741 10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8548 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.887592170 10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7232 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.899749130 10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8387 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.907584023 10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9227 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:17.832151443 10.0.0.211 - - [27/Jul/2023:05:40:17 +0100] "GET /config HTTP/1.1" 200 6641 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:17.879712565 10.0.0.211 - - [27/Jul/2023:05:40:17 +0100] "GET /api/config HTTP/1.1" 200 12623 "https://mydomain.example/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:18.163871560 10.0.0.211 - - [27/Jul/2023:05:40:18 +0100] "GET /api/config/raw HTTP/1.1" 200 1002 "https://mydomain.example/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:18.431577478 10.0.0.211 - - [27/Jul/2023:05:40:18 +0100] "GET //api/config/schema.json HTTP/1.1" 200 25729 "https://mydomain.example/assets/yaml.worker.bundle.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:20.242401440 10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3195 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.262206588 10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3788 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.262212030 10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3522 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.278525963 10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3653 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:21.043582568 10.0.0.211 - - [27/Jul/2023:05:40:21 +0100] "GET /api/stats HTTP/1.1" 200 3650 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:22.710410903 10.0.0.211 - - [27/Jul/2023:05:40:22 +0100] "GET /api/stats HTTP/1.1" 200 3648 "https://mydomain.example/system" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:25.879279294 10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8542 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.899580698 10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7242 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.909508582 10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9258 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.929518059 10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8381 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:26.761249705 10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/nginx HTTP/1.1" 200 2532 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:26.777079919 10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/go2rtc HTTP/1.1" 200 258 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:26.800516415 10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/frigate HTTP/1.1" 200 821 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:28.293681829 10.0.0.211 - - [27/Jul/2023:05:40:28 +0100] "GET /api/stats HTTP/1.1" 200 3683 "https://mydomain.example/storage" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:28.509471747 10.0.0.211 - - [27/Jul/2023:05:40:28 +0100] "GET /api/recordings/storage HTTP/1.1" 200 390 "https://mydomain.example/storage" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:29.052124839 10.0.0.211 - - [27/Jul/2023:05:40:29 +0100] "GET /api/stats HTTP/1.1" 200 3683 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.178105819 10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3218 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.238486528 10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3541 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.247574161 10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3781 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.299575305 10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3650 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:35.887575575 10.0.0.211 - - [27/Jul/2023:05:40:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8571 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:35.923575548 10.0.0.211 - - [27/Jul/2023:05:40:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7263 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:36.799577193 10.0.0.211 - - [27/Jul/2023:05:40:36 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:36.847569285 10.0.0.211 - - [27/Jul/2023:05:40:36 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8423 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:37.043577178 10.0.0.211 - - [27/Jul/2023:05:40:37 +0100] "GET /api/stats HTTP/1.1" 200 3618 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:39.968607191 10.0.0.211 - - [27/Jul/2023:05:40:39 +0100] "GET /api/config HTTP/1.1" 200 12623 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:39.970618926 10.0.0.211 - - [27/Jul/2023:05:40:39 +0100] "GET /api/logs/go2rtc HTTP/1.1" 200 258 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
Host:
I want you to ignore the CPU temps. It's high because ffmpeg is running on another process (immich) via docker to transcode some videos. The issue I have with Frigate not keeping up with frames happens even when the CPU is not stressed. Regardless, if you will insist that this is the issue let me know how to force the frigate container to run as a priority because my security system must always be prioritized above all else.
Is the disk where recordings are stored on the local machine or are you storing them on a network share?
You can turn on debug logging to see the copy times for moving recordings:
logger:
logs:
frigate.record: debug
If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up.
I do not know of a way to specify priority in HassOS for an addon.
Alternatively, you can use motion
mode instead of all
mode for record so fewer segments are copied over. This has helped several users.
Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps.
Is the disk where recordings are stored on the local machine or are you storing them on a network share?
You can turn on debug logging to see the copy times for moving recordings:
logger: logs: frigate.record: debug
If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up.
I do not know of a way to specify priority in HassOS for an addon.
Alternatively, you can use
motion
mode instead ofall
mode for record so fewer segments are copied over. This has helped several users.Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps.
It's a SATA disk in the host device.
I get this issue even when the CPU is in the low percent of usage. I knew this was going to come up... It may be that, I'm not going to say "it's not that", without data neither of us don't know, so it might be. But it happens even when there's no high CPU utilization. Nick told me to increase shm, he said that was the issue. But it didn't fix, you can see my shm is quite high. As per above form answer I don't use HassOS, I run directly on Docker on a Debian 12 system.
I cannot use motion for recording, It's simply not reliable enough and the police said they need an hour before and an hour after any incident and if there is any skipping they cannot accept the evidence.
The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res.
Is the issue related to motion detection? Or is the issue related to recording?
I can try motion masks but the whole point of this project is smart detection isn't it... I mean nobody can say my system is slow, it's better than most people's main personal computers!!
I used to run on a 8GB DDR3 Intel Core i7 3770T processor without GPU (apart from built-in) Now I run on a 32GB DDR5 Intel Core i5 13500 which is latest gen with many more cores and a relatively beefy GTX1080 GPU and it can't keep up.... Something is not right. This is what I said in my "opinion" - Frigate is focusing a lot on delivering features and skipping on stability, meaning the footage is unusable when it's most required to prosecute a criminal. Honestly, please rethink the approach here. I've seen go2rtc get added with added complexity and added bugs... meanwhile we're seeing a system of this level struggle!? It's not a criticism it's just a fact that's happening right now. Why not focus on multiprocessing/GPU processing? Apparently even with a GPU the CPU is still used quite a lot for some tasks like motion. That should not be happening if there is such a beefy GPU in the host.
But that was digressing. I just need support for the issue right now.
In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)?
Oh and I know I shouldn't expect anyone to assume but it goes without saying I appreciate your time and effort in building this application and supporting others and myself. Big thank you. Never think I am criticizing you.
The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res.
This is no longer true, if you use ffmpeg presets for hwaccel (like you are) then the scaling will be done on the GPU instead so it is not a problem.
Frigate is focusing a lot on delivering features and skipping on stability, meaning the footage is unusable when it's most required to prosecute a criminal. Honestly, please rethink the approach here. I've seen go2rtc get added with added complexity and added bugs... meanwhile we're seeing a system of this level struggle!? It's not a criticism it's just a fact that's happening right now. Why not focus on multiprocessing/GPU processing?
That is objectively false. You can look at the commits for the upcoming 0.13 release, there have been many multiprocessing improvements for recording management and otherwise. But for now the recommendations are made for the current version, which is mainly to reduce the amount of segments that are kept so the recordings maintainer has less work to do.
In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)?
The record debug logging will show the amount of time it takes to move the recording segments from cache to the storage drive, which is crucial to understanding where the slow down is.
Also just wanted to confirm, the /db/
volume is not on the Sata SSD correct?
Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you.
Hey Nick.
What resolution do you suggest me to use then?
Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability.
I will enable the debug logging now, thank you.
The /db/ is on the SATA drive because I run my container with a bind mount, and the mount is stored in the /home partition which is in a 14TB mirroring RAID.
This is my docker CLI:
docker create --name frigate --restart=always -p 5000:5000 --gpus=all --cap-add=CAP_PERFMON --mount type=tmpfs,target=/tmp/cache,tmpfs-size=2000000000 --shm-size=5g --device /dev/bus/usb:/dev/bus/usb --device /dev/dri/renderD128 -v /cctv:/media/frigate -v /home/user1/dockerbindmounts/frigate/config.yml:/config/config.yml -v /home/user1/dockerbindmounts/frigate/db/:/db/ -v /etc/localtime:/etc/localtime:ro -v /home/user1/trt-models:/trt-models -e FRIGATE_RTSP_PASSWORD='removedforprivacy' -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,video,utility -e YOLO_MODELS=yolov7-tiny-416 ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt && docker start frigate
Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you.
Right and that's what is making me scratch my head.
For info I run a 3TB WD Red Pro drive for my footage, it's mounted on /cctv
To make absolutely clear I store my footage in a separate SATA drive (3TB WD Red Pro) and my frigate db is in the main RAID array (14TB). The DB and the frigate config/db are in separate storage drives therefore read/write are separate...
What resolution do you suggest me to use then?
Looks like your cameras are 4:3 so you could try
detect:
width: 1280
height: 960
and see if the objects are detected as you would like. Without seeing the camera images it is difficult to suggest fully because smaller objects that are farther away of course require more resolution to be detected reliably. I run all of my cameras with a height of 720 (and the 4:3 cameras with a height of 960) and have not had any issues detection people far away from the camera.
Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability.
I would suggest that there were a number of stability improvements in 0.12 as well, but I understand where you're coming from and of course you wouldn't know about the commits in the unreleased version.
The to-do list for 0.13 is getting closer so a beta should not be too far away.
Another thing that could be tried here given that you require all
recording instead of motion
recording is reducing your cameras bit rate so there is less data for the recordings maintainer to move.
I will enable the debug logging now, thank you.
Great, the values in general will be helpful but it will be mostly helpful to see what the copy times are after the segment cache errors start occurring.
Cam image: [removed for privacy]
Thanks, that's an empathetic response! I really appreciate it!!! I am doing all I can to try to help you help me, and I know you're a stellar person here with your support. It's really nice when someone recognizes comments and takes it on board, understanding that the intentions are always good.
The problem with reducing the bitrate is that it will likely pixellate... this will greatly affect number plate in vehicles especially at night. But the cameras are all feeding directly into a Ubiquiti USW-24-PoE switch which isn't even fully utilized, each camera obviously using their own cable and going into their specific switch port. There is no "in between", it connects directly to my main switch, which the server is also directly connected into.
My cameras are outputting to MediaMTX (rtsp-simple-server) and I input the proxied stream into Frigate. If I input the MediaMTX proxied stream for any of the cams into VLC I get a "perfect" stream. I put perfect into quotes because the audio, for some reason, is slow-mo. You know when you record someone speaking and then you slow-mo and the person sounds like their voice is deeper and slower? This is happening and I don't know why. The video itself is real-time, there is a timestamp as you can see above and it's playing back second to second as expected. I'll now post a new log with debug in a separate comment
2023-07-27 14:51:24.068200519 [INFO] Starting Frigate...
2023-07-27 14:51:25.991688431 [2023-07-27 14:51:25] frigate.app INFO : Starting Frigate (0.12.1-367d724)
2023-07-27 14:51:26.035304264 [2023-07-27 14:51:26] peewee_migrate INFO : Starting migrations
2023-07-27 14:51:26.071227368 [2023-07-27 14:51:26] peewee_migrate INFO : There is nothing to migrate
2023-07-27 14:51:26.101592142 [2023-07-27 14:51:26] frigate.app INFO : Output process started: 865
2023-07-27 14:51:26.103897208 [2023-07-27 14:51:26] detector.tensorrt INFO : Starting detection process: 864
2023-07-27 14:51:26.128042669 [2023-07-27 14:51:26] frigate.app INFO : Camera processor started for frigate_street_front: 869
2023-07-27 14:51:26.128046642 [2023-07-27 14:51:26] frigate.app INFO : Camera processor started for frigate_street_nw: 870
2023-07-27 14:51:26.128049471 [2023-07-27 14:51:26] frigate.app INFO : Camera processor started for frigate_street_se: 871
2023-07-27 14:51:26.128051525 [2023-07-27 14:51:26] frigate.app INFO : Camera processor started for frigate_backyard: 872
2023-07-27 14:51:26.141239340 [2023-07-27 14:51:26] frigate.app INFO : Capture process started for frigate_street_front: 875
2023-07-27 14:51:26.146591593 [2023-07-27 14:51:26] frigate.app INFO : Capture process started for frigate_street_nw: 878
2023-07-27 14:51:26.153775457 [2023-07-27 14:51:26] frigate.app INFO : Capture process started for frigate_street_se: 882
2023-07-27 14:51:26.159047702 [2023-07-27 14:51:26] frigate.app INFO : Capture process started for frigate_backyard: 885
2023-07-27 14:51:26.813366819 [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 248 (MiB)
2023-07-27 14:51:26.978411519 [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt INFO : Loaded engine size: 34 MiB
2023-07-27 14:51:26.979692337 [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 14:51:27.364548466 [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 358 (MiB)
2023-07-27 14:51:27.557194923 [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 404 (MiB)
2023-07-27 14:51:27.559115628 [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 14:51:27.565314022 [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 396 (MiB)
2023-07-27 14:51:27.565484698 [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 404 (MiB)
2023-07-27 14:51:27.565488372 [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 14:51:44.271330251 [2023-07-27 14:51:44] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.27.mp4 in 2.8024659156799316 seconds.
2023-07-27 14:51:47.419609909 [2023-07-27 14:51:47] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.27.mp4 in 2.8160011768341064 seconds.
2023-07-27 14:51:50.261774582 [2023-07-27 14:51:50] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.27.mp4 in 2.7169740200042725 seconds.
2023-07-27 14:51:53.223402149 [2023-07-27 14:51:53] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.27.mp4 in 2.8212969303131104 seconds.
2023-07-27 14:51:56.121896583 [2023-07-27 14:51:56] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.36.mp4 in 2.7229220867156982 seconds.
2023-07-27 14:51:59.070124450 [2023-07-27 14:51:59] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.36.mp4 in 2.8004510402679443 seconds.
2023-07-27 14:52:01.971288144 [2023-07-27 14:52:01] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.36.mp4 in 2.7553539276123047 seconds.
2023-07-27 14:52:05.733478023 [2023-07-27 14:52:05] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.37.mp4 in 3.5945658683776855 seconds.
2023-07-27 14:52:08.670631442 [2023-07-27 14:52:08] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.46.mp4 in 2.7888638973236084 seconds.
2023-07-27 14:52:11.538703695 [2023-07-27 14:52:11] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.46.mp4 in 2.696091890335083 seconds.
2023-07-27 14:52:14.378159551 [2023-07-27 14:52:14] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.46.mp4 in 2.714629888534546 seconds.
2023-07-27 14:52:17.266218893 [2023-07-27 14:52:17] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.47.mp4 in 2.741323947906494 seconds.
2023-07-27 14:52:20.299887659 [2023-07-27 14:52:20] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.56.mp4 in 2.8629918098449707 seconds.
2023-07-27 14:52:23.280067169 [2023-07-27 14:52:23] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.06.mp4 in 2.8107500076293945 seconds.
2023-07-27 14:52:26.167934835 [2023-07-27 14:52:26] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.56.mp4 in 2.734575033187866 seconds.
2023-07-27 14:52:26.336420434 [2023-07-27 14:52:26] frigate.record DEBUG : Start expire recordings (new).
2023-07-27 14:52:26.336423851 [2023-07-27 14:52:26] frigate.record DEBUG : Start deleted cameras.
2023-07-27 14:52:26.919573876 [2023-07-27 14:52:26] frigate.record DEBUG : Expiring 0 recordings
2023-07-27 14:52:26.923573589 [2023-07-27 14:52:26] frigate.record DEBUG : End deleted cameras.
2023-07-27 14:52:26.923578160 [2023-07-27 14:52:26] frigate.record DEBUG : Start all cameras.
2023-07-27 14:52:26.923580102 [2023-07-27 14:52:26] frigate.record DEBUG : Start camera: frigate_street_front.
2023-07-27 14:52:29.056176497 [2023-07-27 14:52:29] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.06.mp4 in 2.7186970710754395 seconds.
2023-07-27 14:52:32.019877456 [2023-07-27 14:52:32] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.56.mp4 in 2.780247926712036 seconds.
2023-07-27 14:52:35.067947513 [2023-07-27 14:52:35] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.06.mp4 in 2.833055019378662 seconds.
2023-07-27 14:52:38.012625476 [2023-07-27 14:52:38] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.57.mp4 in 2.7677149772644043 seconds.
2023-07-27 14:52:40.893284633 [2023-07-27 14:52:40] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.07.mp4 in 2.7566659450531006 seconds.
2023-07-27 14:52:43.747509775 [2023-07-27 14:52:43] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.16.mp4 in 2.7001211643218994 seconds.
2023-07-27 14:52:46.693283952 [2023-07-27 14:52:46] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.27.mp4 in 2.7567219734191895 seconds.
2023-07-27 14:52:49.649205452 [2023-07-27 14:52:49] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.16.mp4 in 2.7733681201934814 seconds.
2023-07-27 14:52:52.815865164 [2023-07-27 14:52:52] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.26.mp4 in 2.766500949859619 seconds.
2023-07-27 14:52:55.708985391 [2023-07-27 14:52:55] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.16.mp4 in 2.753082036972046 seconds.
2023-07-27 14:52:58.697371241 [2023-07-27 14:52:58] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.26.mp4 in 2.7924540042877197 seconds.
2023-07-27 14:53:01.773608922 [2023-07-27 14:53:01] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.17.mp4 in 2.83032488822937 seconds.
2023-07-27 14:53:04.880077904 [2023-07-27 14:53:04] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.27.mp4 in 2.8705949783325195 seconds.
2023-07-27 14:53:07.837504876 [2023-07-27 14:53:07] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.37.mp4 in 2.772428035736084 seconds.
2023-07-27 14:53:10.835078066 [2023-07-27 14:53:10] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.47.mp4 in 2.780242919921875 seconds.
2023-07-27 14:53:13.794528070 [2023-07-27 14:53:13] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.36.mp4 in 2.7508840560913086 seconds.
2023-07-27 14:53:16.815387919 [2023-07-27 14:53:16] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.46.mp4 in 2.7242228984832764 seconds.
2023-07-27 14:53:20.046110795 [2023-07-27 14:53:20] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.36.mp4 in 3.063145160675049 seconds.
2023-07-27 14:53:23.023597905 [2023-07-27 14:53:23] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.46.mp4 in 2.789849042892456 seconds.
2023-07-27 14:53:25.665285688 [2023-07-27 14:53:25] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.37.mp4 in 2.4909911155700684 seconds.
2023-07-27 14:53:28.683837655 [2023-07-27 14:53:28] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.46.mp4 in 2.7974019050598145 seconds.
2023-07-27 14:53:31.696797664 [2023-07-27 14:53:31] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.57.mp4 in 2.8094191551208496 seconds.
2023-07-27 14:53:34.678908144 [2023-07-27 14:53:34] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.07.mp4 in 2.807781934738159 seconds.
2023-07-27 14:53:37.700928489 [2023-07-27 14:53:37] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.17.mp4 in 2.8578808307647705 seconds.
2023-07-27 14:53:40.810218748 [2023-07-27 14:53:40] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.56.mp4 in 2.919498920440674 seconds.
2023-07-27 14:53:43.785736486 [2023-07-27 14:53:43] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.06.mp4 in 2.787822961807251 seconds.
2023-07-27 14:53:46.896286987 [2023-07-27 14:53:46] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.16.mp4 in 2.8843400478363037 seconds.
2023-07-27 14:53:49.849318334 [2023-07-27 14:53:49] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.57.mp4 in 2.8049111366271973 seconds.
2023-07-27 14:53:54.355680230 [2023-07-27 14:53:54] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.07.mp4 in 4.337353944778442 seconds.
2023-07-27 14:53:59.343875193 [2023-07-27 14:53:59] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.17.mp4 in 4.821834087371826 seconds.
2023-07-27 14:54:04.400656221 [2023-07-27 14:54:04] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.56.mp4 in 4.908252954483032 seconds.
2023-07-27 14:54:09.597482962 [2023-07-27 14:54:09] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.06.mp4 in 5.061578989028931 seconds.
2023-07-27 14:54:14.727572184 [2023-07-27 14:54:14] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.16.mp4 in 4.95068883895874 seconds.
2023-07-27 14:54:18.858567075 [2023-07-27 14:54:18] frigate.record DEBUG : Expiring 192 recordings
2023-07-27 14:54:19.030255548 [2023-07-27 14:54:19] frigate.record DEBUG : End camera: frigate_street_front.
2023-07-27 14:54:19.030336823 [2023-07-27 14:54:19] frigate.record DEBUG : Start camera: frigate_street_nw.
2023-07-27 14:54:19.209223005 [2023-07-27 14:54:19] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.27.mp4 in 4.3155388832092285 seconds.
2023-07-27 14:54:22.139087749 [2023-07-27 14:54:22] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.37.mp4 in 2.776745080947876 seconds.
2023-07-27 14:54:27.209055086 [2023-07-27 14:54:27] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.47.mp4 in 4.884838819503784 seconds.
2023-07-27 14:54:32.727577331 [2023-07-27 14:54:32] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.57.mp4 in 4.9430060386657715 seconds.
2023-07-27 14:54:37.911973178 [2023-07-27 14:54:37] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.26.mp4 in 5.032839059829712 seconds.
2023-07-27 14:54:43.181262654 [2023-07-27 14:54:43] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.36.mp4 in 5.1069581508636475 seconds.
2023-07-27 14:54:48.060652981 [2023-07-27 14:54:48] frigate.record DEBUG : Expiring 190 recordings
2023-07-27 14:54:48.115701657 [2023-07-27 14:54:48] frigate.record DEBUG : End camera: frigate_street_nw.
2023-07-27 14:54:48.115706041 [2023-07-27 14:54:48] frigate.record DEBUG : Start camera: frigate_street_se.
2023-07-27 14:54:48.141978684 [2023-07-27 14:54:48] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.46.mp4 in 4.81535005569458 seconds.
2023-07-27 14:54:51.013401169 [2023-07-27 14:54:51] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.56.mp4 in 2.715502977371216 seconds.
2023-07-27 14:54:55.920729949 [2023-07-27 14:54:55] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.27.mp4 in 4.715717077255249 seconds.
2023-07-27 14:55:01.098977707 [2023-07-27 14:55:01] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.37.mp4 in 4.994410991668701 seconds.
2023-07-27 14:55:06.160359378 [2023-07-27 14:55:06] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.47.mp4 in 4.908405065536499 seconds.
2023-07-27 14:55:11.219192641 [2023-07-27 14:55:11] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.57.mp4 in 4.902930021286011 seconds.
2023-07-27 14:55:15.146357792 [2023-07-27 14:55:15] frigate.record DEBUG : Expiring 173 recordings
2023-07-27 14:55:15.222282726 [2023-07-27 14:55:15] frigate.record DEBUG : End camera: frigate_street_se.
2023-07-27 14:55:15.222387908 [2023-07-27 14:55:15] frigate.record DEBUG : Start camera: frigate_backyard.
2023-07-27 14:55:15.498984302 [2023-07-27 14:55:15] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.26.mp4 in 4.104842185974121 seconds.
2023-07-27 14:55:18.325911082 [2023-07-27 14:55:18] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.36.mp4 in 2.706555128097534 seconds.
2023-07-27 14:55:23.467614561 [2023-07-27 14:55:23] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.46.mp4 in 4.991831064224243 seconds.
2023-07-27 14:55:28.530655564 [2023-07-27 14:55:28] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.56.mp4 in 4.9489710330963135 seconds.
2023-07-27 14:55:28.561354799 [2023-07-27 14:55:28] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.565064279 [2023-07-27 14:55:28] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.567689923 [2023-07-27 14:55:28] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.570692819 [2023-07-27 14:55:28] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:33.305559493 [2023-07-27 14:55:33] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.36.mp4 in 4.606783151626587 seconds.
2023-07-27 14:55:38.091694580 [2023-07-27 14:55:38] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.46.mp4 in 4.640094995498657 seconds.
2023-07-27 14:55:43.146840524 [2023-07-27 14:55:43] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.56.mp4 in 4.912054061889648 seconds.
2023-07-27 14:55:44.605703963 [2023-07-27 14:55:44] frigate.record DEBUG : Expiring 192 recordings
2023-07-27 14:55:44.705574828 [2023-07-27 14:55:44] frigate.record DEBUG : End camera: frigate_backyard.
2023-07-27 14:55:44.705579297 [2023-07-27 14:55:44] frigate.record DEBUG : End all cameras.
2023-07-27 14:55:44.705581368 [2023-07-27 14:55:44] frigate.record DEBUG : End expire recordings (new).
2023-07-27 14:55:44.705582996 [2023-07-27 14:55:44] frigate.record DEBUG : Start expire files (legacy).
2023-07-27 14:55:44.751749462 [2023-07-27 14:55:44] frigate.record DEBUG : Oldest recording in the db: 1689601642.515418
2023-07-27 14:55:48.178487178 [2023-07-27 14:55:48] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/55.06.mp4 in 4.88518500328064 seconds.
2023-07-27 14:55:51.890108621 [2023-07-27 14:55:51] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/55.16.mp4 in 3.5519468784332275 seconds.
2023-07-27 14:55:56.885556220 [2023-07-27 14:55:56] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.36.mp4 in 4.819516897201538 seconds.
2023-07-27 14:56:01.939756131 [2023-07-27 14:56:01] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.46.mp4 in 4.904176950454712 seconds.
2023-07-27 14:56:06.808378756 [2023-07-27 14:56:06] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.56.mp4 in 4.73555588722229 seconds.
2023-07-27 14:56:12.329512784 [2023-07-27 14:56:12] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/55.07.mp4 in 5.364140033721924 seconds.
2023-07-27 14:56:18.414092681 [2023-07-27 14:56:18] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/55.17.mp4 in 5.933212041854858 seconds.
2023-07-27 14:56:24.186002145 [2023-07-27 14:56:24] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.36.mp4 in 5.628190040588379 seconds.
2023-07-27 14:56:30.308024214 [2023-07-27 14:56:30] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.46.mp4 in 5.9718639850616455 seconds.
2023-07-27 14:56:35.452113307 [2023-07-27 14:56:35] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.56.mp4 in 4.97012996673584 seconds.
2023-07-27 14:56:40.990185255 [2023-07-27 14:56:40] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/55.06.mp4 in 5.417521953582764 seconds.
2023-07-27 14:56:46.175705449 [2023-07-27 14:56:46] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/55.16.mp4 in 4.98787784576416 seconds.
2023-07-27 14:56:51.373647527 [2023-07-27 14:56:51] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.36.mp4 in 5.038317918777466 seconds.
2023-07-27 14:56:54.940395663 [2023-07-27 14:56:54] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.46.mp4 in 3.3942999839782715 seconds.
2023-07-27 14:56:58.543646121 [2023-07-27 14:56:58] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.56.mp4 in 3.4380130767822266 seconds.
2023-07-27 14:57:02.081852655 [2023-07-27 14:57:02] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/55.06.mp4 in 3.396026134490967 seconds.
Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view.
As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written.
I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now.
Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view.
As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written.
I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now.
Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.
I suggest... How about we look into my camera output in more detail and see if we can debug what's being sent to Frigate? Do you want a raw output stream? I can send you a link from cloud storage for debug purposes. What do you propose other than the storage? It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.
SMART output follows in next comment
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-10-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-xxxxxxxxxx
LU WWN Device Id: 5 0014ee xxxxxx
Firmware Version: 80.00A80
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Jul 27 16:53:55 2023 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (40500) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 406) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 191 175 021 Pre-fail Always - 5450
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 852
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 001 001 000 Old_age Always - 78776
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 775
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 226
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 625
194 Temperature_Celsius 0x0022 110 103 000 Old_age Always - 40
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 52260 -
# 2 Short offline Completed without error 00% 52236 -
# 3 Short offline Completed without error 00% 46009 -
# 4 Short offline Completed without error 00% 45985 -
# 5 Short offline Completed without error 00% 45961 -
# 6 Short offline Completed without error 00% 45937 -
# 7 Short offline Completed without error 00% 45913 -
# 8 Short offline Completed without error 00% 45889 -
# 9 Short offline Completed without error 00% 45865 -
#10 Short offline Completed without error 00% 45841 -
#11 Short offline Completed without error 00% 45817 -
#12 Short offline Completed without error 00% 45793 -
#13 Short offline Completed without error 00% 45769 -
#14 Short offline Completed without error 00% 45745 -
#15 Short offline Completed without error 00% 45721 -
#16 Short offline Completed without error 00% 45697 -
#17 Short offline Completed without error 00% 45673 -
#18 Short offline Completed without error 00% 45649 -
#19 Short offline Completed without error 00% 45625 -
#20 Extended offline Completed without error 00% 43179 -
#21 Short offline Completed without error 00% 8992 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.
Frigate 0.11 did not have these warning logs so there would be no way to know except notice the recording segment wasn't there.
It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.
For one it's not constant write it's separate writes for each segment. The timing logs are very simple around the ffmpeg call to optimize the segment metadata and write to storage so there's no room for error in that calculation.
Can you show your docker compose? One explanation would be if /tmp/cache isn't using RAM but is using disk instead.
I cannot do write test because I'm on the live system
Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.
Frigate 0.11 did not have these warning logs so there would be no way to know except notice the recording segment wasn't there.
It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.
For one it's not constant write it's separate writes for each segment. The timing logs are very simple around the ffmpeg call to optimize the segment metadata and write to storage so there's no room for error in that calculation.
Can you show your docker compose? One explanation would be if /tmp/cache isn't using RAM but is using disk instead.
I don't use compose. Per previous reply I posted by CLI command. Copying again for your convenience:
docker create --name frigate --restart=always -p 5000:5000 --gpus=all --cap-add=CAP_PERFMON --mount type=tmpfs,target=/tmp/cache,tmpfs-size=2000000000 --shm-size=5g --device /dev/bus/usb:/dev/bus/usb --device /dev/dri/renderD128 -v /cctv:/media/frigate -v /home/user1/dockerbindmounts/frigate/config.yml:/config/config.yml -v /home/user1/dockerbindmounts/frigate/db/:/db/ -v /etc/localtime:/etc/localtime:ro -v /home/user1/trt-models:/trt-models -e FRIGATE_RTSP_PASSWORD='removedforprivacy' -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,video,utility -e YOLO_MODELS=yolov7-tiny-416 ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt && docker start frigate
"/home/user1" is in a mirrored RAID of 14TB. "/cctv" is the 3TB disk I posted above where Frigate stores the recordings.
Right. For one I would suggest setting shm size to 1g. I can't remember the specifics of my previous recommendation but for 4 cameras 5g is excessive.
We saw another case with a user seeing this issue where for some reason when they specified 2GB for /tmp/cache it would use the storage but 1GB would use RAM. If you run df
inside the frigate container it should give more info on the type of mount that is being used for /tmp/cache
docker exec -it frigate bash root@4ea5bdb42c53:/opt/frigate# df Filesystem 1K-blocks Used Available Use% Mounted on overlay 256920004 29829712 213966708 13% / tmpfs 65536 0 65536 0% /dev shm 5242880 11752 5231128 1% /dev/shm /dev/mapper/RAID-Home 15625744384 4463767664 11159045040 29% /trt-models /dev/sdc1 2928229216 2354485980 453834536 84% /media/frigate /dev/nvme0n1p3 256920004 12746140 231050280 6% /usr/share/zoneinfo/Etc/UTC /dev/nvme0n1p4 256920004 29829712 213966708 13% /etc/hosts tmpfs 1953128 23568 1929560 2% /tmp/cache tmpfs 16297808 12 16297796 1% /proc/driver/nvidia tmpfs 3259564 6276 3253288 1% /run/nvidia-persistenced/socket udev 16251384 0 16251384 0% /dev/nvidia0 tmpfs 16297808 0 16297808 0% /proc/asound tmpfs 16297808 0 16297808 0% /proc/acpi tmpfs 16297808 0 16297808 0% /sys/firmware root@4ea5bdb42c53:/opt/frigate#
I increased the shm considerably just so there was absolutely zero doubt that it's not that.
shm now 1g.
What do copy times look like currently?
Do you mean you want the logs? or benchmark from gnome disk?
logs
I just spun a new container because of changing the cli paramenter for the shm so frigate has not been running for more than 5 mins. Here's what it's showing so far and don't worry it will show the same issue we're discussing in a few mins.
2023-07-27 17:45:38.870272750 [INFO] Starting Frigate...
2023-07-27 17:45:39.577489775 [2023-07-27 17:45:39] frigate.app INFO : Starting Frigate (0.12.1-367d724)
2023-07-27 17:45:39.587169227 [2023-07-27 17:45:39] peewee_migrate INFO : Starting migrations
2023-07-27 17:45:39.658487787 [2023-07-27 17:45:39] peewee_migrate INFO : There is nothing to migrate
2023-07-27 17:45:39.681029022 [2023-07-27 17:45:39] detector.tensorrt INFO : Starting detection process: 864
2023-07-27 17:45:39.681904459 [2023-07-27 17:45:39] frigate.app INFO : Output process started: 866
2023-07-27 17:45:39.687633131 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_street_front: 869
2023-07-27 17:45:39.687687773 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_street_nw: 870
2023-07-27 17:45:39.693073311 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_street_se: 873
2023-07-27 17:45:39.693119466 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_backyard: 874
2023-07-27 17:45:39.696113238 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_street_front: 875
2023-07-27 17:45:39.698927529 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_street_nw: 878
2023-07-27 17:45:39.700931817 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_street_se: 882
2023-07-27 17:45:39.703259426 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_backyard: 887
2023-07-27 17:45:39.927048454 [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 403 (MiB)
2023-07-27 17:45:39.937806476 [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO : Loaded engine size: 34 MiB
2023-07-27 17:45:39.938086135 [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 17:45:40.087250178 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 513 (MiB)
2023-07-27 17:45:40.165109271 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 559 (MiB)
2023-07-27 17:45:40.166873912 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 17:45:40.169431761 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 551 (MiB)
2023-07-27 17:45:40.169435428 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 559 (MiB)
2023-07-27 17:45:40.169448689 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 17:45:58.288109604 [2023-07-27 17:45:58] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.41.mp4 in 3.342507839202881 seconds.
2023-07-27 17:46:01.760975016 [2023-07-27 17:46:01] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.41.mp4 in 3.342647075653076 seconds.
2023-07-27 17:46:05.129348800 [2023-07-27 17:46:05] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.41.mp4 in 3.2932169437408447 seconds.
2023-07-27 17:46:08.550982720 [2023-07-27 17:46:08] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.41.mp4 in 3.347458839416504 seconds.
2023-07-27 17:46:11.924035339 [2023-07-27 17:46:11] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.50.mp4 in 3.2893340587615967 seconds.
2023-07-27 17:46:15.317924284 [2023-07-27 17:46:15] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.50.mp4 in 3.279114007949829 seconds.
2023-07-27 17:46:18.629513704 [2023-07-27 17:46:18] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.50.mp4 in 3.2351670265197754 seconds.
2023-07-27 17:46:21.979261148 [2023-07-27 17:46:21] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.50.mp4 in 3.268557071685791 seconds.
2023-07-27 17:46:25.345868751 [2023-07-27 17:46:25] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.00.mp4 in 3.2627789974212646 seconds.
2023-07-27 17:46:28.706942690 [2023-07-27 17:46:28] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.10.mp4 in 3.2569751739501953 seconds.
2023-07-27 17:46:32.092718168 [2023-07-27 17:46:32] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.00.mp4 in 3.310637950897217 seconds.
2023-07-27 17:46:35.474481726 [2023-07-27 17:46:35] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.10.mp4 in 3.2653961181640625 seconds.
2023-07-27 17:46:38.891189180 [2023-07-27 17:46:38] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.00.mp4 in 3.295841932296753 seconds.
2023-07-27 17:46:39.860844442 [2023-07-27 17:46:39] frigate.record DEBUG : Start expire recordings (new).
2023-07-27 17:46:39.861027446 [2023-07-27 17:46:39] frigate.record DEBUG : Start deleted cameras.
2023-07-27 17:46:39.864309059 [2023-07-27 17:46:39] frigate.record DEBUG : Expiring 0 recordings
2023-07-27 17:46:39.865500156 [2023-07-27 17:46:39] frigate.record DEBUG : End deleted cameras.
2023-07-27 17:46:39.865556063 [2023-07-27 17:46:39] frigate.record DEBUG : Start all cameras.
2023-07-27 17:46:39.865609077 [2023-07-27 17:46:39] frigate.record DEBUG : Start camera: frigate_street_front.
2023-07-27 17:46:42.292586308 [2023-07-27 17:46:42] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.10.mp4 in 3.279989004135132 seconds.
2023-07-27 17:46:45.706613798 [2023-07-27 17:46:45] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.00.mp4 in 3.264413833618164 seconds.
2023-07-27 17:46:51.528580781 [2023-07-27 17:46:51] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.10.mp4 in 5.636636018753052 seconds.
2023-07-27 17:46:56.901660140 [2023-07-27 17:46:56] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.20.mp4 in 5.226005792617798 seconds.
2023-07-27 17:47:02.976255345 [2023-07-27 17:47:02] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.29.mp4 in 5.962038993835449 seconds.
2023-07-27 17:47:08.963150298 [2023-07-27 17:47:08] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.39.mp4 in 5.863729953765869 seconds.
2023-07-27 17:47:15.009231881 [2023-07-27 17:47:15] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.20.mp4 in 5.9348108768463135 seconds.
2023-07-27 17:47:20.818260089 [2023-07-27 17:47:20] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.30.mp4 in 5.67697286605835 seconds.
Annnd it happened again:
2023-07-27 17:45:38.870272750 [INFO] Starting Frigate...
2023-07-27 17:45:39.577489775 [2023-07-27 17:45:39] frigate.app INFO : Starting Frigate (0.12.1-367d724)
2023-07-27 17:45:39.587169227 [2023-07-27 17:45:39] peewee_migrate INFO : Starting migrations
2023-07-27 17:45:39.658487787 [2023-07-27 17:45:39] peewee_migrate INFO : There is nothing to migrate
2023-07-27 17:45:39.681029022 [2023-07-27 17:45:39] detector.tensorrt INFO : Starting detection process: 864
2023-07-27 17:45:39.681904459 [2023-07-27 17:45:39] frigate.app INFO : Output process started: 866
2023-07-27 17:45:39.687633131 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_street_front: 869
2023-07-27 17:45:39.687687773 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_street_nw: 870
2023-07-27 17:45:39.693073311 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_street_se: 873
2023-07-27 17:45:39.693119466 [2023-07-27 17:45:39] frigate.app INFO : Camera processor started for frigate_backyard: 874
2023-07-27 17:45:39.696113238 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_street_front: 875
2023-07-27 17:45:39.698927529 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_street_nw: 878
2023-07-27 17:45:39.700931817 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_street_se: 882
2023-07-27 17:45:39.703259426 [2023-07-27 17:45:39] frigate.app INFO : Capture process started for frigate_backyard: 887
2023-07-27 17:45:39.927048454 [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 403 (MiB)
2023-07-27 17:45:39.937806476 [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt INFO : Loaded engine size: 34 MiB
2023-07-27 17:45:39.938086135 [2023-07-27 17:45:39] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 17:45:40.087250178 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 513 (MiB)
2023-07-27 17:45:40.165109271 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 559 (MiB)
2023-07-27 17:45:40.166873912 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 17:45:40.169431761 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 551 (MiB)
2023-07-27 17:45:40.169435428 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 559 (MiB)
2023-07-27 17:45:40.169448689 [2023-07-27 17:45:40] frigate.detectors.plugins.tensorrt INFO : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 17:45:58.288109604 [2023-07-27 17:45:58] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.41.mp4 in 3.342507839202881 seconds.
2023-07-27 17:46:01.760975016 [2023-07-27 17:46:01] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.41.mp4 in 3.342647075653076 seconds.
2023-07-27 17:46:05.129348800 [2023-07-27 17:46:05] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.41.mp4 in 3.2932169437408447 seconds.
2023-07-27 17:46:08.550982720 [2023-07-27 17:46:08] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.41.mp4 in 3.347458839416504 seconds.
2023-07-27 17:46:11.924035339 [2023-07-27 17:46:11] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/45.50.mp4 in 3.2893340587615967 seconds.
2023-07-27 17:46:15.317924284 [2023-07-27 17:46:15] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/45.50.mp4 in 3.279114007949829 seconds.
2023-07-27 17:46:18.629513704 [2023-07-27 17:46:18] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/45.50.mp4 in 3.2351670265197754 seconds.
2023-07-27 17:46:21.979261148 [2023-07-27 17:46:21] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/45.50.mp4 in 3.268557071685791 seconds.
2023-07-27 17:46:25.345868751 [2023-07-27 17:46:25] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.00.mp4 in 3.2627789974212646 seconds.
2023-07-27 17:46:28.706942690 [2023-07-27 17:46:28] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.10.mp4 in 3.2569751739501953 seconds.
2023-07-27 17:46:32.092718168 [2023-07-27 17:46:32] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.00.mp4 in 3.310637950897217 seconds.
2023-07-27 17:46:35.474481726 [2023-07-27 17:46:35] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.10.mp4 in 3.2653961181640625 seconds.
2023-07-27 17:46:38.891189180 [2023-07-27 17:46:38] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.00.mp4 in 3.295841932296753 seconds.
2023-07-27 17:46:39.860844442 [2023-07-27 17:46:39] frigate.record DEBUG : Start expire recordings (new).
2023-07-27 17:46:39.861027446 [2023-07-27 17:46:39] frigate.record DEBUG : Start deleted cameras.
2023-07-27 17:46:39.864309059 [2023-07-27 17:46:39] frigate.record DEBUG : Expiring 0 recordings
2023-07-27 17:46:39.865500156 [2023-07-27 17:46:39] frigate.record DEBUG : End deleted cameras.
2023-07-27 17:46:39.865556063 [2023-07-27 17:46:39] frigate.record DEBUG : Start all cameras.
2023-07-27 17:46:39.865609077 [2023-07-27 17:46:39] frigate.record DEBUG : Start camera: frigate_street_front.
2023-07-27 17:46:42.292586308 [2023-07-27 17:46:42] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.10.mp4 in 3.279989004135132 seconds.
2023-07-27 17:46:45.706613798 [2023-07-27 17:46:45] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.00.mp4 in 3.264413833618164 seconds.
2023-07-27 17:46:51.528580781 [2023-07-27 17:46:51] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.10.mp4 in 5.636636018753052 seconds.
2023-07-27 17:46:56.901660140 [2023-07-27 17:46:56] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.20.mp4 in 5.226005792617798 seconds.
2023-07-27 17:47:02.976255345 [2023-07-27 17:47:02] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.29.mp4 in 5.962038993835449 seconds.
2023-07-27 17:47:08.963150298 [2023-07-27 17:47:08] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/46.39.mp4 in 5.863729953765869 seconds.
2023-07-27 17:47:15.009231881 [2023-07-27 17:47:15] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.20.mp4 in 5.9348108768463135 seconds.
2023-07-27 17:47:20.818260089 [2023-07-27 17:47:20] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.30.mp4 in 5.67697286605835 seconds.
2023-07-27 17:47:26.696242611 [2023-07-27 17:47:26] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/46.40.mp4 in 5.780220031738281 seconds.
2023-07-27 17:47:30.947251243 [2023-07-27 17:47:30] frigate.record DEBUG : Expiring 254 recordings
2023-07-27 17:47:31.160119645 [2023-07-27 17:47:31] frigate.record DEBUG : End camera: frigate_street_front.
2023-07-27 17:47:31.160155044 [2023-07-27 17:47:31] frigate.record DEBUG : Start camera: frigate_street_nw.
2023-07-27 17:47:31.374643984 [2023-07-27 17:47:31] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.20.mp4 in 4.5776848793029785 seconds.
2023-07-27 17:47:35.087833908 [2023-07-27 17:47:35] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.30.mp4 in 3.580332040786743 seconds.
2023-07-27 17:47:40.649230966 [2023-07-27 17:47:40] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_nw/46.40.mp4 in 5.437561988830566 seconds.
2023-07-27 17:47:46.234215616 [2023-07-27 17:47:46] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.20.mp4 in 5.468185186386108 seconds.
2023-07-27 17:47:51.996834563 [2023-07-27 17:47:51] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.30.mp4 in 5.669494152069092 seconds.
2023-07-27 17:47:57.703996357 [2023-07-27 17:47:57] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_se/46.40.mp4 in 5.624220132827759 seconds.
2023-07-27 17:47:57.740316252 [2023-07-27 17:47:57] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:47:57.741450451 [2023-07-27 17:47:57] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:47:57.741472674 [2023-07-27 17:47:57] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:47:57.741709245 [2023-07-27 17:47:57] frigate.record WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-07-27 17:48:03.737077274 [2023-07-27 17:48:03] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.00.mp4 in 5.928970098495483 seconds.
2023-07-27 17:48:04.878764998 [2023-07-27 17:48:04] frigate.record DEBUG : Expiring 175 recordings
2023-07-27 17:48:04.988821564 [2023-07-27 17:48:04] frigate.record DEBUG : End camera: frigate_street_nw.
2023-07-27 17:48:04.989021816 [2023-07-27 17:48:04] frigate.record DEBUG : Start camera: frigate_street_se.
2023-07-27 17:48:07.479430642 [2023-07-27 17:48:07] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.10.mp4 in 3.604843854904175 seconds.
2023-07-27 17:48:13.026042592 [2023-07-27 17:48:13] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.20.mp4 in 5.439263105392456 seconds.
2023-07-27 17:48:19.169508223 [2023-07-27 17:48:19] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.30.mp4 in 6.029353141784668 seconds.
2023-07-27 17:48:25.108971065 [2023-07-27 17:48:25] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_backyard/47.40.mp4 in 5.83636999130249 seconds.
2023-07-27 17:48:31.183972145 [2023-07-27 17:48:31] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.00.mp4 in 5.951390027999878 seconds.
2023-07-27 17:48:37.581601426 [2023-07-27 17:48:37] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.10.mp4 in 6.262295961380005 seconds.
2023-07-27 17:48:43.592764277 [2023-07-27 17:48:43] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.20.mp4 in 5.860684156417847 seconds.
2023-07-27 17:48:49.617885615 [2023-07-27 17:48:49] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.30.mp4 in 5.906958103179932 seconds.
By the way, Immich finished transcoding my videos so now the CPU is idling at 20% tops with CPU temps mid 50C on average..It is not system load.
This is currently how my CPU is being utilized:
Yeah, these copy times are very high:
2023-07-27 17:48:31.183972145 [2023-07-27 17:48:31] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.00.mp4 in 5.951390027999878 seconds.
2023-07-27 17:48:37.581601426 [2023-07-27 17:48:37] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.10.mp4 in 6.262295961380005 seconds.
2023-07-27 17:48:43.592764277 [2023-07-27 17:48:43] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.20.mp4 in 5.860684156417847 seconds.
2023-07-27 17:48:49.617885615 [2023-07-27 17:48:49] frigate.record DEBUG : Copied /media/frigate/recordings/2023-07-27/16/frigate_street_front/47.30.mp4 in 5.906958103179932 seconds.
There's nothing frigate can do if the I/O is unable to keep up. For context, the users who were CPU bound had copy times of ~0.1 seconds per segment with segments at an average of 5MB. Personally, my highest bandwidth camera that is CBR 8192 kbps and has segments 10MB has a move time of ~0.5 seconds per segment.
But I don't understand, you're saying it's the IO, however I provided all sorts of data and even a disk benchmark... surely there's a way to debug deeper to understand what is going on?
Memory usage:
Benchmark with docker service stopped and zero containers running. I will shortly do the same with all containers running to compare.
user@server:~$ sudo systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
docker.socket
user@server:~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=TGS --filename=TGS --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
TGS: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
TGS: Laying out IO file (1 file / 4096MiB)
^Cbs: 1 (f=1): [m(1)][14.9%][r=4092KiB/s,w=1268KiB/s][r=1023,w=317 IOPS][eta 09m:35s]
fio: terminating on signal 2
Jobs: 1 (f=1): [m(1)][15.1%][r=4480KiB/s,w=1533KiB/s][r=1120,w=383 IOPS][eta 09m:34s]
TGS: (groupid=0, jobs=1): err= 0: pid=561308: Thu Jul 27 18:11:42 2023
read: IOPS=1173, BW=4693KiB/s (4806kB/s)(463MiB/101086msec)
bw ( KiB/s): min= 816, max= 5368, per=100.00%, avg=4695.14, stdev=689.26, samples=202
iops : min= 204, max= 1342, avg=1173.78, stdev=172.32, samples=202
write: IOPS=390, BW=1563KiB/s (1601kB/s)(154MiB/101086msec); 0 zone resets
bw ( KiB/s): min= 320, max= 1984, per=100.00%, avg=1564.15, stdev=256.99, samples=202
iops : min= 80, max= 496, avg=391.02, stdev=64.24, samples=202
cpu : usr=0.57%, sys=5.14%, ctx=126373, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=118602,39501,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=4693KiB/s (4806kB/s), 4693KiB/s-4693KiB/s (4806kB/s-4806kB/s), io=463MiB (486MB), run=101086-101086msec
WRITE: bw=1563KiB/s (1601kB/s), 1563KiB/s-1563KiB/s (1601kB/s-1601kB/s), io=154MiB (162MB), run=101086-101086msec
I can run a benchmark on my Samsung 980 Pro and get a write speed of 5 GB/s, that does not mean that a 10MB segment will be written in 0.002 seconds.
Benchmark with docker service stopped and zero containers running. I will shortly do the same with all containers running to compare.
user@server:~$ sudo systemctl stop docker Warning: Stopping docker.service, but it can still be activated by: docker.socket user@server:~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=TGS --filename=TGS --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 TGS: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.33 Starting 1 process TGS: Laying out IO file (1 file / 4096MiB) ^Cbs: 1 (f=1): [m(1)][14.9%][r=4092KiB/s,w=1268KiB/s][r=1023,w=317 IOPS][eta 09m:35s] fio: terminating on signal 2 Jobs: 1 (f=1): [m(1)][15.1%][r=4480KiB/s,w=1533KiB/s][r=1120,w=383 IOPS][eta 09m:34s] TGS: (groupid=0, jobs=1): err= 0: pid=561308: Thu Jul 27 18:11:42 2023 read: IOPS=1173, BW=4693KiB/s (4806kB/s)(463MiB/101086msec) bw ( KiB/s): min= 816, max= 5368, per=100.00%, avg=4695.14, stdev=689.26, samples=202 iops : min= 204, max= 1342, avg=1173.78, stdev=172.32, samples=202 write: IOPS=390, BW=1563KiB/s (1601kB/s)(154MiB/101086msec); 0 zone resets bw ( KiB/s): min= 320, max= 1984, per=100.00%, avg=1564.15, stdev=256.99, samples=202 iops : min= 80, max= 496, avg=391.02, stdev=64.24, samples=202 cpu : usr=0.57%, sys=5.14%, ctx=126373, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=118602,39501,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=4693KiB/s (4806kB/s), 4693KiB/s-4693KiB/s (4806kB/s-4806kB/s), io=463MiB (486MB), run=101086-101086msec WRITE: bw=1563KiB/s (1601kB/s), 1563KiB/s-1563KiB/s (1601kB/s-1601kB/s), io=154MiB (162MB), run=101086-101086msec
You are getting a write speed of 1563KiB/s
which if you compare that to your 8MB segment size is 5.3 seconds, which again shows this to be the problem.
Please wait, I will finish the benchmarks and once I post all the results we can continue to discuss. I'm not done yet. I am not trying to prove anything. If the problem is my system, the problem is my system, I will accept that and ask for suggestions. I have not done the comparison and another script bench. Bear with.
This is with all docker containers running:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=TGS --filename=TGS --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
TGS: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 process
^Cbs: 1 (f=1): [m(1)][18.0%][r=1933KiB/s,w=760KiB/s][r=483,w=190 IOPS][eta 18m:38s]
fio: terminating on signal 2
TGS: (groupid=0, jobs=1): err= 0: pid=575208: Thu Jul 27 18:17:53 2023
read: IOPS=576, BW=2305KiB/s (2360kB/s)(555MiB/246583msec)
bw ( KiB/s): min= 16, max= 5120, per=100.00%, avg=2312.20, stdev=801.47, samples=491
iops : min= 4, max= 1280, avg=578.05, stdev=200.37, samples=491
write: IOPS=191, BW=767KiB/s (785kB/s)(185MiB/246583msec); 0 zone resets
bw ( KiB/s): min= 56, max= 1840, per=100.00%, avg=767.74, stdev=294.21, samples=492
iops : min= 14, max= 460, avg=191.93, stdev=73.55, samples=492
cpu : usr=0.33%, sys=2.82%, ctx=139743, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=142064,47276,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=2305KiB/s (2360kB/s), 2305KiB/s-2305KiB/s (2360kB/s-2360kB/s), io=555MiB (582MB), run=246583-246583msec
WRITE: bw=767KiB/s (785kB/s), 767KiB/s-767KiB/s (785kB/s-785kB/s), io=185MiB (194MB), run=246583-246583msec
YABS on the /home volume (16TB RAID) where the /db is stored:
curl -sL yabs.sh | bash
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
# Yet-Another-Bench-Script #
# v2023-04-23 #
# https://github.com/masonr/yet-another-bench-script #
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
Thu 27 Jul 18:18:28 BST 2023
Basic System Information:
---------------------------------
Uptime : 0 days, 17 hours, 22 minutes
Processor : 13th Gen Intel(R) Core(TM) i5-13500
CPU cores : 20 @ 1430.338 MHz
AES-NI : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM : 31.1 GiB
Swap : 64.0 GiB
Disk : 18.1 TiB
Distro : Debian GNU/Linux 12 (bookworm)
Kernel : 6.1.0-10-amd64
VM Type : NONE
IPv4/IPv6 : ✔ Online / ❌ Offline
IPv4 Network Information:
---------------------------------
ISP : Virgin Media
ASN : AS5089 Virgin Media Limited
Location : City, England (ENG)
Country : United Kingdom
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 2.67 MB/s (669) | 19.02 MB/s (297)
Write | 2.68 MB/s (670) | 19.64 MB/s (306)
Total | 5.35 MB/s (1.3k) | 38.67 MB/s (603)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 18.72 MB/s (36) | 23.42 MB/s (22)
Write | 20.20 MB/s (39) | 26.42 MB/s (25)
Total | 38.92 MB/s (75) | 49.84 MB/s (47)
iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider | Location (Link) | Send Speed | Recv Speed | Ping
----- | ----- | ---- | ---- | ----
Clouvider | London, UK (10G) | 35.1 Mbits/sec | 729 Mbits/sec | 21.2 ms
Scaleway | Paris, FR (10G) | 35.8 Mbits/sec | 718 Mbits/sec | 29.8 ms
NovoServe | North Holland, NL (40G) | 36.2 Mbits/sec | 686 Mbits/sec | 19.9 ms
Uztelecom | Tashkent, UZ (10G) | 21.8 Mbits/sec | 612 Mbits/sec | 120 ms
Clouvider | NYC, NY, US (10G) | 20.9 Mbits/sec | 640 Mbits/sec | 110 ms
Clouvider | Dallas, TX, US (10G) | 21.7 Mbits/sec | 625 Mbits/sec | 152 ms
Clouvider | Los Angeles, CA, US (10G) | 19.5 Mbits/sec | 513 Mbits/sec | 186 ms
Geekbench 6 Benchmark Test:
---------------------------------
Test | Value
|
Single Core | 2513
Multi Core | 13865
Full Test | https://browser.geekbench.com/v6/cpu/2058641
YABS completed in 10 min 49 sec
Next will be YABS in the /cctv where the recordings are stored.
I am struggling to get the volume tested. I think it's being used so the system is not releasing it. Any idea? I am really tired and my head isn't thinking very well because of that, could use with some suggestions. I really want to benchmark this drive and see if it is the drive. Not sure how to release the mount so it can run.
I'm now running sudo e2fsck -cfpv /dev/sdc1 - sdc1 being the /cctv drive. I managed to unmount using sudo fuser -km /dev/sdc1 and sudo fuser -km /cctv. I don't want to waste your time, I want to make sure the drive isn't the problem. Because the server definitely isn't... the drive has been running for 9 years mostly 24/7... WD drives are freaking impressive. SMART says it's fine, but who knows. It's checking for bad blocks now.
I'm sure you know, but worth saying that a drive can pass SMART, have no bad sectors, but still be slow, and of course speed can vary by the workload. The first benchmark shows a write speed that matches the copy segment times.
I don't know what it would be if not the drive, or something related to the drive from the host or the mount.
The command is still running... it's a large drive. But I opened iostat and it seems like it's reading at a healthy 135MBps
When you say "... but still be slow...", well, if it's not a hardware issue then it is fixable. I really want to find out. Because if I can't, it means I just can't use your solution as my NVR and that would be a huge pity for me and anyone with the same type of issue. Maybe people don't have this problem now because they potentially run 1 or 2 cams... what happens when they scale, their hardware is totally fine but there is a bug or whatever? I mean, it's your time and effort, you decide, but I hope you will take the route of walking me through and assisting. It will increase both our knowledge and ability to resolve this type of issue in the future as well as help you diagnose similar problems from others. As soon as the badblock scan is done, I will see about benchmarking properly. I was running docker - I thought I stopped it but I forgot all my containers have "restart=always" so they were getting spun again and affecting the benchmarks. We are currently looking at what the disk itself is doing. Let's rule out a hardware problem and hopefully go from there.
Describe the problem you are having
Per the title Log says "unable to keep up with recording segments". I don't remember ever getting this on 0.11. I am getting lots of skips in my recordings. The server is more powerful than the one I was using before, but the HDD is the same. The only difference is the Frigate version. What's wrong with my config? I enabled G711 audio in my camera's stream, this is the only difference, but Frigate does not accept it anyway. Even with AAC it doesn't really work and never has. I tried a lot of different ways/presets. Do I have to completely disable audio in the camera to make this work?
Version
0.12.1-367D724
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
Operating system
HassOS
Install method
HassOS Addon
Coral version
USB
Network connection
Wired
Camera make and model
Anpviz 5MP
Any other information that may be helpful
No response