search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.88k stars 1.72k forks source link

[Support] TensorRT invalid key -17 #9608

Closed tkffaul closed 7 months ago

tkffaul commented 8 months ago

Describe the problem you are having

I've read up on this... but i still dont understand fully. I have frigate running on a Xeon E5 v2 in LXC -> Docker. I have gone over the config 100 different ways... If I load the same config on version 12 i have no issues... but a minute after loading 13 i get: frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for . Keeping the 6 most recent segments out of 7 and discarding the rest...

I get that it could be a speed issue (transfer)... but if it is, how does it work with the same docker and same LXC with version 12? The storage device is NFS and its local... i've tested the throughput and its big. Its out main storage server with a dedicated drive and pool just for this container. I have 8gb of Ram and 8 CPU cores with an nvidia card.... I just cant see how this is possible, so i'm checking to see if its a bug? I have 10 cameras most issues i see are over 30...

Version

0.13.1

Frigate config file

My config is huge and i dont feel like scrubbing it, if you want to see a section just ask and I'll post it.

Relevant log output

_

FFprobe output from your camera

_

Frigate stats

No response

Operating system

Proxmox

Install method

Docker CLI

Coral version

Other

Network connection

Wired

Camera make and model

Reolink

Any other information that may be helpful

No response

NickM-27 commented 8 months ago

as part of the rework on recordings we discovered a number of bugs in 0.12 that would lead to recordings silently being deleted when they shouldn't in 0.12. These have been fixed in 0.13 but that can lead to this issue occurring where it was masked before.

at a minimum would want to see a full screenshot of the system page and logs with frigate.record.maintainer set to debug

tkffaul commented 8 months ago

Not a Problem: Its going to take a couple screen shots for the system page, anything you want in particular?

Here is a snip out of the log...

`2024-02-02 14:02:29.828001850 [2024-02-02 14:02:29] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Warehouse_Rear/02.14.mp4 in 0.4718739986419678 seconds.

2024-02-02 14:02:30.008517676 [2024-02-02 14:02:30] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Yard/02.14.mp4 in 0.4629700183868408 seconds.

2024-02-02 14:02:30.242060239 [2024-02-02 14:02:30] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Office_Front/02.13.mp4 in 0.47750306129455566 seconds.

2024-02-02 14:02:39.871840204 [2024-02-02 14:02:39] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Storage_Rear/02.25.mp4 in 1.381500005722046 seconds.

2024-02-02 14:02:39.873524127 [2024-02-02 14:02:39] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Warehouse/02.25.mp4 in 1.1140990257263184 seconds.

2024-02-02 14:02:39.874704077 [2024-02-02 14:02:39] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Office_Parking/02.24.mp4 in 0.9019098281860352 seconds.

2024-02-02 14:02:39.875980047 [2024-02-02 14:02:39] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Warehouse_Rear/02.24.mp4 in 0.6692240238189697 seconds.

2024-02-02 14:02:39.968057261 [2024-02-02 14:02:39] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Storage_Front/02.24.mp4 in 0.5750370025634766 seconds.

2024-02-02 14:02:40.042952338 [2024-02-02 14:02:40] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Yard/02.24.mp4 in 0.4578261375427246 seconds.

2024-02-02 14:02:40.285648107 [2024-02-02 14:02:40] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Office_Front/02.23.mp4 in 0.4895501136779785 seconds.

2024-02-02 14:02:40.401544070 [2024-02-02 14:02:40] frigate.record.maintainer DEBUG : Copied /media/frigate/recordings/2024-02-02/19/Yard2/02.23.mp4 in 0.5660190582275391 seconds.`

NickM-27 commented 8 months ago

we will need a lot more logs than that. Preferably a full set of logs starting when frigate started and ending when the "unable to keep up" begins

the whole system page would be good to see

tkffaul commented 8 months ago

Ok part 1 System Page: 1

Part 2 Log File:

s6-rc: info: service s6rc-fdholder: starting

s6-rc: info: service s6rc-oneshot-runner: starting

s6-rc: info: service s6rc-oneshot-runner successfully started

s6-rc: info: service fix-attrs: starting

s6-rc: info: service s6rc-fdholder successfully started

s6-rc: info: service fix-attrs successfully started

s6-rc: info: service legacy-cont-init: starting

s6-rc: info: service legacy-cont-init successfully started

s6-rc: info: service trt-model-prepare: starting

s6-rc: info: service log-prepare: starting

s6-rc: info: service log-prepare successfully started

s6-rc: info: service nginx-log: starting

s6-rc: info: service go2rtc-log: starting

s6-rc: info: service frigate-log: starting

s6-rc: info: service nginx-log successfully started

s6-rc: info: service go2rtc-log successfully started

s6-rc: info: service go2rtc: starting

s6-rc: info: service frigate-log successfully started

s6-rc: info: service go2rtc successfully started

s6-rc: info: service go2rtc-healthcheck: starting

s6-rc: info: service go2rtc-healthcheck successfully started

2024-02-02 13:18:54.514354470  [INFO] Preparing new go2rtc config...

No models to convert.

s6-rc: info: service trt-model-prepare successfully started

s6-rc: info: service frigate: starting

s6-rc: info: service frigate successfully started

s6-rc: info: service nginx: starting

s6-rc: info: service nginx successfully started

s6-rc: info: service legacy-services: starting

2024-02-02 13:18:54.605253953  [INFO] Preparing Frigate...

2024-02-02 13:18:54.609072555  [INFO] Starting NGINX...

s6-rc: info: service legacy-services successfully started

2024-02-02 13:18:54.640065632  [INFO] Starting Frigate...

2024-02-02 13:18:55.472659146  [INFO] Starting go2rtc...

2024-02-02 13:18:55.669567675  13:18:55.669 INF go2rtc version 1.8.4 linux/amd64

2024-02-02 13:18:55.671187846  13:18:55.671 INF [api] listen addr=:1984

2024-02-02 13:18:55.671953330  13:18:55.671 INF [rtsp] listen addr=:8554

2024-02-02 13:18:55.672665491  13:18:55.672 INF [webrtc] listen addr=:8555

2024-02-02 13:18:59.206057661  [2024-02-02 13:18:59] frigate.app                    INFO    : Starting Frigate (0.13.1-34fb1c2)

2024-02-02 13:18:59.924001124  [2024-02-02 13:18:59] peewee_migrate.logs            INFO    : Starting migrations

2024-02-02 13:19:00.089377634  [2024-02-02 13:19:00] peewee_migrate.logs            INFO    : There is nothing to migrate

2024-02-02 13:19:00.131689058  [2024-02-02 13:19:00] frigate.app                    INFO    : Recording process started: 1512

2024-02-02 13:19:00.140773533  [2024-02-02 13:19:00] frigate.app                    INFO    : go2rtc process pid: 97

2024-02-02 13:19:00.239179163  [2024-02-02 13:19:00] frigate.app                    INFO    : Output process started: 1524

2024-02-02 13:19:00.881481693  [2024-02-02 13:19:00] detector.tensorrt              INFO    : Starting detection process: 1522

2024-02-02 13:19:01.147704620  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB

2024-02-02 13:19:01.565836020  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +7, GPU +8, now: CPU 183, GPU 75 (MiB)

2024-02-02 13:19:01.579389599  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 184, GPU 85 (MiB)

2024-02-02 13:19:01.591237548  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)

2024-02-02 13:19:01.624697906  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 150, GPU 77 (MiB)

2024-02-02 13:19:01.625012475  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 150, GPU 85 (MiB)

2024-02-02 13:19:01.630223055  [2024-02-02 13:19:01] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +13, now: CPU 0, GPU 47 (MiB)

2024-02-02 13:19:04.096434018  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Office_Front: 1559

2024-02-02 13:19:04.098592078  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Yard: 1560

2024-02-02 13:19:04.113480690  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Yard2: 1562

2024-02-02 13:19:04.128551291  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Office_Parking: 1564

2024-02-02 13:19:04.143229272  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Warehouse_Rear: 1566

2024-02-02 13:19:04.158112571  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Storage_Rear: 1568

2024-02-02 13:19:04.172956200  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Warehouse: 1570

2024-02-02 13:19:04.188663652  [2024-02-02 13:19:04] frigate.app                    INFO    : Camera processor started for Storage_Front: 1572

2024-02-02 13:19:04.203700929  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Office_Front: 1574

2024-02-02 13:19:04.218685683  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Yard: 1578

2024-02-02 13:19:04.234699251  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Yard2: 1581

2024-02-02 13:19:04.252516487  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Office_Parking: 1585

2024-02-02 13:19:04.269129042  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Warehouse_Rear: 1590

2024-02-02 13:19:04.288848737  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Storage_Rear: 1600

2024-02-02 13:19:04.308626801  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Warehouse: 1608

2024-02-02 13:19:04.330959630  [2024-02-02 13:19:04] frigate.app                    INFO    : Capture process started for Storage_Front: 1621

2024-02-02 13:19:04.507657708  [INFO] Starting go2rtc healthcheck service...

2024-02-02 13:19:14.856314953  192.168.88.5 - - [02/Feb/2024:13:19:14 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.268767619  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET / HTTP/1.1" 200 6637 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.468163158  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.621474595  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Office_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.631687156  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Office_Parking/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.634322886  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Storage_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.639524798  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Storage_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.644951314  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Warehouse/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.649432483  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.653685625  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Yard/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.657690083  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Yard2/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.954394463  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Office_Parking/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:16.964248438  192.168.88.5 - - [02/Feb/2024:13:19:16 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:17.036419876  192.168.88.5 - - [02/Feb/2024:13:19:17 -0500] "GET /api/Office_Front/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:17.040102272  192.168.88.5 - - [02/Feb/2024:13:19:17 -0500] "GET /api/Warehouse/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:17.043026781  192.168.88.5 - - [02/Feb/2024:13:19:17 -0500] "GET /api/Storage_Front/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:17.079386004  192.168.88.5 - - [02/Feb/2024:13:19:17 -0500] "GET /api/Yard/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:17.096037001  192.168.88.5 - - [02/Feb/2024:13:19:17 -0500] "GET /api/Storage_Rear/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:17.111488389  192.168.88.5 - - [02/Feb/2024:13:19:17 -0500] "GET /api/Yard2/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:24.309749627  [2024-02-02 13:19:24] watchdog.Yard                  INFO    : No frames received from Yard in 20 seconds. Exiting ffmpeg...

2024-02-02 13:19:24.310130079  [2024-02-02 13:19:24] watchdog.Yard                  INFO    : Waiting for ffmpeg to exit gracefully...

2024-02-02 13:19:24.325712340  [2024-02-02 13:19:24] watchdog.Yard2                 INFO    : No frames received from Yard2 in 20 seconds. Exiting ffmpeg...

2024-02-02 13:19:24.326378141  [2024-02-02 13:19:24] watchdog.Yard2                 INFO    : Waiting for ffmpeg to exit gracefully...

2024-02-02 13:19:24.357351185  [2024-02-02 13:19:24] watchdog.Office_Parking        INFO    : No frames received from Office_Parking in 20 seconds. Exiting ffmpeg...

2024-02-02 13:19:24.357815428  [2024-02-02 13:19:24] watchdog.Office_Parking        INFO    : Waiting for ffmpeg to exit gracefully...

2024-02-02 13:19:24.373349544  [2024-02-02 13:19:24] watchdog.Warehouse_Rear        INFO    : No frames received from Warehouse_Rear in 20 seconds. Exiting ffmpeg...

2024-02-02 13:19:24.373899287  [2024-02-02 13:19:24] watchdog.Warehouse_Rear        INFO    : Waiting for ffmpeg to exit gracefully...

2024-02-02 13:19:24.691814334  [2024-02-02 13:19:24] frigate.video                  ERROR   : Yard: Unable to read frames from ffmpeg process.

2024-02-02 13:19:24.692803641  [2024-02-02 13:19:24] frigate.video                  ERROR   : Yard: ffmpeg process is not running. exiting capture thread...

2024-02-02 13:19:24.830594248  [2024-02-02 13:19:24] frigate.video                  ERROR   : Yard2: Unable to read frames from ffmpeg process.

2024-02-02 13:19:24.831149138  [2024-02-02 13:19:24] frigate.video                  ERROR   : Yard2: ffmpeg process is not running. exiting capture thread...

2024-02-02 13:19:25.541841758  [2024-02-02 13:19:25] frigate.video                  ERROR   : Office_Parking: Unable to read frames from ffmpeg process.

2024-02-02 13:19:25.542556319  [2024-02-02 13:19:25] frigate.video                  ERROR   : Office_Parking: ffmpeg process is not running. exiting capture thread...

2024-02-02 13:19:25.609721131  [2024-02-02 13:19:25] frigate.video                  ERROR   : Warehouse_Rear: Unable to read frames from ffmpeg process.

2024-02-02 13:19:25.610481676  [2024-02-02 13:19:25] frigate.video                  ERROR   : Warehouse_Rear: ffmpeg process is not running. exiting capture thread...

2024-02-02 13:19:25.612874162  [2024-02-02 13:19:25] ffmpeg.Warehouse_Rear.record   ERROR   : rtsp://127.0.0.1:8554/Warehouse_Rear?video=copy&audio=copy: Invalid data found when processing input

2024-02-02 13:19:25.630450992  [2024-02-02 13:19:25] watchdog.Warehouse_Rear        INFO    : Terminating the existing ffmpeg process...

2024-02-02 13:19:25.630720503  [2024-02-02 13:19:25] watchdog.Warehouse_Rear        INFO    : Waiting for ffmpeg to exit gracefully...

2024-02-02 13:19:25.673485287  192.168.88.5 - - [02/Feb/2024:13:19:25 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:26.719422077  192.168.88.5 - - [02/Feb/2024:13:19:26 -0500] "GET /ws HTTP/1.1" 101 384 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:26.733258215  192.168.88.5 - - [02/Feb/2024:13:19:26 -0500] "GET / HTTP/1.1" 200 6637 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:26.924649773  192.168.88.5 - - [02/Feb/2024:13:19:26 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.044375882  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.045110010  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Parking/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.051625486  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.052816000  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.077361424  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Front/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.086887120  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Parking/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.088020938  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Front/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.092368941  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Rear/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.097833183  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Warehouse/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.100103666  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.102620777  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Yard/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.114246901  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Yard2/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.126159338  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Front/latest.jpg?h=486 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.145589562  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Parking/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.146620237  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Front/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.148035542  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Rear/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.148578203  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Warehouse/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.150252058  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.151403745  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Yard/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.152842335  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Yard2/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.391113070  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Parking/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.393403621  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Office_Front/latest.jpg?h=492 HTTP/1.1" 200 51329 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.395478735  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Rear/latest.jpg?h=492 HTTP/1.1" 200 70714 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.398220290  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Warehouse/latest.jpg?h=492 HTTP/1.1" 200 67318 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.400318738  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Storage_Front/latest.jpg?h=492 HTTP/1.1" 200 37204 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.424173067  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.438767519  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Yard/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:27.456657308  192.168.88.5 - - [02/Feb/2024:13:19:27 -0500] "GET /api/Yard2/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:29.906852981  192.168.88.5 - - [02/Feb/2024:13:19:29 -0500] "GET /ws HTTP/1.1" 101 4 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:29.922295910  192.168.88.5 - - [02/Feb/2024:13:19:29 -0500] "GET / HTTP/1.1" 200 6637 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.070127847  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.192300291  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.193515041  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Parking/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.194442355  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.196134127  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.231689907  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Front/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.234372318  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Parking/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.238877072  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Front/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.243582995  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Rear/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.247080128  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Warehouse/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.253290957  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.254596780  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Yard/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.258289426  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Yard2/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.275644504  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Front/latest.jpg?h=486 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.292737854  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Parking/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.294003257  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Front/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.295274293  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Rear/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.295651929  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Warehouse/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.296593435  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.300530827  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Yard/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.301483962  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Yard2/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.582022947  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Front/latest.jpg?h=492 HTTP/1.1" 200 51271 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.582211174  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Office_Parking/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.584533974  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Front/latest.jpg?h=492 HTTP/1.1" 200 37229 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.591905834  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Warehouse/latest.jpg?h=492 HTTP/1.1" 200 67138 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.592996559  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Storage_Rear/latest.jpg?h=492 HTTP/1.1" 200 70650 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.629221577  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.647434203  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Yard/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:30.664723923  192.168.88.5 - - [02/Feb/2024:13:19:30 -0500] "GET /api/Yard2/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:19:33.320871804  Process camera_processor:Office_Front:

2024-02-02 13:19:33.323633275  Traceback (most recent call last):

2024-02-02 13:19:33.323675068    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap

2024-02-02 13:19:33.323679458      self.run()

2024-02-02 13:19:33.323697860    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run

2024-02-02 13:19:33.323701737      self._target(*self._args, **self._kwargs)

2024-02-02 13:19:33.323705253    File "/opt/frigate/frigate/video.py", line 436, in track_camera

2024-02-02 13:19:33.323717252      process_frames(

2024-02-02 13:19:33.323721845    File "/opt/frigate/frigate/video.py", line 689, in process_frames

2024-02-02 13:19:33.323725938      detect(

2024-02-02 13:19:33.323745420    File "/opt/frigate/frigate/video.py", line 474, in detect

2024-02-02 13:19:33.323749320      region_detections = object_detector.detect(tensor_input)

2024-02-02 13:19:33.323753096    File "/opt/frigate/frigate/object_detection.py", line 225, in detect

2024-02-02 13:19:33.323756653      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))

2024-02-02 13:19:33.323766969  KeyError: -17

2024-02-02 13:19:34.699227649  [2024-02-02 13:19:34] watchdog.Yard                  ERROR   : Ffmpeg process crashed unexpectedly for Yard.

2024-02-02 13:19:34.699657399  [2024-02-02 13:19:34] watchdog.Yard                  ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

2024-02-02 13:19:34.699686703  [2024-02-02 13:19:34] ffmpeg.Yard.detect             ERROR   : Finishing stream 0:0 without any data written to it.

2024-02-02 13:19:34.699919630  [2024-02-02 13:19:34] ffmpeg.Yard.detect             ERROR   : Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

2024-02-02 13:19:34.837628972  [2024-02-02 13:19:34] watchdog.Yard2                 ERROR   : Ffmpeg process crashed unexpectedly for Yard2.

2024-02-02 13:19:34.837818328  [2024-02-02 13:19:34] watchdog.Yard2                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

2024-02-02 13:19:34.838121007  [2024-02-02 13:19:34] ffmpeg.Yard2.detect            ERROR   : Finishing stream 0:0 without any data written to it.

2024-02-02 13:19:34.838354054  [2024-02-02 13:19:34] ffmpeg.Yard2.detect            ERROR   : Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

2024-02-02 13:19:35.549276048  [2024-02-02 13:19:35] watchdog.Office_Parking        ERROR   : Ffmpeg process crashed unexpectedly for Office_Parking.

2024-02-02 13:19:35.549578980  [2024-02-02 13:19:35] watchdog.Office_Parking        ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

2024-02-02 13:19:35.549804498  [2024-02-02 13:19:35] ffmpeg.Office_Parking.detect   ERROR   : Finishing stream 0:0 without any data written to it.

2024-02-02 13:19:35.550014076  [2024-02-02 13:19:35] ffmpeg.Office_Parking.detect   ERROR   : Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

2024-02-02 13:19:35.644471559  [2024-02-02 13:19:35] watchdog.Warehouse_Rear        ERROR   : Ffmpeg process crashed unexpectedly for Warehouse_Rear.

2024-02-02 13:19:35.644915311  [2024-02-02 13:19:35] watchdog.Warehouse_Rear        ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

2024-02-02 13:19:35.644968311  [2024-02-02 13:19:35] ffmpeg.Warehouse_Rear.detect   ERROR   : Finishing stream 0:0 without any data written to it.

2024-02-02 13:19:35.645236632  [2024-02-02 13:19:35] ffmpeg.Warehouse_Rear.detect   ERROR   : Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

2024-02-02 13:20:07.601158286  192.168.88.5 - - [02/Feb/2024:13:20:07 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:19.561318884  192.168.88.5 - - [02/Feb/2024:13:20:19 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.020230572  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /ws HTTP/1.1" 101 56317 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.034462698  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET / HTTP/1.1" 200 6637 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.262091985  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.403988482  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Office_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.407724606  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Office_Parking/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.410811875  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Storage_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.415073907  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Storage_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.419088274  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Warehouse/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.422916750  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.427977146  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Yard/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:21.434565452  192.168.88.5 - - [02/Feb/2024:13:20:21 -0500] "GET /api/Yard2/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.412667798  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Office_Parking/latest.jpg?h=492 HTTP/1.1" 200 49609 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.413071012  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Office_Front/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.415082782  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Storage_Front/latest.jpg?h=492 HTTP/1.1" 200 37460 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.425507577  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Storage_Rear/latest.jpg?h=492 HTTP/1.1" 200 70135 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.426317909  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Warehouse/latest.jpg?h=492 HTTP/1.1" 200 66786 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.429446710  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=492 HTTP/1.1" 200 66578 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:23.863779457  192.168.88.5 - - [02/Feb/2024:13:20:23 -0500] "GET /api/Yard/latest.jpg?h=492 HTTP/1.1" 200 63692 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:24.137678222  192.168.88.5 - - [02/Feb/2024:13:20:24 -0500] "GET /api/Yard2/latest.jpg?h=492 HTTP/1.1" 200 43306 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:36.111422944  192.168.88.5 - - [02/Feb/2024:13:20:36 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.034774631  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET / HTTP/1.1" 200 6637 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.036277268  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /ws HTTP/1.1" 101 4 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.798293063  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.953732020  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Office_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.954728576  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Office_Parking/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.955639235  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Storage_Front/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.957536397  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Storage_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.980291523  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Office_Front/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.983117914  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Office_Parking/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:37.998627058  192.168.88.5 - - [02/Feb/2024:13:20:37 -0500] "GET /api/Storage_Front/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.001120931  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Storage_Rear/latest.jpg?h=12 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.006090850  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Warehouse/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.008392965  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.012684311  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Yard/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.016431995  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Yard2/latest.jpg?h=100 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.031000254  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Office_Front/latest.jpg?h=486 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.059212011  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Office_Parking/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.060400145  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Storage_Front/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.061018411  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Storage_Rear/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.061988478  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Warehouse/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.063376157  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:38.063451233  192.168.88.5 - - [02/Feb/2024:13:20:38 -0500] "GET /api/Yard/latest.jpg?h=498 HTTP/1.1" 499 0 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:40.206479919  [2024-02-02 13:20:40] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for Office_Front. Keeping the 6 most recent segments out of 7 and discarding the rest...

2024-02-02 13:20:43.971729831  192.168.88.5 - - [02/Feb/2024:13:20:43 -0500] "GET /api/Office_Front/latest.jpg?h=492 HTTP/1.1" 200 15258 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:43.974477175  192.168.88.5 - - [02/Feb/2024:13:20:43 -0500] "GET /api/Storage_Front/latest.jpg?h=492 HTTP/1.1" 200 37993 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:43.979999212  192.168.88.5 - - [02/Feb/2024:13:20:43 -0500] "GET /api/Office_Parking/latest.jpg?h=492 HTTP/1.1" 200 49863 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:43.982512976  192.168.88.5 - - [02/Feb/2024:13:20:43 -0500] "GET /api/Storage_Rear/latest.jpg?h=492 HTTP/1.1" 200 70345 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:43.984587072  192.168.88.5 - - [02/Feb/2024:13:20:43 -0500] "GET /api/Warehouse/latest.jpg?h=492 HTTP/1.1" 200 67158 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:43.987367448  192.168.88.5 - - [02/Feb/2024:13:20:43 -0500] "GET /api/Warehouse_Rear/latest.jpg?h=492 HTTP/1.1" 200 66866 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:44.526363173  192.168.88.5 - - [02/Feb/2024:13:20:44 -0500] "GET /api/Yard/latest.jpg?h=492 HTTP/1.1" 200 63625 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:44.565592536  192.168.88.5 - - [02/Feb/2024:13:20:44 -0500] "GET /api/Yard2/latest.jpg?h=492 HTTP/1.1" 200 43644 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:20:50.205869778  [2024-02-02 13:20:50] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for Office_Front. Keeping the 6 most recent segments out of 7 and discarding the rest...

2024-02-02 13:21:00.218605881  [2024-02-02 13:21:00] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for Office_Front. Keeping the 6 most recent segments out of 7 and discarding the rest...

2024-02-02 13:21:10.215574963  [2024-02-02 13:21:10] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for Office_Front. Keeping the 6 most recent segments out of 7 and discarding the rest...

2024-02-02 13:21:14.625648278  192.168.88.5 - - [02/Feb/2024:13:21:14 -0500] "GET /api/config HTTP/1.1" 200 43099 "http://192.168.88.7:5000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"

2024-02-02 13:21:16.460392391  192.168.88.5 - - [02/Feb/2024:13:21:16 -0500] "GET /api/config/raw HTTP/1.1" 200 4220 "http://192.168.88.7:5000/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0" "-"
NickM-27 commented 8 months ago

looks like tensorrt is not setup correctly

2024-02-02 13:19:33.320871804  Process camera_processor:Office_Front:

2024-02-02 13:19:33.323633275  Traceback (most recent call last):

2024-02-02 13:19:33.323675068    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap

2024-02-02 13:19:33.323679458      self.run()

2024-02-02 13:19:33.323697860    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run

2024-02-02 13:19:33.323701737      self._target(*self._args, **self._kwargs)

2024-02-02 13:19:33.323705253    File "/opt/frigate/frigate/video.py", line 436, in track_camera

2024-02-02 13:19:33.323717252      process_frames(

2024-02-02 13:19:33.323721845    File "/opt/frigate/frigate/video.py", line 689, in process_frames

2024-02-02 13:19:33.323725938      detect(

2024-02-02 13:19:33.323745420    File "/opt/frigate/frigate/video.py", line 474, in detect

2024-02-02 13:19:33.323749320      region_detections = object_detector.detect(tensor_input)

2024-02-02 13:19:33.323753096    File "/opt/frigate/frigate/object_detection.py", line 225, in detect

2024-02-02 13:19:33.323756653      (self.labels[int(d[0])], float(d[1]), (d[2], d[3], d[4], d[5]))

2024-02-02 13:19:33.323766969  KeyError: -17

this usually happens when the nvidia driver is not updated to 535

tkffaul commented 8 months ago

image it is 535

NickM-27 commented 8 months ago

I see you have a quadro, I think the versioning might be different for that. @NateMeyer can you confirm?

NickM-27 commented 8 months ago

@tkffaul can you show the detector and model sections of your config

tkffaul commented 8 months ago

i use a quadro because of the encode/decode limitations on the consumer side.

tkffaul commented 8 months ago

detectors: tensorrt: type: tensorrt device: 0 model: path: /config/model_cache/tensorrt/8.5.3/yolov7-tiny-416.trt input_tensor: nchw input_pixel_format: rgb width: 416 height: 416

Sorry I'm trying to fix the code spacing but i think im making it worse.

tkffaul commented 8 months ago

it did do a fresh compile of the model at install, just an FYI

tkffaul commented 8 months ago

I see there is a newer driver, do you want me to upgrade? 535.154.05... I'm also running this same setup in another spot with a P4000 with the 535.129.03 driver without issue. So i assumed it wasn't a driver issue.

NickM-27 commented 8 months ago

interesting, I am not sure why there would be a difference

tkffaul commented 8 months ago

Thats it! I updated the driver and it looks rock solid now! Hope this helps someone else! I'll just push them all up since this looks good!

tkffaul commented 8 months ago

I was wrong... dang... it looked so good. It just went a little longer.

hargrove4570 commented 8 months ago

I've been tracking these same errors with the opt/frigate/frigate/object_detection.py", line 225, I wonder if it is very sensitive to the latest driver for older nvidia, most running compute 5.0. Adding another data point before opening my own ticket.

hargrove4570 commented 8 months ago

Hey can you check you Nvidia container toolkit version with nvidia-ctk --version

Mine is 1.13.5 and I get a similar error wi th compute 5.0 and driver 535.54.03 on a GTX 750 ti. Given the refactor of tensorrt I wonder if this is actually the culprit that is seemingly fixed by going to the latest driver

Update: Runtime updated 1.14.4, no change so that's not it

Updated: reran with yolov4-tiny-416 and 0.13.1 works. So this specific error might be with the new tricks that yolo7 has that isn't agreeing with legacy cards (ie Maxwell)

Updated again: nope it only works with v4-tiny-416 and v4-tiny-288. Modified are loading but similar key error. Probably hardware limitation although at most it used 1 GB of memory across all the models I tested.

Last update before separate ticket: I tried rolling my old models from onnx directly in tensorrt 23.03 but am getting other model errors, not surprising since I was just winging it and playing around. But I'm eventually losing streams and inference is bonkers with the " same" model - getting 800+% confidence on random nothing (including a textured wall one camera is point at).

nordeep commented 8 months ago

Came here with the same issue: KeyError: -17 On the Frigate 0.12 I used tensor model yolov7-tiny-416 on 0.13.1 Frigate can't start with this model. nvidia-smi output:

Sun Feb  4 17:39:54 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 750 Ti      On  | 00000000:01:00.0 Off |                  N/A |
| 42%   47C    P0               2W /  38W |   1110MiB /  2048MiB |     15%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    195175      C   frigate.detector.cuda                       104MiB |
|    0   N/A  N/A    195204      C   ffmpeg                                       71MiB |
|    0   N/A  N/A    195214      C   ffmpeg                                      104MiB |
|    0   N/A  N/A    195221      C   ffmpeg                                       39MiB |
|    0   N/A  N/A    197601      C   uwsgi                                       783MiB |
+---------------------------------------------------------------------------------------+

nvidia-ctk --version output:

NVIDIA Container Toolkit CLI version 1.14.4
commit: d167812ce3a55ec04ae2582eff1654ec812f42e1

For now managed to run without error model yolov4-tiny-416 but it doesn't detect any persons.

hargrove4570 commented 8 months ago

@nordeep yes my detection is completely whack even though before it was perfect. When I have time I'm going to find the old docker release that was actually good so that rolling back permanently is an option.

I'm guessing there was some major changes in the new tensorrt that doesn't support this card even though it meets the driver and compute version. When I run the container separately it gives an ERROR that GTX 750 Ti is detected but is not supported. Right now I'm waiting for my streams to stop like they did last night.

nordeep commented 8 months ago

@hargrove4570 for myself i decided to go back to 0.12.1 version which work perfectly or me.

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt
nordeep commented 8 months ago

As I can see the changes haven't been significant from 0.12.1 at tensorrt docker container - https://github.com/blakeblackshear/frigate/tree/dev/docker/tensorrt

hargrove4570 commented 8 months ago

Did you just delete frigate.db or have to do anything else to roll back? I'd did it late last night and noticed a memory error which is why I tried to get 0.13.1 going again.

nordeep commented 8 months ago

@hargrove4570 I just gave up on the record archive. It was fine to me. frigate.db was re-created.

tkffaul commented 8 months ago

I also rolled back to 12.1, but i ordered a p400 to test on this. I think you guys are on to something with the compute 5.0, I just don't know enough about cuda to be useful trouble shooting.

tkffaul commented 8 months ago

So same install, same driver, just pulled the K2200 and put in a P400. Works perfect, there must be something with the older 5.0 cards.

NickM-27 commented 8 months ago

Thanks for updating, that is good info.

qubex22 commented 8 months ago

So same install, same driver, just pulled the K2200 and put in a P400. Works perfect, there must be something with the older 5.0 cards.

Had the same problem with a K620, just bought a P600 and works well too

KastB commented 8 months ago

Same issues for my GTX850M - is there any hope to get this fixed with frigate 0.13 or should I try downgrading as well? Is there any info we could provide / anything to test to help sorting this out?

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.