Open vchrizz opened 7 months ago
In your config you have both darknet and edgetpu object detector for camera_2, which can lead to unexpected results. Only one will be setup successfully (the other should show an error in the log on restart)
I suggest removing one of them and see if the issue persists
Thank you for the suggestion, I removed darknet entirely and restarted viseron. After about 6-7 hours the issue happened again, log:
ESC[80DESC[1AESC[KESC[36m[2023-11-09 14:35:52] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: [], message repeated 4 timesESC[0m
ESC[80DESC[1AESC[KESC[36m[2023-11-09 14:35:52] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: [], message repeated 5 timesESC[0m
ESC[32m[2023-11-09 14:35:52] [INFO ] [viseron.components.nvr.nvr.camera_2] - Stopping recording in: 1ESC[0m
ESC[36m[2023-11-09 14:35:52] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: []ESC[0m
ESC[80DESC[1AESC[KESC[36m[2023-11-09 14:35:52] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: [], message repeated 2 timesESC[0m
ESC[80DESC[1AESC[KESC[36m[2023-11-09 14:35:52] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: [], message repeated 3 timesESC[0m
ESC[80DESC[1AESC[KESC[36m[2023-11-09 14:35:53] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: [], message repeated 4 timesESC[0m
ESC[80DESC[1AESC[KESC[36m[2023-11-09 14:35:53] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: [], message repeated 5 timesESC[0m
ESC[32m[2023-11-09 14:35:53] [INFO ] [viseron.components.nvr.nvr.camera_2] - Stopping recording in: 0ESC[0m
ESC[32m[2023-11-09 14:35:53] [INFO ] [viseron.components.ffmpeg.recorder.camera_2] - Stopping recorderESC[0m
ESC[36m[2023-11-09 14:35:53] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Pausing segment cleanupESC[0m
ESC[36m[2023-11-09 14:35:53] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Concatenating segmentsESC[0m
ESC[36m[2023-11-09 14:35:53] [DEBUG ] [viseron.components.edgetpu.object_detector.camera_2] - Objects: []ESC[0m
ESC[36m[2023-11-09 14:35:53] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Not recording, pausing object detectorESC[0m
ESC[36m[2023-11-09 14:35:54] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - /segments/camera_2/20231109143550.mp4 is locked. Trying again in 1 secondESC[0m
ESC[36m[2023-11-09 14:35:55] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Segment information: {'20231109143540.mp4': {'start_time': 1699536940.0, 'end_time': 1699536945.038}, '20231109143535.mp4': {'start_time': 1699536
935.0, 'end_time': 1699536939.998}, '20231109143530.mp4': {'start_time': 1699536930.0, 'end_time': 1699536935.015}, '20231109143545.mp4': {'start_time': 1699536945.0, 'end_time': 1699536949.987}, '20231109143550.mp4': {'start_time': 1
699536950.0, 'end_time': 1699536954.998}, '20231109143525.mp4': {'start_time': 1699536925.0, 'end_time': 1699536929.999}, '20231109143520.mp4': {'start_time': 1699536920.0, 'end_time': 1699536924.979}}ESC[0m
ESC[36m[2023-11-09 14:35:55] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Start event: 1699536923.671719, segment: 20231109143520.mp4ESC[0m
ESC[36m[2023-11-09 14:35:55] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - End event: 1699536953.341363, segment: 20231109143550.mp4ESC[0m
ESC[36m[2023-11-09 14:35:55] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Concatenation command: ffmpeg -hide_banner -loglevel error -y -protocol_whitelist file,pipe -f concat -safe 0 -i - -c:v copy -c:a copy -movflags +
faststart /recordings/camera_2/2023-11-09/14:35:28.mp4ESC[0m
ESC[36m[2023-11-09 14:35:55] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Segment script:
file 'file:/segments/camera_2/20231109143520.mp4'
inpoint 3
file 'file:/segments/camera_2/20231109143525.mp4'
file 'file:/segments/camera_2/20231109143530.mp4'
file 'file:/segments/camera_2/20231109143535.mp4'
file 'file:/segments/camera_2/20231109143540.mp4'
file 'file:/segments/camera_2/20231109143545.mp4'
file 'file:/segments/camera_2/20231109143550.mp4'
outpoint 3ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Removing segment: 20231109143520.mp4ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Removing segment: 20231109143525.mp4ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Removing segment: 20231109143530.mp4ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Removing segment: 20231109143535.mp4ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Removing segment: 20231109143540.mp4ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Removing segment: 20231109143545.mp4ESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Segments concatenatedESC[0m
ESC[36m[2023-11-09 14:35:56] [DEBUG ] [viseron.components.ffmpeg.recorder.camera_2] - Resuming segment cleanupESC[0m
ESC[36m[2023-11-09 14:49:25] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Starting object detectorESC[0m
ESC[31m[2023-11-09 14:49:28] [ERROR ] [viseron.components.nvr.nvr.camera_2] - Failed to retrieve result for object_detectorESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.8061981201171875 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.610076427459717 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.403458833694458 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.211101531982422 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.011807918548584 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.813913345336914 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.612987756729126 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.4091050624847412 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.2112698554992676 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:28] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.011817216873169 seconds old. DiscardingESC[0m
ESC[31m[2023-11-09 14:49:31] [ERROR ] [viseron.components.nvr.nvr.camera_2] - Failed to retrieve result for object_detectorESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.6164591312408447 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.4165172576904297 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.2226879596710205 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.0208640098571777 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.8228445053100586 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.6213109493255615 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.4028966426849365 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.220019578933716 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 2.02463436126709 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.8243985176086426 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.62497878074646 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.3848557472229004 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.2250900268554688 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:31] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 1.0271141529083252 seconds old. DiscardingESC[0m
ESC[31m[2023-11-09 14:49:34] [ERROR ] [viseron.components.nvr.nvr.camera_2] - Failed to retrieve result for object_detectorESC[0m
ESC[36m[2023-11-09 14:49:34] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.6340036392211914 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:34] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.4118499755859375 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:34] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.2345492839813232 seconds old. DiscardingESC[0m
ESC[36m[2023-11-09 14:49:34] [DEBUG ] [viseron.components.nvr.nvr.camera_2] - Frame is 3.0347182750701904 seconds old. DiscardingESC[0m
...
Thanks, seems the EdgeTPU stops functioning for some reason, could be a power issue perhaps.
There is another open issue on a similar case, cant find it right now tho. The solution could be for Viseron to reload the detector automatically when it fails the retrieve results. Am quite busy atm tho so it will take me some time
Thank you for feedback. The EdgeTPU is connected via USB directly to the host (without any hub). As far as I can tell, the EdgeTPU works fine in other applications (when viseron is not running), so I would rule out hardware issues. Is there anything I could debug to find out more details?
Hmm, not that i can think of.
Maybe if you could catch the dmesg
logs when the problem appears it could give some hints
Hello,
after some time running viseron fine and everything seems to work nicely I noticed that I have to restart it every few days because it stops detecting (and recording on object detection).
Mostly the issue happens after about 2-3 days after (re)start of viseron. I tried to log everything to find the cause but am unsuccessful to find an obvious problem.
I thought, maybe it is because the connection to the camera was interrupted, but no, even if I interrupt the connection to the camera on purpose (after a restart of viseron), the connection is re-established again, so this seems not to be the issue.
What could be the reason, that it suddently stops detecting objects and seemingly runs in that loop not detecting objects anymore?
current configuration (commented lines removed):
log with last successfull recording - after that, the log shows only those failed and old frame messages for days, no matter if there is movement or not: