Closed brenninger closed 2 years ago
I've observed similar issue just yesterday. My "server" become very slow. Today I've found that some events last from a few minutes and up to several hours.
I've upgraded to the recent release about a week ago and converted all my cameras settings to the new settings (events - active_objects) and didn't enable stationary objects detections. I do not expect to have events more than 10-15 seconds but see some which lasts much longer. E.g. one event was triggered by a moving car and started at 10:07:36 and finished at 17:18:18 (I had to restart frigate in the evening and don't remember when). Another example started at 22:31:50 finished at 09:38:43.
I'm working on an automation (via a custom script) which pulls events as well as I've other automation via HA, which sends events/videos via telegram. So I suspect these long events along with my automations caused the high CPU utilization.
Did you observe "long" events?
@brenninger your cameras are continuously losing connection causing the ffmpeg process to restart. Are they setup via WiFi and if so is it on the 2.4 or 5 GHz network?
I've observed similar issue just yesterday. My "server" become very slow. Today I've found that some events last from a few minutes and up to several hours.
I've upgraded to the recent release about a week ago and converted all my cameras settings to the new settings (events - active_objects) and didn't enable stationary objects detections. I do not expect to have events more than 10-15 seconds but see some which lasts much longer. E.g. one event was triggered by a moving car and started at 10:07:36 and finished at 17:18:18 (I had to restart frigate in the evening and don't remember when). Another example started at 22:31:50 finished at 09:38:43.
I'm working on an automation (via a custom script) which pulls events as well as I've other automation via HA, which sends events/videos via telegram. So I suspect these long events along with my automations caused the high CPU utilization.
Did you observe "long" events?
Your case is very different from the OPs as you're not reporting ffmpeg crashes. You should keep a few things in mind:
Either way I'd recommend creating your own issues as it doesn't seem to be related to this one.
@brenninger your cameras are continuously losing connection causing the ffmpeg process to restart. Are they setup via WiFi and if so is it on the 2.4 or 5 GHz network?
@NickM-27 My cameras all wired connection into my ubiquiti network. I used VLC to watch the RTSP stream of my cameras and I was not seeing any connection losses, so I don't know why Frigate would be losing the connection.
Edit: I also am using Blue Iris as my main NVR as I only wanted Frigate to be used as my object detection. I also haven't seen any problems with BI losing connection. Does anyone know a way to grab the feed from BI and input that into Frigate to see if that fixes the issue?
@brenninger your cameras are continuously losing connection causing the ffmpeg process to restart. Are they setup via WiFi and if so is it on the 2.4 or 5 GHz network?
@NickM-27 My cameras all wired connection into my ubiquiti network. I used VLC to watch the RTSP stream of my cameras and I was not seeing any connection losses, so I don't know why Frigate would be losing the connection.
Edit: I also am using Blue Iris as my main NVR as I only wanted Frigate to be used as my object detection. I also haven't seen any problems with BI losing connection. Does anyone know a way to grab the feed from BI and input that into Frigate to see if that fixes the issue?
I know it's the case with wifi cameras, but I think it's probably the case with Ethernet connections as well, the camera likely has a max number of connections and seems to be refusing the frigate connection.
@brenninger your cameras are continuously losing connection causing the ffmpeg process to restart. Are they setup via WiFi and if so is it on the 2.4 or 5 GHz network?
@NickM-27 My cameras all wired connection into my ubiquiti network. I used VLC to watch the RTSP stream of my cameras and I was not seeing any connection losses, so I don't know why Frigate would be losing the connection. Edit: I also am using Blue Iris as my main NVR as I only wanted Frigate to be used as my object detection. I also haven't seen any problems with BI losing connection. Does anyone know a way to grab the feed from BI and input that into Frigate to see if that fixes the issue?
I know it's the case with wifi cameras, but I think it's probably the case with Ethernet connections as well, the camera likely has a max number of connections and seems to be refusing the frigate connection.
I'll try to restart the cameras again, even though I have tried it already. Any other suggestions on what to do with it?
If you have all the other services still connecting I don't think restarting will help. Is this something that happens continuously as soon as you restart?
It happens continuously. So I've been tinkering with it and removing items to see what happens. I removed all the input and output args and that seems to have fixed the issue of the high CPU usage. It also seems that the stream is at least running continuously now. However, I am now getting new errors.
[2022-04-03 11:17:03] ffmpeg.driveway.rtmp ERROR : [flv @ 0x558c07bffa80] Failed to update header with correct duration.
[2022-04-03 11:17:03] ffmpeg.driveway.rtmp ERROR : [flv @ 0x558c07bffa80] Failed to update header with correct filesize.
[2022-04-03 11:17:03] watchdog.driveway INFO : Terminating the existing ffmpeg process...
[2022-04-03 11:17:03] watchdog.driveway INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-03 11:17:34] frigate.video ERROR : front_door: Unable to read frames from ffmpeg process.
[2022-04-03 11:17:34] frigate.video ERROR : front_door: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:17:34] frigate.video ERROR : west: Unable to read frames from ffmpeg process.
[2022-04-03 11:17:34] frigate.video ERROR : west: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:17:34] frigate.video ERROR : driveway: Unable to read frames from ffmpeg process.
[2022-04-03 11:17:34] frigate.video ERROR : driveway: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:17:34] frigate.video ERROR : backyard: Unable to read frames from ffmpeg process.
[2022-04-03 11:17:34] frigate.video ERROR : backyard: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:17:39] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:49048]
[2022-04-03 11:17:43] watchdog.front_door ERROR : Ffmpeg process crashed unexpectedly for front_door.
[2022-04-03 11:17:43] watchdog.front_door ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:17:43] ffmpeg.front_door.detect ERROR : [flv @ 0x561a19f4da40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-04-03 11:17:43] ffmpeg.front_door.detect ERROR : [flv @ 0x561a19f4da40] Failed to update header with correct duration.
[2022-04-03 11:17:43] ffmpeg.front_door.detect ERROR : [flv @ 0x561a19f4da40] Failed to update header with correct filesize.
[2022-04-03 11:17:43] watchdog.west ERROR : Ffmpeg process crashed unexpectedly for west.
[2022-04-03 11:17:43] watchdog.west ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:17:43] ffmpeg.west.detect ERROR : [flv @ 0x55eea034ed40] Failed to update header with correct duration.
[2022-04-03 11:17:43] ffmpeg.west.detect ERROR : [flv @ 0x55eea034ed40] Failed to update header with correct filesize.
[2022-04-03 11:17:43] watchdog.driveway ERROR : Ffmpeg process crashed unexpectedly for driveway.
[2022-04-03 11:17:43] watchdog.backyard ERROR : Ffmpeg process crashed unexpectedly for backyard.
[2022-04-03 11:17:43] watchdog.backyard ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:17:43] ffmpeg.backyard.detect ERROR : [flv @ 0x55c11dfab500] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-04-03 11:17:43] ffmpeg.backyard.detect ERROR : [flv @ 0x55c11dfab500] Failed to update header with correct duration.
[2022-04-03 11:17:43] ffmpeg.backyard.detect ERROR : [flv @ 0x55c11dfab500] Failed to update header with correct filesize.
[2022-04-03 11:17:43] watchdog.driveway ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
Okay nice, that is progress. If you disable RTMP that should fix those errors.
Why is it that RTMP would cause the crashes?
RTMP is very sensitive to higher resolutions and also instability in camera streams. The flv
in the crash logs points to RTMP.
Ahh okay I see. I appreciate the help! So far so good, I'll leave the issue open a bit to make sure no more errors and then will close. Thank you again!
Not quite fixed yet. I'm getting this error now, however, it seems like Frigate is still working, just seems like its constantly restarting itself. Is this still related to the camera streams crashing or something new now.
[2022-04-03 11:57:14] watchdog.west INFO : Terminating the existing ffmpeg process...
[2022-04-03 11:57:14] ffmpeg.backyard.record ERROR : [segment @ 0x55eef187ef40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-04-03 11:57:14] watchdog.backyard INFO : Terminating the existing ffmpeg process...
[2022-04-03 11:57:14] watchdog.backyard INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-03 11:57:14] ffmpeg.front_door.record ERROR : [segment @ 0x55bbbf282e00] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-04-03 11:57:14] ffmpeg.front_door.record ERROR : [segment @ 0x55bbbf282e00] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[2022-04-03 11:57:14] watchdog.front_door INFO : Terminating the existing ffmpeg process...
[2022-04-03 11:57:14] watchdog.front_door INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-03 11:57:14] watchdog.driveway INFO : Terminating the existing ffmpeg process...
[2022-04-03 11:57:14] watchdog.driveway INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-03 11:57:14] watchdog.west INFO : Waiting for ffmpeg to exit gracefully...
[2022-04-03 11:57:44] frigate.video ERROR : backyard: Unable to read frames from ffmpeg process.
[2022-04-03 11:57:44] frigate.video ERROR : backyard: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:57:44] frigate.video ERROR : driveway: Unable to read frames from ffmpeg process.
[2022-04-03 11:57:44] frigate.video ERROR : driveway: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:57:46] frigate.video ERROR : west: Unable to read frames from ffmpeg process.
[2022-04-03 11:57:46] frigate.video ERROR : west: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:57:46] frigate.video ERROR : front_door: Unable to read frames from ffmpeg process.
[2022-04-03 11:57:46] frigate.video ERROR : front_door: ffmpeg process is not running. exiting capture thread...
[2022-04-03 11:57:54] watchdog.west ERROR : Ffmpeg process crashed unexpectedly for west.
[2022-04-03 11:57:54] watchdog.west ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:57:54] watchdog.front_door ERROR : Ffmpeg process crashed unexpectedly for front_door.
[2022-04-03 11:57:54] watchdog.front_door ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:57:54] watchdog.backyard ERROR : Ffmpeg process crashed unexpectedly for backyard.
[2022-04-03 11:57:54] watchdog.backyard ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:57:54] watchdog.driveway ERROR : Ffmpeg process crashed unexpectedly for driveway.
[2022-04-03 11:57:54] watchdog.driveway ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-04-03 11:58:34] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
[2022-04-03 11:58:34] root INFO : Waiting for detection process to exit gracefully...
[2022-04-03 11:59:04] root INFO : Detection process didnt exit. Force killing...
[2022-04-03 11:59:04] detector.coral_pci INFO : Starting detection process: 1284
[2022-04-03 11:59:04] frigate.edgetpu INFO : Attempting to load TPU as pci
[2022-04-03 11:59:04] frigate.edgetpu INFO : TPU found
If you are seeing Detection appears to be stuck. Restarting detection process
that means frigate is unable to establish a connection to your Coral. This is most often due to power or other hardware issues. Are you still seeing high cpu usage?
@blakeblackshear I am no longer seeing high CPU usage after I took out the arg parameters. I don't know of any issues with the Coral and losing connection. But I can try to look into it some more. Do you have any recommendations to check if there are issues with it?
These are the latest errors:
[2022-04-09 20:49:26] frigate.video ERROR : front_door: Unable to read frames from ffmpeg process. [2022-04-09 20:49:26] frigate.video ERROR : front_door: ffmpeg process is not running. exiting capture thread... [2022-04-09 20:49:28] frigate.video ERROR : west: Unable to read frames from ffmpeg process. [2022-04-09 20:49:28] frigate.video ERROR : west: ffmpeg process is not running. exiting capture thread... [2022-04-09 20:49:28] frigate.video ERROR : backyard: Unable to read frames from ffmpeg process. [2022-04-09 20:49:28] frigate.video ERROR : backyard: Unable to read frames from ffmpeg process. [2022-04-09 20:49:28] frigate.video ERROR : backyard: ffmpeg process is not running. exiting capture thread... [2022-04-09 20:49:28] frigate.video ERROR : driveway: Unable to read frames from ffmpeg process. [2022-04-09 20:49:28] frigate.video ERROR : driveway: ffmpeg process is not running. exiting capture thread... [2022-04-09 20:49:30] watchdog.driveway ERROR : Ffmpeg process crashed unexpectedly for driveway. [2022-04-09 20:49:30] watchdog.driveway ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Describe the problem you are having
Within the last couple of days, I started to getting endless errors in my frigate logs and my CPU usage has jumped to 100% and I'm not sure what has caused the problems.
This is basically causing my HA to shut down because it uses up all my CPU usage. I can't think of any updates that I have done that would have caused this issue. I am up to date with my HA version and up to date with my Frigate version. I have restarted my Amcrest cameras in hopes to restart the RTSP stream but can't figure out the issue. It appears that the issue is happening with all my cameras and not just one.
Does anyone have any solutions about what is causing my problems?
Version
0.10.1-83481af
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Debian
Install method
HassOS Addon
Coral version
M.2
Network connection
Wired
Camera make and model
Amcrest
Any other information that may be helpful
No response