search

hjdhjd / homebridge-unifi-protect

:video_camera: Complete HomeKit integration for all UniFi Protect device types with full support for most features including HomeKit Secure Video, and more. https://homebridge.io
Other
1.39k stars 84 forks source link

Max CPU usage and 32 Protect API errors per second #1073

Closed verisgit closed 1 month ago

verisgit commented 1 month ago

Homebridge UniFi Protect Version

6.22.0

Homebridge Platform and OS

RPI4B Debian GNU/Linux Bullseye (11)

Homebridge Version

1.8.1

Node Version

18.16.0

UniFi OS Version

3.1.16

UniFi Protect Controller Version

3.0.26

Describe the problem

I have had Unifi Protect running happily for over a year on this device. I've been regularly patching it and keeping homebridge and the plugin up to date. Recently, I have noticed that after a number of days, I start seeing repeating errors in Homekit: "Camera online, Camera offline, Camera online" If I check Homebridge, the CPU is at 100%, temp is high, and I always see screenfuls of these errors, sometimes at a rate of up to 32 per second: [20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.

As soon as I restart it, the temps drop to normal, the CPU goes back to normal, and everything works fine - the messages don't appear for a few days, but then they come back.

I only have 2 cameras in Protect. 1 home bridge ffmpeg camera and an air conditioner integration, so it's very low in terms of whats installed in configured, until it recently started happening I've not had any issues with the CPU usage or these kinds of issues.

Any ideas? It looks as if it's smashing the protect APIs as fast as it can, and the pause isn't working.

Homebridge HBUP JSON configuration

{
    "controllers": [
        {
            "address": "******",
            "password": "******",
            "username": "******"
        }
    ],
    "name": "UniFi Protect",
    "options": [
        "Enable.Motion.OccupancySensor.******",
        "Enable.Motion.OccupancySensor.Duration.******.300",
        "Enable.Motion.OccupancySensor.******",
        "Enable.Motion.OccupancySensor.Vehicle.******",
        "Enable.Audio.Filter.Noise.******",
        "Enable.Audio.Filter.Noise.FftNr.******.90",
        "Enable.Audio.Filter.Noise.HighPass.******.200",
        "Enable.Audio.Filter.Noise.LowPass.******.1000",
        "Enable.Log.Motion.******"
    ],
    "platform": "UniFi Protect",
    "_bridge": {
        "username": "******",
        "port": ******
    }
}

Relevant log output

[20/05/2024, 21:45:35] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:35] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:35] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:35] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:35] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:35] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:38] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:45:38] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:35] [UniFi Protect] 192.168.1.1: API error: Resuming connectivity to the UniFi Protect API after pausing for 5 minutes.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Throttling API calls due to errors with the 10 previous attempts. Pausing communication with the Protect controller for 5 minutes.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:40] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:41] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:43] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:43] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[20/05/2024, 21:50:46] [UniFi Protect] 192.168.1.1: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.

Acknowledgment that you are only running UniFi OS and UniFi Protect releases from the Ubiquiti Official release channel

hjdhjd commented 1 month ago

Thanks for the report. The errors are telling you that your Protect controller's struggling. This can be due to a number of reasons:

  1. Protect firmware versions...some are more bug-free than others.
  2. Network connection issues between Homebridge and the Protect controller. If you're using WiFi, and consuming a lot of bandwidth or are on a weaker signal, you can run into this.

I understand why you believe the throttling isn't working, but do keep in mind that it only logs errors not every request made to the controller. The moment a request succeeds, the API throttle mechanism will reset for another 10 attempts.

Try rebooting your Protect controller device (UDMP, UCK, etc) and see if that does the trick. You're also running an old version of UniFi OS...I would recommend you remain on the latest GA releases of both Protect and UniFi OS.

TL;DR:

Let me know if those help. Best of luck.

verisgit commented 1 month ago

Hi there, thanks for the detailed reply.

-The RPI is connected by GbE to the controller. No wifi involved. -I've had a look at the CPU of the UDMPro, and it's around 10-25%. I'll check it again when the issues are next occurring and see if there's any random spike. -I did notice the UDMConsole version is slightly older, but looking at the release notes for the latest version it appears to be unstable. Nonetheless, I'll try updating it and see how I go.

hjdhjd commented 1 month ago

@Altivec-Dan One person, one report please. I will not provide support to multiple people on a single issue. Each environment is unique. Happy to provide you support, but please create a new issue. Unfortunately, there's no mechanism to move your issue to a new one in GH, so I've deleted your comment. Thanks for respecting the way I choose to manage my support queue. 😄

hjdhjd commented 1 month ago

Hi there, thanks for the detailed reply.

-The RPI is connected by GbE to the controller. No wifi involved. -I've had a look at the CPU of the UDMPro, and it's around 10-25%. I'll check it again when the issues are next occurring and see if there's any random spike. -I did notice the UDMConsole version is slightly older, but looking at the release notes for the latest version it appears to be unstable. Nonetheless, I'll try updating it and see how I go.

@verisgit If you have a followup, would love to hear how the update went so we can continue to troubleshoot this.

verisgit commented 1 month ago

I updated the UDM and the problem still exists.

verisgit commented 1 month ago

Some further observations from today: -Seems to be happening even more frequently now I've updated the UDM -today when the issue started I checked Homebridge and it is at 99% CPU, the UDM is at it's normal CPU of around 20% -Hombride logs had miles of that previous error I posted. -Restarted homebridge it and all the logs and functionality was perfect again. All streams work fine, logs look as expected no errors. After I did that I then thought I should have shutdown just the HBUP child bridge to see if it was responsible for the CPU issue. I'll try that next time rather than restart the whole thing - just to rule out one of the others had gone rogue.

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

verisgit commented 1 month ago

Any further info you need from me? Not sure what else I can check.

Altivec-Dan commented 1 month ago

Still seeing some errors in the logs with most features disabled (except for forcing high quality), without that Home likes to give a low res stream (as you mentioned). I think I can live with this for now, so you can close this out. My Protect updated to 4.x today, so new things to watch out for. Appreciate your help on this.

hjdhjd commented 1 month ago

Any further info you need from me? Not sure what else I can check.

Any further info you need from me? Not sure what else I can check.

Out of curiosity, try to update to the latest Protect controller firmware now that it’s GA. Thanks.

hjdhjd commented 1 month ago

Still seeing some errors in the logs with most features disabled (except for forcing high quality), without that Home likes to give a low res stream (as you mentioned). I think I can live with this for now, so you can close this out. My Protect updated to 4.x today, so new things to watch out for. Appreciate your help on this.

Still seeing some errors in the logs with most features disabled (except for forcing high quality), without that Home likes to give a low res stream (as you mentioned). I think I can live with this for now, so you can close this out. My Protect updated to 4.x today, so new things to watch out for. Appreciate your help on this.

Wrong thread! 😄

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 month ago

This issue is locked to prevent necroposting on closed issues. Please create a new issue for related support requests, bug reports, or feature suggestions.