Protect Controller taking too long to respond

Altivec-Dan commented 1 month ago

Homebridge UniFi Protect Version

6.22.0

Homebridge Platform and OS

MacOS Sonoma (17.5)

Homebridge Version

1.8.2

Node Version

20.13.1

UniFi OS Version

3.2.12

UniFi Protect Controller Version

3.0.26

Describe the problem

Log shows repetitive API errors, associated with Protect Controller taking too long to respond. Recently I've seen the CPU of the UDM Pro near 100%, causing lockups to basic functionality. Upon disabling the HBUP plugin, the CPU was returned back to normal. I re-enabled the plugin again recently, and haven't had any UDM Pro lockups, but output from the plugin, such as detecting camera motion to turn on a light have not been working reliably.

Homebridge HBUP JSON configuration

"name": "UniFi Protect",
            "options": [
                "Enable.Audio.Filter.Noise.E063DA805269",
                "Enable.Video.DynamicBitrate.E063DA805269",
                "Enable.Video.Transcode.Hardware.E063DA805269",
                "Disable.Doorbell.Messages.FromDoorbell.E063DA805269",
                "Disable.Doorbell.Messages.E063DA805269",
                "Enable.Video.HKSV.Record.Only.High.E063DA805269",
                "Enable.Video.Stream.Only.High.E063DA805269",
                "Enable.Nvr.Service.Playlist.E063DA805269.10110",
                "Enable.Motion.SmartDetect.E063DA805269",
                "Enable.Audio.Filter.Noise",
                "Enable.Doorbell.Trigger",
                "Disable.Doorbell.Trigger.E063DA805269"
            ],
            "_bridge": {
                "username": "0E:52:0E:DA:64:7B",
                "port": 42482
            },
            "platform": "UniFi Protect"

Relevant log output

[5/21/2024, 10:25:25 AM] [UniFi Protect] Front Porch - G4 Doorbell Pro [G4 Doorbell Pro]: Doorbell ring detected.
[Front Porch - G4 Doorbell Pro] The image snapshot handler for the given accessory is slow to respond! See https://homebridge.io/w/JtMGR for more info.
[5/21/2024, 10:49:11 AM] [UniFi Protect] Peacock Hill UDM Pro [UDM-PRO]: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[5/21/2024, 10:49:15 AM] [UniFi Protect] Peacock Hill UDM Pro [UDM-PRO]: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[5/21/2024, 10:49:15 AM] [UniFi Protect] Peacock Hill UDM Pro [UDM-PRO]: API error: Unable to retrieve the UniFi Protect controller configuration.
[5/22/2024, 12:49:09 AM] [UniFi Protect] Peacock Hill UDM Pro [UDM-PRO]: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[5/22/2024, 12:49:13 AM] [UniFi Protect] Peacock Hill UDM Pro [UDM-PRO]: API error: Protect controller is taking too long to respond to a request. This error can usually be safely ignored.
[5/22/2024, 12:49:13 AM] [UniFi Protect] Peacock Hill UDM Pro [UDM-PRO]: API error: Unable to retrieve the UniFi Protect controller configuration.

Acknowledgment that you are only running UniFi OS and UniFi Protect releases from the Ubiquiti Official release channel

[X] I acknowledge that I am not running any early access / beta firmwares or operating systems from either Ubiquiti or Apple.

hjdhjd commented 1 month ago

This started short and ended long...at least I tried to be comprehensive. 😄

Thanks for the report. Let's unpack a few things...

First - define "recently" please. Some of the more recent Protect controller firmwares have been notably buggy...it's a well-worn pattern unfortunately with Ubiquiti, especially when they've turned their attention to new stuff (which judging by the 4.0 series that is in EA they have).

While UDMP CPU utilization is certainly a data point...that's really a Ubiquiti thing, not a me thing...other than to note that high CPU utilization can certainly lead to slower performance characteristics including the Protect controller being slower to send out events. TL;DR: correlation isn't the same as causation - there's been no meaningful changes in HBUP or the underlying ways in which we use the Protect API in a very long time.

Lastly - motion events not working reliably...if you can provide me meaningful data and ideally a reproducible use case, I'm happy to look into it, but I haven't seen these issues myself, but it doesn't mean they aren't there.

Back to where it all began: I don't see anything here that I can address on the HBUP end of things on first look. If you're concerned about CPU utilization of the UDMP, you can do the following things to reduce the workload on the UDMP, and accept the performance and functional tradeoffs:

Disable the timeshift buffer if you're using HKSV. This will significantly reduce the workload on the UDMP since it won't be providing HBUP video on a constant basis.
Disable high quality snapshots. This will also further reduce the workload on the UDMP.

Can you give those a shot and see how things evolve? I suspect you're dealing with buggy Protect controller firmware, but I'm open (and would prefer it) to it being an HBUP-specific issue because I could triage and address it without waiting on Ubiquiti, but my instinct is that it's on their end.

Another bonus thing to try, if you'd like: try previous version of HBUP. Using the Homebridge webUI it's pretty trivial to install previous versions of HBUP and A/B test them to see if the problem can be pinpointed to a specific version.

Altivec-Dan commented 1 month ago

If I had to estimate, I'd say these issues in the logs go back ~2 months, which seems to conincide around Protect 3.0.x release. Regarding the motion event, I have a simple automation in HomeKit if garage camera detects motion, turn on ceiling lights. This is the most noticeable aspect of the issue over the last two months is I would have to wait 5 or so seconds of walking around would to trigger the motion and turn on the lights. Of course I tried to capture a log right now and response time was quicker. I would caveat this with I restarted home bridge an hour or so earlier, which often cleans up the API error messages for a bit. As for your recommendations, I am happy to try them. To be honest I'm not sure I understand how the time shift buffer is used (beneficial or detrimental), so that was part of what prompted me to highlight the issue. With all the development of your plugin, at a certain point it's hard to know what are the optimal settings. Appreciate the help, will keep you posted on impact of the changes.

hjdhjd commented 1 month ago

Read here. I’d also advise taking a longer read through the documentation.

Best of luck.

Altivec-Dan commented 1 month ago

I'll read through it again more thoroughly, but my first couple passes now the CPU requirements you refer to seem to be the Homebridge controller, what has the biggest impact on the Protect controller (UDM, etc.) might be beneficial to others as well. I was leaning towards enabling as much as possible due to a fairly powerful Mac Mini, but not really considering where it takes its toll on the Ubiquiti side of things.

hjdhjd commented 1 month ago

Several of the things you enabled outside of the defaults, I’m not sure you realize the implications of…they won’t affect performance, but they likely hinder different parts of the experience. All of my plugins, this one especially, the defaults exist for good reason and while the options are there for you to tailor to your heart’s content…I would not do so. Specifically my suggestions:

Stop forcing any stream type. Let HBUP do what it does. It is going to know what to use in your environment based on what HomeKit is asking it to do.
Do not use dynamic bitrates. It has serious negative quality implications to Protect.
Do not use any of the audio filters unless absolutely required. Ubiquiti’s filtering is pretty decent/aggressive already in the G4 series and beyond.
Unless you have specific automation use cases (which I presume you do not given you haven’t enabled the corresponding automation-centric capabilities) I’m not sure why you’re enabling smart motion detection if you’re using HKSV. These capabilities are mutually exclusive unless you have specific automation use cases.

Just because an option or button exists, doesn’t mean you should press it. 😄

I have a Mac as well and well north of 20 cameras that have significant motion activity every day. It’s a terrific platform. HBUP is asking the Protect controller to constantly have multiple streams open to HBUP. It’s a definitive load…not an unbearable one, but a load. When there are bugs in Ubiquiti Protect firmware release, we tend to feel them more acutely in the HBUP world because we are an edge case for them…most client devices aren’t maintaining a constant connection (ViewPort being a notable exception). I personally wouldn’t run a significant Protect footprint on a UDMP. It’s my home router…I want it doing one thing and that’s it. But it’s fine for small Protect footprints.

Again - if you suspect it’s an HBUP-specific issue, please rollback HBUP versions and test. There haven’t been too many releases in the past few months.

Altivec-Dan commented 1 month ago

Appreciate the comments. I've been running your plugin almost as long as it's been available, and many of the new features you add seem beneficial, but agree over time its easy to lose track of what the defaults are vs what's been enabled in the past. I'll back out many of the items you suggested and see if I can tell a difference and monitor. Again, my post was not with the intent to blame the plugin, but to understand how I could be using it in a way which is detrimental. I'm probably not the only one, as I have a friend who was sharing similar issues and neither of us could figure out where we went wrong. Thanks again

hjdhjd commented 1 month ago

My apologies - I didn’t mean to imply that you were saying that, not that there’s anything wrong with it if there were an issue in HBUP itself.

If anything, I’d prefer that the issue be in HBUP because that’s something I have control over (and a new fun puzzle to solve in Ubiquiti-land!).

I was trying to (apparently inartfully) say that if there’s a use case you can recreate, great let’s work on it. If not…there’s not much that’s inherently changed in HBUP aside from some new capabilities, the most meaningful recently being high-quality snapshots to overcome new constraints introduced by Ubiquiti.

Re: dynamic bitrates…here’s why it’s bad: it varies the quality at the source, the Protect controller. That means if you are recording Protect video on your UDMP, it’s quality is now going to vary based on what HomeKit requests/demands - and HomeKit typically requests some pretty low bitrates (a G4 Pro, for instance, can kick out 16,000kbps….HomeKit will, in typical scenarios, request no more than 800kbps…and 1000kbps is about the most I’ve ever seen be requested). That’s quite the stark quality differential. So why does the feature exist? For people with particularly low-power systems that can’t handle a high-bitrate stream when transcoding and get good performance. You are definitely not in that camp. 😄 If you do disable dynamic bitrates…I would then go back to your cameras on the Protect controller and ensure you have the image quality set to high (or whatever you want it to be).

Similar story with forcing a certain recording or viewing quality…they largely exist to deal with constrained environments, or environments where you really want to enforce a specific quality level. Virtually nobody should need to do so unless you’re running on constrained hardware (e.g. RPi).

Smart motion detection - you’re basically telling HBUP to (potentially) use two different mechanisms to relay motion events. Smart motion detection = the Protect controller decides when a motion event is interesting and delivers it to you. In the pre-HKSV world, that was a good thing for those who didn’t want a random leaf blowing across the camera to trigger a motion event. In the post-HKSV world, this option doesn’t make sense if you enable HKSV on your cameras in the Home app…because HomeKit will decide when an event is interesting (animal, vehicle, package, person) and inform you about it. Given you’re using this plugin, my bias is always toward adopting HomeKit-centric features…so smart motion detection isn’t enabled because its functionality directly overlaps with HKSV. In fact, if you enable it and you’re using HKSV, the HBUP logs will warn you that you are doing something you probably don’t want to be doing. The only use cases where it makes sense to use this (which you haven’t enabled given your feature options) is where you want to automate something like say “when you see a person on this camera turn on a light”…Apple doesn’t allow HomeKit to inform us directly that it’s seen a person, etc…it’s all a black box. The only way to do that kind of automation is to rely on the Protect controller telling us that the Protect controller thinks it sees a person, and act accordingly.

Noise filtering…it’s a tradeoff. Protect is pretty good at it these days, especially on newer (G4+) hardware. Enabling this will add a little time delay in your audio and your audio/video won’t appear to be in sync. It’s a tradeoff in performance/responsiveness/capability. Enable it if you need it, but really most people don’t is my experience, especially on newer stuff.

I, in my desire for brevity and my natural direct nature, didn’t want to write all that…because it’s a lot. And I’ve had to write all of it a bunch of times in the past that you can find if you search previous issues. I apologize again if I came off as something less than gracious.

github-actions[bot] commented 1 month ago

This issue is locked to prevent necroposting on closed issues. Please create a new issue for related support requests, bug reports, or feature suggestions.

hjdhjd / homebridge-unifi-protect