unifiprotect unexpectedly reloads during setup and from conflicting discovery

[Metal-Eagle]

The problem

The caramas stopped working after update to Core 2022.6.1

It seems that after a reboot the cameras will work for a few minutes, but then I get the following error in the log and they stop working.

Logger: homeassistant.components.switch
Source: helpers/entity_platform.py:598
Integration: Switch (documentation, issues)
First occurred: 07:55:45 (54 occurrences)
Last logged: 07:55:46

Platform unifiprotect does not generate unique IDs. ID 6006f96a010d0e03870003f1_privacy_mode already exists - ignoring switch.privacy_mode_side_yard_right
Platform unifiprotect does not generate unique IDs. ID 6006f96a010d0e03870003f1_osd_name already exists - ignoring switch.side_yard_right_overlay_show_name
Platform unifiprotect does not generate unique IDs. ID 6006f96a010d0e03870003f1_osd_date already exists - ignoring switch.side_yard_right_overlay_show_date
Platform unifiprotect does not generate unique IDs. ID 6006f96a010d0e03870003f1_osd_logo already exists - ignoring switch.side_yard_right_overlay_show_logo
Platform unifiprotect does not generate unique IDs. ID 6006f96a010d0e03870003f1_osd_bitrate already exists - ignoring switch.side_yard_right_overlay_show_bitrate

I already tried to remove and add the integration again but unfortunately with the same result. Maybe also good to know I have 4 cameras in unifi protect.

Could it be because i have unifi protect 2.0.1? Because I can find in the documentation that this version is supported.

What version of Home Assistant Core has the issue?

Core 2022.6.1

What was the last working version of Home Assistant Core?

Core 2022.5.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

UniFi Protect

Link to integration documentation on our website

https://www.home-assistant.io/integrations/unifiprotect

Diagnostics information

config_entry-unifiprotect-a8178bc7a8d26d363b0c048dcb462315.json.txt

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant[bot]]

unifiprotect documentation unifiprotect source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @briis, @angellusmortis, @bdraco, mind taking a look at this issue as it has been labeled with an integration (unifiprotect) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[xpfgsyb]

Same here, but wanted to add some additional information. The problem started already in 2022.6.0. Last working version was 2022.5.5.

I get the same error in the log. If I visit a dashboard page with a camera I can see in the browser developer options that home assistant does get a http 404 error when accessing the api (https://ha-url/api/camera_proxy/entity). Until 2022.5.5 it got a 200 and an image.

I tried to uninstall, restart, reinstall the protect integration, but that didn't help. A rollback to 2022.5.5 does fix the issue instantly and doesn't show the errors in the log.

[rqmartins73]

Good-morning,

Exact same issue here, and also started with 2022.6.0. Also last working version was 2022.5.5. I also uninstalled Protect Integration and re-installed which appears to solve the problem for awhile but after some time the problem comes back. Sometimes rebooting HA solves the problem for some hours or minutes, sometimes (like it is now) doesn't.

I have a UDMP 1.12.22 with Protect 2.0.1.

Thanks and regards.

Platform unifiprotect does not generate unique IDs. ID 60b10fc70276a7038700099d_system_sounds already exists - ignoring switch.entrada_sul_e_corredor_r_c_system_sounds 10:35:25 – (ERROR) Switch - message first occurred at 10:28:27 and shows up 408 times Platform unifiprotect does not generate unique IDs. ID 60350b3a037da203870003e9_resolution_HD already exists - ignoring sensor.rqmudmpro_resolution_hd_video 10:35:25 – (ERROR) Sensor - message first occurred at 10:28:27 and shows up 240 times Platform unifiprotect does not generate unique IDs. ID 60b263680030a70387001370_infrared already exists - ignoring select.escritorio_infrared_mode 10:35:25 – (ERROR) Select - message first occurred at 10:28:27 and shows up 96 times Platform unifiprotect does not generate unique IDs. ID 61ceefe203d0de0387005572_mic_level already exists - ignoring number.garage_front_microphone_level 10:35:25 – (ERROR) Number - message first occurred at 10:28:27 and shows up 42 times Platform unifiprotect does not generate unique IDs. ID 60ba3f0001473d0387001011_speaker already exists - ignoring media_player.back_yard_speaker 10:35:25 – (ERROR) Media player - message first occurred at 10:28:27 and shows up 30 times Platform unifiprotect does not generate unique IDs. ID 61ceefe203d0de0387005572_0 already exists - ignoring camera.garage_front_high 10:35:25 – (ERROR) Camera - message first occurred at 10:28:27 and shows up 42 times Platform unifiprotect does not generate unique IDs. ID 61ddf871001ce30387000667_motion already exists - ignoring binary_sensor.sala_motion 10:35:25 – (ERROR) Binary sensor - message first occurred at 10:28:27 and shows up 96 times

[Trixxr]

I too am experiencing the exact same as the above posters, as of upgrading past 2022.5.5.

[correctomundo79]

I'm experiencing the same issue as well after upgrading from 2022.5.5 to 2022.6.2

[fribse]

I think it's the same as this report: https://github.com/home-assistant/core/issues/72595

[xpfgsyb]

@fribse I don't think so. #72595 reads more like a mix of "I used an old HA version", "I forgot to remove the old HACS integration" and #72927. At least in my case all of the 3 options doesn't apply and also the error logs are different. Only your comment reads similar to here in my opinion.

As I use HA in a virtual machine it's relatively easy for me to roll back HA, and can therefore say that it's really just the HA version. The Protect version is always the same on my UDM Pro (2.0.1) and the breaking change was somewhere between HA 2022.5.5 and HA 2022.6.0, because as soon as I roll back to 2022.5.5 the integration works fine.

[andras-tim]

I experienced the same issue that @xpfgsyb described. (HA 2022.5.5 + Protect 2.0.1 worked well)

I tried to remove the integration, restart the HA, then reinstall it. Everything started to work... but until the first restart only.

(I think, the integration wants to re-create the sensors instead of use the previously created ones)

[bdraco]

This looks like https://github.com/home-assistant/core/pull/73145 which is fixed in 2202.7.x which scheduled for next month (It cannot be backported to 2022.6 as it adds a new state to config entries and needs to go out in a beta first).

If its still a problem after that we will probably have to ignore discovery updates from a different subnet, but that is a breaking change that we would like to avoid making unless there is no other option to solve this.

[bartbakels]

I experienced the same issue that @xpfgsyb described. (HA 2022.5.5 + Protect 2.0.1 worked well)

I tried to remove the integration, restart the HA, then reinstall it. Everything started to work... but until the first restart only.

(I think, the integration wants to re-create the sensors instead of use the previously created ones)

Exactly same here, until restart. Next to that iam still on protect 1.21.6

[xpfgsyb]

(...) discovery updates from a different subnet (...)

I'm not sure if I interpret this correct. So in cases where the Protect server has multiple IP addresses in different subnets it may happen that discovery find them too and try to add them with the same ID? So a possible workaround until the release of 2022.7 could be to make sure that the HA server can only access the configured IP of the Protect server and no other IPs?

[bdraco]

So a possible workaround until the release of 2022.7 could be to make sure that the HA server can only access the configured IP of the Protect server and no other IPs?

Maybe. You'd need to look at the debug logs for pyunifiprotect and unifi-discovery and homeassistant.components.unifiprotect to verify that is what is happening and its being reloaded in the middle of startup

[fribse]

@bdraco Interesting, having a Dream Machine Pro protect will have multiple IP's, as it's on the router itself. The only VLAN that can access all of them (for my setup) is the LAN, where the HA is, so I guess I could make a firewall rule that drops the access to the other 'gateway' ip's the for the VLANS. I'll try it out.

[bartbakels]

Additional remark, i rolled back to 2006.6.3 from .4 and now it is stable, until i restart. But when on .4 all protect streams and sensors are not updated after an hour orso. And the log is flooded then with cant generate unique ids (on .4)

[xpfgsyb]

Ok, I tried the workaround idea with HA 2022.6.4 now. While it didn't work without the block rule as expected, it does work with the blocking rule after a HA restart now. I don't get the unique ID error in the log anymore and the camera stream is working since >30 minutes. I also did a complete reboot of the HA server just to be sure and it's still fine. Thanks for the side note about discovery, @bdraco!

[hunterdrayman]

Got this problem too and it's quite debilitating having motion/person detections not reaching my automations - aargh! My UDM-Pro has nine VLANs of which two are connected to the HA virtual machine.

Is a possible workaround option to disable the discovery process (wherever that is)?

[andras-tim]

Got this problem too and it's quite debilitating having motion/person detections not reaching my automations - aargh! My UDM-Pro has nine VLANs of which two are connected to the HA virtual machine.

Is a possible workaround option to disable the discovery process (wherever that is)?

You can manually select the interfaces that you needed by unchecking the "Auto Configure" item on the https://my.home-assistant.io/redirect/network/ page

[hunterdrayman]

Got this problem too and it's quite debilitating having motion/person detections not reaching my automations - aargh! My UDM-Pro has nine VLANs of which two are connected to the HA virtual machine. Is a possible workaround option to disable the discovery process (wherever that is)?

You can manually select the interfaces that you needed by unchecking the "Auto Configure" item on the https://my.home-assistant.io/redirect/network/ page

Thanks, yes I need both (for some IoT broadcast updates).

[AngellusMortis]

I think as a workaround (do not know the discovery code very well), remove your UniFi Protect config entry, restart HA, wait for it to be discovered, ignore it and then wait for it to be discovered again (if HA is not on your default LAN) and ignore that one too. Then after they are both ignored, configure it manually using the IP address.

[hunterdrayman]

Nice idea - I'm giving it a try. I ignored it once but it didn't come back with another discovery after half an hour so I added the integration manually. So far, so good. In the meantime I found that the manifest.json gives a thumbs-up to SSDP discovery "ssdp": [ { "manufacturer": "Ubiquiti Networks", "modelDescription": "UniFi Dream Machine" }, { "manufacturer": "Ubiquiti Networks", "modelDescription": "UniFi Dream Machine Pro" }, { "manufacturer": "Ubiquiti Networks", "modelDescription": "UniFi Dream Machine SE" } ],

So, I'm wondering is there's a way to install it as a custom component overriding the inbuilt integration temporarily and avoiding any further discoveries. Probably not a good idea.

[bdraco]

This likely only happens if direct connect doesn't work. If you check your .storage/core.config_entiries you'll likely find its connecting to the ip instead of the direct connect name.

Usually the fix for this is to make sure your HA instance is using your UI device for dns lookups and to recreate the config entry.

[bdraco]

Additionally it would be helpful if everyone affected provided the logs mentioned here https://github.com/home-assistant/core/issues/73013#issuecomment-1152115976

[xpfgsyb]

Debug log attached: protect.log I think the interesting line is 192, where unifiprotect.discovery found the UDM Pro with it's primary IP (10.C.D.2), which is not the IP I use in the integration config (10.A.B.1). It's aware of both IPs in that line and doesn't change the IP for the following communication.

[fribse]

I added firewall rules to the Dream Machine Pro so HA can only see the primary IP, and after doing that, the integration works after a reboot.

[bdraco]

@xpfgsyb Based on your logs, I think https://github.com/home-assistant/core/pull/73145 will solve the problem unique id problem, but it won't stop your instance from flip-flopping ips.

I don't see a solution to that problem that doesn't result in ignoring discovery updates from a different subnet which is a breaking change for others.

So we are kinda stuck with either breaking discovery on other subnets or fixing the ip flip-flop issue. Obviously everyone here is going to prefer breaking discovery on other subnets in exchange for fixing this issue as the users who rely on discovery on other subnets are not represented here.

[hunterdrayman]

Here's the log from my system almost exactly 2 hours after the integration was removed/re-added and HA restarted:

uni-protect.log (tokens redacted)

The default LAN is 192.168.1.0/24 192.168.2.0/24 is my office VLAN - the UDM-Pro has 192.168.2.1 and HASSOS has 192.168.2.230 192.168.4.0/24 is my Home Automation VLAN - the UDM-Pro has 192.168.4.1 and HASSOS has 192.168.4.249

Protect version 2.0.1

Version | core-2022.6.5 -- | -- Installation Type | Home Assistant OS Development | false Supervisor | true Docker | true User | root Virtual Environment | false Python Version | 3.9.12 Operating System Family | Linux Operating System Version | 5.15.41 CPU Architecture | x86_64 Timezone | Europe/London

[bdraco]

I've been testing solution on this for a few hours and I have yet to find something that accommodates both use cases. I'm pretty much ready to call this as unfixable without a breaking change

[hunterdrayman]

Thanks for looking into it. I have disabled one of my virtual NICs as a workaround and can easily live with the situation for a month. A better temporary remedy would be to firewall-block the discovery conversation but I'm not clear on what ports it uses. I'm guessing 9000 (SSDP) and 80/443 (API). Anyone know?

[bdraco]

Another potential solution if you are using a UDM is to get your Home Assistant box using the UDM for dns as it should switch over to using the ui.direct connect method which would prevent it from swapping ips.

[hunterdrayman]

I'm giving that a try - thanks. I have a Pi-hole as my DNS server generally, but have just now hacked /etc/resolv.conf on HASSOS as a test. If the HA core does its own name resolution that probably won't work but should be interesting. I've not heard of 'ui.direct connect' before so I'll need to look that up to understand fully.

[fribse]

@hunterdrayman I made a rule, one with a list of IP's on the HA, and a list of 'wrong' IP's for the UDM Pro (ie. all the other IP's it has), and then blocked 'any' TCP. The rule is put into 'LAN Local'. Made it work, and is quite simple once it works.

@bdraco I am using an AdGuard as DNS, but it's upstream DNS is the UDM Pro, so I just tried disabling my newly created firewall rule, removed the integration, and added it back with the DNS name (had to NOT set 'verify SSL certificate' as it's not possible to put a valid certificate on the UDM pro) and after a reboot, it still works like a charm, so I'll leave at that for the moment, and see if it 'keeps'.

[bdraco]

Check https://x.x.x.x/api/system for the directConnectDomain

If you are using your UDM for dns, this domain should have a valid ssl certificate and resolve to your device.

[fribse]

@bdraco That works as well, and as you said, it has a valid SSL.

[AngellusMortis]

I'm giving that a try - thanks. I have a Pi-hole as my DNS server generally, but have just now hacked /etc/resolv.conf on HASSOS as a test.

Point your UDM at your PiHole and let everything use your UDM for DNS via the default DHCP DNS. That is what I do. Then you can also point HA OS to your gateway DNS as well if you have already changed it or it is not set up right:

ha dns info # check current config
ha dns options --servers dns://x.x.x.x
ha dns restart

If I recall correctly, HA OS uses Google (8.8.8.8) as a fallback if your DHCP DNS fails. So, adding the servers manually will make it your PiHole as a fallback instead.

[bartbakels]

@hunterdrayman I made a rule, one with a list of IP's on the HA, and a list of 'wrong' IP's for the UDM Pro (ie. all the other IP's it has), and then blocked 'any' TCP. The rule is put into 'LAN Local'. Made it work, and is quite simple once it works.

@bdraco I am using an AdGuard as DNS, but it's upstream DNS is the UDM Pro, so I just tried disabling my newly created firewall rule, removed the integration, and added it back with the DNS name (had to NOT set 'verify SSL certificate' as it's not possible to put a valid certificate on the UDM pro) and after a reboot, it still works like a charm, so I'll leave at that for the moment, and see if it 'keeps'.

Could you post a screenshot of that fw rule? My has is in 5 vlans. So are you blocking source as udm and destination the “wrong” has ips or the other way around?

[hunterdrayman]

@bartbakels Could you post a screenshot of that fw rule? My has is in 5 vlans. So are you blocking source as udm and destination the “wrong” has ips or the other way around?

This one is doing it for me at the moment - It's my only LAN_LOCAL rule.

192.168.2.0/24 is my office VLAN - the UDM-Pro has 192.168.2.1 and HASSOS has 192.168.2.230. I leave this unblocked.

192.168.4.0/24 is my Home Automation VLAN - the UDM-Pro has 192.168.4.1 and HASSOS has 192.168.4.249. This rule rejects attempts to query the UDM-Pro API from the Home Automation VLAN.

[xpfgsyb]

Update: works for me without the firewall rule with the patch in 2022.6.6. Thank you :)

[Metal-Eagle]

Also an update from me after the update to 2022.6.6 everything works again, thank you very much!