Unifi Integration - devices go "away" when still connected to Unifi system, stay that way.

mkmer commented 3 years ago

The problem

Devices on the tracker show "AWAY" when they are connected - stay that way until they actually disconnect and reconnect to the Unifi WIFI. Unifi controller logs don't appear to show the device roaming or disconnecting during the time frame the device changed to "AWAY". Additionally, restarting home-assistant updates the devices to the correct "HOME" state.

This behavior seems to have appeared on the 2021.5.x update and continues on the 2021.6.x

I am running Unifi Controller 6.2.25.0 (in a docker container linuxserver/unifi-controller) - same docker type configuration for a couple of years, just keep the versions updated.

I have turned logging on, and the problem seems to disappear... then when I turned it back off it reappeared. I'm sure you will need some logging to figure out what the unifi controller is sending to confuse home assistant. Let me know which logs specifically to turn on and I'll try to capture it.

What is version of Home Assistant Core has the issue?

core-2021.6.3 (started in core-2021.5.x)

What was the last working version of Home Assistant Core?

core-2021.4.x

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Unifi

Link to integration documentation on our website

https://www.home-assistant.io/integrations/unifi/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 3 years ago

Hey there @kane610, mind taking a look at this issue as its been labeled with an integration (unifi) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

davesilva commented 3 years ago

I don't have much to contribute here in terms of new information but I've been having this same issue for a long time now. I don't know exactly when it started but it was definitely before core-2021.5.x because I've been seeing it for months now.

After the integration has been running for a while some (but not all) devices will go into the Away state even though I can still see them as connected clients in the Unifi dashboard. If I go to the integrations config page and click Reload on the Unifi integration then everything shows up in the correct state again. Several hours later though the issue will come back and I'll have to reload it again.

CrouchingWorm commented 3 years ago

Similar issue for me - mostly see it happening on Apple devices for me - probably as that's what I'm interested in tracking. I'm happy to capture whatever logs I need to help.

mkmer commented 3 years ago

Maybe this is interesting information: It appears that EVERY time a device shows "away" on HA, tracking the log file for the device in Unifi Controller - the last event for that device is "Disconnected". Even though the device shows connected on the dashboard, the last event in the controller log is disconnected. Maybe the log & interface agree - aka the last state change sent to HA was disconnected, so the device is away?

I have logging turned on at the moment, next time I find a device stuck, we should have a good picture of what the controller sent and when.

Currently I see on the web socket, a status message about all of the devices - maybe somehow we could use that to update the status of devices that are actually connected?

I think you need to turn on the following: - it will make a HUGE file, so be ready :) logger: default: info logs: aiounifi: debug homeassistant.components.unifi: debug homeassistant.components.device_tracker.unifi: debug homeassistant.components.switch.unifi: debug

I think all we need to do is find the time the device went away, then filter the events after that time.... really need some feedback from @Kane610 ...

Kane610 commented 3 years ago

I'm planning on looking at the different issues reported on the unifi integration. I'm grateful you investigate the issue.

I cant promise when exactly I will start. Just started my vacation following a busy spring

mkmer commented 3 years ago

Understood. The attached log file shows "Mike's Pixel" as away from 11:46AM to 12:21PM and again 1:08PM to 1:41PM on the 26th. The device was connected and showed as such on the Unifi Controller. Somewhere in there may be helpful information.

Thanks for all your hard work on this integration - it's key to my entire system! ( I guess I can't attach the log, but have it available for you and will send as needed)

mkmer commented 3 years ago

@Kane610 : I have done a review of the logs - it appears Unifi is to blame. In my examples the controller sends a EVT_WU_Disconnect at 11:41:45 for the given device (and the device goes "away"). Then the next event is at 12:21:10 is a ET_WU_RoamRadio which causes the device to go "home".
The entire time, there are updates for the device coming in on the interface but not EVT messages :( At least this lines up with the log files in the Unifi Controller... for whatever that's worth.

Maybe it occasionally needs to act like a startup and verify the devices (say every 15 minutes?) Even at 15 minutes, many automations will occur even though the user hasn't left (in my house that's lights turning off in the kids bedrooms).

szulik commented 3 years ago

same problem here. It started on 2021.4 or 2021.5 version. Restarting HA immediately fixes the status for 1-2 days.

mkmer commented 3 years ago

OK - I've backed the Unifi Controller down to 6.1.71.0 and the problem appears to have "gone away" for the last 24 hours. I've tried several of the 6.2 series and the problem shows up relatively quickly - all the way to 6.2.26.
As observed on my previous post, the controller is not logging or sending a connect event message yet the devices are connected. What versions of the unifi controller are y'all seeing this problem on?

szulik commented 3 years ago

6.2.26 . I'm not 100% sure but probably problem has started before Unifi controller update from 6.1. Will try to revert to 6.1.71

mkmer commented 3 years ago

On my 4th day - at least for my configuration 6.1.71 has been stable. I should mention I'm running HA in the docker container as well as Unifi Controller in a Docker container (this version: linuxserver/unifi-controller).

One of the symptoms I noticed in the UC was the logs would show a disconnect with no reconnect or roam while HA thought it was away - then when a device finally roamed or disconnect/reconnect in the log, it shows on HA. I suspect UC is not sending the reconnect/connect message when the device connects (just like the log shows), yet on the device tab the device shows connected.

At this point, I suspect the only "hack" would be for HA to poll UC every now and then to get a "clean" update of what's connected and disconnected (seems like there used to be an option like that...)- maybe it could be a configurable option with configurable rate? OR maybe poll a bit after a device disconnects to "see" if it reconnected without a state change message. Too bad :(

szulik commented 3 years ago

Updated HA to 2021.7.4 more than 10 days ago (still running 6.2.26 Unifi controller) and home/away status works perfectly fine. So, maybe it was fixed in HA update?

Kane610 commented 3 years ago

I haven't changed anything related to device tracker for 2021.7. I've done minor fixes and reintroducing myself to the unifi integration with the aim to solve this issue.

rwarner commented 3 years ago

Just chiming in here, I am also running into me and my wife’s phone suddenly going “away” while we’re sitting on the couch using the wifi on our phones. No random MAC address generation on either phone, both consistent. Doesn’t resolve until physically getting off wifi and getting back on.

Was happening more with the iPhone than my Pixel, but just yesterday I noticed it happened to both of us.

on 2021.7.4 and unifi controller is 6.2.25. Am going to upgrade to 6.2.26 though as I just noticed there’s an update available

CrouchingWorm commented 3 years ago

I also use apple devices and am having the same issues. Strangely, it's only one of the six iphones/ipads I track. the last two I've added is affected (both iPhone12's). Also run HA 2021.7.4 and controller is 6.2.26.

Kane610 commented 3 years ago

I was thinking if device firmware of APs and switches affects this issue as well, what firmwares do you guys have on your hardware?

rwarner commented 3 years ago

Right now I’m running the following. Updated a few devices recently:

Unifi AP AC LR: 5.43.38.12731 Unifi AP AC Pro: 4.3.28.11361 Unifi Nano HD: 5.60.9.12980

Unifi OS Network: 6.2.26

Have a few other Unifi devices, not sure if it’s be worth mentioning their firmware versions though. I haven’t seen this issue occur in a few days fwiw. Totally could be a unifi firmware thing. Haven’t upgraded HASS to 2021.8 yet

JayOne73 commented 3 years ago

As a workaround: I reload the integration by an automation when a mobile device state goes to "not_home" (from: "home").

- alias: "Reload Unifi Integration"
  trigger:
    - platform: state
      entity_id:
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
      from: "home"
      to: "not_home"
  action:
    - service: homeassistant.reload_config_entry
      data:
        entry_id: xxx

Kane610 commented 3 years ago

I've tried to reproduce it here and so far I have been out of luck. But my dev environment is on a laptop so might be too volatile

rwarner commented 3 years ago

Still running into this issue here and there with 2021.8.8

@JayOne73 does this resolve for you once reloading unifi? I’ve noticed that if my wifes phone was suddenly “away” and reloaded unifi integration it didn’t resolve the issue still. Including restarting HASS

Kane610 commented 3 years ago

Hey all! It seems that the user from the other issue solved his issue by resetting his udm pro (controller?) can anyone else try the same thing, since I haven't been able to reproduce this yet it would be a good thing to try.

rwarner commented 3 years ago

I unfortunately, can't reset mine at the moment (Unifi UCK-G2)

I will take more notice recently with iOS15 now though on her phone and there are some updates to apply to the unifi devices. Will see how it keeps going. Also am on the latest 2021.9.7 and fortunately haven't seen an oddity for my wifes phone in the last week

townsmcp commented 3 years ago

Just as an FYI for anyone interested, I had to follow the emergency recovery mode procedure to downgrade UDM Pro to UniFiOS v1.10.0 and Controller v6.2.26 (reset to factory settings doesn’t downgrade OS firmware) - my HA install is 2021.9.6

https://help.ui.com/hc/en-us/articles/360043360253-UniFi-UDM-UXG-Emergency-Recovery-Mode-UI

jonpage88 commented 3 years ago

I wanted to add to this. I am also having this problem but it seems to be related to moving across different APs in my house. I am running the Unifi Controller in a docker (linuxserver/docker-unifi-controller) that manages my Unifi hardware. I have the Home Assistance integration connected that pulls in connected devices that I use for presence detection.

After a lot of testing, I have figured out the issue happens when my devices change access points. I have three of them in my house and the network I am connecting to on all 3 has the same SSID and same VLAN. The first AP that one of my devices connects to will work fine but if the device connects to a different AP, the device will go to “not home” after 300 seconds. If I turn off wifi and turn it back on it starts working again till the device changes AP or if I restart Home Assistant everything will work till the device changes AP.

I have verified in the Unifi Controller the device says that it is active the whole time. I verified that the time is set correctly on both docker containers, I have updated Home Assistant and the Unifi Controller. I removed the integration and added it back. I have enabled logging for the Unifi integration but I have not been able to find anything useful. The device never shows as wired as I have seen in some posts and all my network hardware is Unifi brand managed by the controller.

Kane610 commented 3 years ago

That's great! I only have one access point so could be the reason why I haven't been able to reproduce it. If you enable debug logging for the integration do you see any events with roaming messages?

arnoldpistorius commented 3 years ago

I also have this issue with the Unifi integration. Since it appears to happen when I go upstairs or downstairs, I suspect that it happens when the phones are switching between APs (like @jonpage88 says). Altough, I don't have proof for this.

My GF uses a Samsung A52 and I use a Xiaomi Mi9. Both have the same behavior.

Kane610 commented 3 years ago

Any update here? Anyone who can provide logs to better identify the issue with roaming between APs?

fdaemon commented 3 years ago

I'm having this same problem. Happening (but not always) when I'm roaming from 1 AP to the other. The WIFI works fine on the device, no issues there. Unifi UI also showing the client as connected. It's just this integration that saying my device is not_home. Restarting wifi or reloading the integration helps.

Kane610 commented 3 years ago

Hello! Have anyone tried out the notification settings in UniFi controller? I noticed them reading through the latest release notes for 6.4.54 (https://community.ui.com/releases/UniFi-Network-Application-6-4-54/c1be3b7f-44c4-4d6f-af1e-707bf017110d) one of the first posts. The issues could stem from UniFi changing default settings for notifications, I will try to look at this in the next few days but good if more try it out.

github-actions[bot] commented 3 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

wilpig commented 3 years ago

This is definitely still happening. I've been trying to catch it in the act but I ran out of disk space for the logs so i've added some extra space and reset the logs to begin the wait again.

I did look at that alerts control panel and made sure the device updates were on didn't seem to make a difference.

UniFi Controller: 6.4.54 Hass: 2021.10.6

CrouchingWorm commented 2 years ago

I also believe this issue still exists. I concur it seems to arise when roaming from one AP to another. I am however using a workaround for my phones at the moment as a result of this issue.

Kane610 commented 2 years ago

Yes I don't believe those alerts are affecting this either.

ZuluWhiskey commented 2 years ago

Just as an FYI, I only have one AP and am still seeing this issue. I am currently experiencing the issue with a wired laptop connected to a UniFi switch, so I don't think this is WiFi / AP related unfortunately. UniFi GUI shows device is connected, HA shows "not_home" - I reload the config entry and then it shows as "home" in HA.

LucidityCrash commented 2 years ago

I too am seeing this with 2021.11 and controller 6.4.54.0 In particular I have a wired laptop that regularily shows Away when it is on. Even if the laptop has been powered off overnight powering it back on doesn't make the state change to Home. It seems that once it gets to the Away state it gets stuck. Reloading the integration fixes it for a while

Kane610 commented 2 years ago

Just as an FYI, I only have one AP and am still seeing this issue. I am currently experiencing the issue with a wired laptop connected to a UniFi switch, so I don't think this is WiFi / AP related unfortunately. UniFi GUI shows device is connected, HA shows "not_home" - I reload the config entry and then it shows as "home" in HA.

Ok, it might be related or something else. Only logs will tell.

rwarner commented 2 years ago

What would be the best way to get logs for you? Also still seeing this intermittently but I know it’s frustrating that I don’t have logs to provide whilst complaining lol

Are you looking for logs on the Unifi side? Or the HASS side?

LucidityCrash commented 2 years ago

What would be the best way to get logs for you? Also still seeing this intermittently but I know it’s frustrating that I don’t have logs to provide whilst complaining lol

Are you looking for logs on the Unifi side? Or the HASS side?

I was just thinking the same ... what logs exactly are you looking for ?

Kane610 commented 2 years ago

Debug logs from hass (its pasted in the early posts, can also be found here https://www.home-assistant.io/integrations/unifi/#debugging-integration)

notarised around the time the behaviour is identified together with what Mac address of the client that the behaviour can be observed with, without this its a needle in a haystack

Do a replacement of your external IP address and wifi passwords can be printed in them as well (though only relevant with local access), you can share the log files privately with me through home assistant discord (Robban)

rwarner commented 2 years ago

Debug logs from hass (its pasted in the early posts, can also be found here https://www.home-assistant.io/integrations/unifi/#debugging-integration)

notarised around the time the behaviour is identified together with what Mac address of the client that the behaviour can be observed with, without this its a needle in a haystack

Do a replacement of your external IP address and wifi passwords can be printed in them as well (though only relevant with local access), you can share the log files privately with me through home assistant discord (Robban)

Thanks, will add these and try to capture / notice when it occurs

rwarner commented 2 years ago

Okay I enabled the logging you linked @Kane610

Then while I was on my computer and my phone was charging it suddenly went to "Just Left" and it's been in "Away" status ever since. Device Tracker is "not_home" for it as of the past three hours or so (7:53 p.m. till 10:07 p.m.)

Will get you the logs on discord when we can sync up -- sent you a message

Kane610 commented 2 years ago

@wilpig I couldn't event find a disconnected event for that device :/

@LucidityCrash according to you and the logs your wired client gets marked as disconnected through an event at 2021-11-18 11:02:43 But UniFi keeps giving details about the wired client with updated last_seen timestamp.

@rwarner according to your logs the wireless client gets marked as disconnected through an event at 2021-11-15 19:48:23 But UniFi keeps giving details about the client with updated last_seen timestamp.

It might be time to rework state calculations for client tracker...

LucidityCrash commented 2 years ago

@Kane610 The reason it starts giving details about the wired client at about 2021-11-18 11:08:24 is because that is when I turned the device back on. There is no event in the Unifi events log when the device rejoined. If I look in my event log in the unifi controller there are only disconnect events : Wifi devices however get both disconnect and connect events

rwarner commented 2 years ago

@rwarner according to your logs the wireless client gets marked as disconnected through an event at 2021-11-15 19:48:23 But UniFi keeps giving details about the client with updated last_seen timestamp.

Do you think my phone temporarily disconnected and connected quicker than the Unifi Integration perhaps expected? Anything else I can assist with in particular to this situation?

Kane610 commented 2 years ago

It appears that the UniFi events aren't good enough to be trusted as a sole source of data. Or I'm misunderstanding how to properly utilise them.

mkmer commented 2 years ago

@Kane610 I apologize for dropping this issue - somehow I stopped getting email updates and completely forgot to follow up! I really appreciate you sticking with the issue.

If it's not clear, the original problem (and continued issue) is that the Unifi controller log shows a disconnect event without a paired connect event, while at the same time down the road there will be roaming events for a user without the connect event!!! Clearly the controller is failing to declare a connect event...

Unfortunately, the disconnect event when not actually disconnecting will not be completely solved by adding "roaming" to be a connected state action in HA because 1) I may only have one access point, or 2) the phone doesn't roam before the away event occurs.

The only way I can see fixing this issue at this point would be to drive a poll event just before declaring a device AWAY to confirm we didn't "miss" a connect event (aka the controller didn't declare one). I think this proposed solution would keep most of the event based performance improvements as well as fix the issue (and make my wife happy that the lights stay on :) )

BTW: The events logged in the UNIFI event log have always matched the events sent to HA via the socket.... Earlier in the conversation I had shared a log file and have reviewed MANY events - ALL of them have Unifi declaring a disconnect event and next reporting a roaming event, or wait until the device actually disconnects to declare reconnect/connect. Any solution to the issue will have to be and HA workaround, as it's fairly clear Unifi events are not reliable.

I suppose if not "polling" for the device before declaring away, maybe any activity on the device via other messages could declare it "home"... but who's to say the disconnect event doesn't fail sometimes???? If that direction fails we will never go "away" - but we just haven't noticed it yet.... This would required some sort of timer on activity messages for each device... Just ugly all around :(

szulik commented 2 years ago

For couple of months the problem with switching away-home-away devices by Unifi integration disappeared. Suddenly, two days ago (didn't changed anything in HA or in Unifi controller for many days) it has started again. Every 20 seconds all wifi devices go away and than go home. Restarting HA or Unifi controller doesn't help. Every minute all devices switches home-away-home.

I have just one Unifi AP so it is not because of roaming. According to Unifi controller there are no disconnect events at all

Here is log file:

home-assistant (2).log

mkmer commented 2 years ago

@JayOne73

As a workaround: I reload the integration by an automation when a mobile device state goes to "not_home" (from: "home").

- alias: "Reload Unifi Integration"
  trigger:
    - platform: state
      entity_id:
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
        - device_tracker.mobiel_xxx
      from: "home"
      to: "not_home"
  action:
    - service: homeassistant.reload_config_entry
      data:
        entry_id: xxx

This is a great solution solving the "stuck away" - but it breaks automation if there is a timer say x minutes for home to not_home - because it goes "home" to "not_home" to "Unavalable" to "not_home" effectively killing the automation timer. If you remove the timer, the momentary "not_home" causes the automation to run (which is not desired) - and if you remove the "from Home" event, the automation triggers every time the integration is reset (because it goes from "not_home" to "unavailable" to "not_home" causing the automation to trigger again....

Have I missed something?

Kane610 commented 2 years ago

@Kane610 I apologize for dropping this issue - somehow I stopped getting email updates and completely forgot to follow up! I really appreciate you sticking with the issue.

If it's not clear, the original problem (and continued issue) is that the Unifi controller log shows a disconnect event without a paired connect event, while at the same time down the road there will be roaming events for a user without the connect event!!! Clearly the controller is failing to declare a connect event...

Unfortunately, the disconnect event when not actually disconnecting will not be completely solved by adding "roaming" to be a connected state action in HA because 1) I may only have one access point, or 2) the phone doesn't roam before the away event occurs.

The only way I can see fixing this issue at this point would be to drive a poll event just before declaring a device AWAY to confirm we didn't "miss" a connect event (aka the controller didn't declare one). I think this proposed solution would keep most of the event based performance improvements as well as fix the issue (and make my wife happy that the lights stay on :) )

BTW: The events logged in the UNIFI event log have always matched the events sent to HA via the socket.... Earlier in the conversation I had shared a log file and have reviewed MANY events - ALL of them have Unifi declaring a disconnect event and next reporting a roaming event, or wait until the device actually disconnects to declare reconnect/connect. Any solution to the issue will have to be and HA workaround, as it's fairly clear Unifi events are not reliable.

I suppose if not "polling" for the device before declaring away, maybe any activity on the device via other messages could declare it "home"... but who's to say the disconnect event doesn't fail sometimes???? If that direction fails we will never go "away" - but we just haven't noticed it yet.... This would required some sort of timer on activity messages for each device... Just ugly all around :(

I have concluded as well from different logs that improving roaming events will not solve this issue. The only path to solve this issue is to ignore events and revert back to looking at the uptime value more or less.

I hope to solve this within a reasonable time, though that will be at the earliest for next beta cycle, a lot is happening outside of my home automation endeavours and I'm really looking forward to Christmas vacation prior to any major new contributions here.

mkmer commented 2 years ago

Thanks for the update... This is mostly annoying, and for the moment, the reload automation recommended above covers the lights going out when my wife is actually at home ----- happy wife happy life :)

On the thoughts about abandoning events, I feel we can get by with the events, as most of the "disconnect" type events are "true"... I haven't found a condition where we missed a disconnect event (aka showing home when it's actually away). The unfortunate part is missing "connect" events - so maybe it should be a "confirmation" that the device is disconnected when the event occurs just before declaring it away.

I'm not sure exactly how the "initialize automation" works, but it always correctly figures out devices are connected regardless of the events - so maybe we can use that part of the mechanism to "check" the disconnect event for a device is real after the integration away timer is expiring? (without the full reset of the integration & the subsequent unavailable event on the entity)

Good luck AND happy holidays!

home-assistant / core