search

travisghansen / hass-opnsense

OPNsense integration with Home Assistant
Apache License 2.0
219 stars 29 forks source link

OPNsense entities reports as "Unavailable" - except... #52

Closed larhedse closed 1 year ago

larhedse commented 1 year ago

So I am trying this integration out, and I like everything about it, except it goes "Unavailable" as soon as I close my main (wired ethernet) PC down. For some reason, OPNsense integration ONLY works when my main PC is on - it is like the connection between Home Assistant and OPNsense has to go thru my main PC - and that can just not be true...

Yes they are part of separate networks so to speak, and so is my main PC also.

Home Assistant is IP: x.y.2.20 OPNsense is IP: x.y.1.1 And my main PC, LAN in OPNsense world, is x.y.1.20

So if it has to be on the same segment, then I could understand - but why?

Also the same applies in a way for presence detection, I am tracking my wife's and mine Android phones, and they also goes "Unavailable" from time to another, except they are also reported as offline, even though they are not (however I think this "offline" quirk is more a Samsung Mobile Phone sleep thing, since it in OPNsense is clearly marked as "offline" even thought we are both at home - hard to say how to get that working better...).

larhedse commented 1 year ago

Found this in the error log, got a bunch of them:

2023-01-04 17:06:45.565 ERROR (SyncWorker_9) [homeassistant.components.unifi_direct.device_tracker] Failed to decode response from AP

Not sure if this relates to OPNsens plug in - it could be the Unifi integration of course - I am running both for device presence detection, hoping one or the other works so to speak...

larhedse commented 1 year ago

This one can be closed, it is more of a Home Assistant issue than OPNsense. I have different network interfaces that has there own rules and DHCP server in my OPNsense installation (8-interfaces). This is not possible to handle from within Home Assistant, which requires that anything and everything needs to be on the same network segment, or else...

travisghansen commented 1 year ago

Hey! I’m just really busy and haven’t had a chance to respond or look at things. If you still need something let me know and I will get to it, but may take me a bit.

This integration actually makes it so hass doesn’t need to necessarily be on the same network etc as it fully gets its data from opnsense (this could be across the internet even). Sometimes the api is slow to respond in certain scenarios which leaves the entities unavailable. Some of internals could be reworked to avoid those scenarios but it really depends on which specific api call is taking the time and why.

larhedse commented 1 year ago

Hey!

No worries!

Yes it would be nice to get it to work over all segments.

I solved it for testing this theory by simply moving my HA server (pi4 with SSD) to LAN interface - and move that PC to HA interface. This solved everything. Do note that when my LAN connected PC was disconnected (turned off) HA lost connection with OPNsense and all entities was "unavailable" - that is why I decided to move the HA installation to same segment as OPNsense = LAN interface.... Not perfect, not what I would like, but it did solve my issue.

So feel free to contact me when and if you like to dig into this, if this is possible to solve with different segments (not VLANs, difference interfaces on my OPNsense 8-interface hardware), and I will for sure help with anything I could!

larhedse commented 1 year ago

We could add: strange dropouts in the presence detection. My mobile, and my wife's, drops out from time to time from presence in OPNsense - however that is for sure not true. I currently combine OPNsense plugin presence detection with Unifi detection - the combination works close to 100% correct. So if there is time, we might put a bit onto that also?

travisghansen commented 1 year ago

The presence detection algorithm/approach of this plugin is much better than pretty much any other similarly-styled approach (nmap, ping, etc). What did you put into the config for that?

larhedse commented 1 year ago

My config parameters are 30 second device tracker, and 600 seconds (!) for Consider Home...

Now I use a combo of OPNsense presence and UniFi presence detection. This combo seems to work most of the time.

Just to show what happens, this is from a device (my wifes) logging:

19:21:29 - För 1 minut sedan var hemma 19:11:29 - För 11 minuter sedan var borta 18:55:29 - För 27 minuter sedan var hemma 18:45:29 - För 37 minuter sedan var borta 18:29:29 - För 1 timme sedan var hemma 18:19:29 - För 1 timme sedan var borta 18:03:29 - För 1 timme sedan var hemma 17:53:29 - För 1 timme sedan var borta 17:42:29 - För 2 timmar sedan var hemma 17:24:29 - För 2 timmar sedan var borta 17:20:59 - För 2 timmar sedan var hemma 16:58:59 - För 2 timmar sedan var borta 16:56:59 - För 2 timmar sedan var hemma 16:46:59 - För 3 timmar sedan var borta 16:41:29 - För 3 timmar sedan var hemma 16:14:29 - För 3 timmar sedan var borta 15:47:29 - För 4 timmar sedan var hemma 15:37:29 - För 4 timmar sedan var borta 15:35:29 - För 4 timmar sedan var hemma

Swedish - english translation then: hemma = Home borta = Away timme/timmar = hours minut/minuter = minutes sedan var = ago (or something like that)

So all that time above, my wife (and I) have been home. So there is something going on that well makes this strange. And do observe the 600 seconds (10 minutes!) consider home...

travisghansen commented 1 year ago

Are the logs for the opnsense integration only or for the combo unifi+opnsense detection? Are the phones actively using the network/wifi when home?

The behavior of the integration is:

After all of that

larhedse commented 1 year ago

The log is from OPNsense only. The combo would be boring, everyone is home. No entries at all.

In this case, that mobile is not used, it has a broken screen and there is no way to turn it off for the moment (Samsung Flip Z 3) until it runs out of battery (another day or so I guess - well tomorrow it will go in for service, and it will be back in a month or two...). So no active usage at all. For the record, the UniFi log is also empty - so in this case, there is only the OPNsense presence detection that say away.

I follow how the presence detection works, but if I look into dhcp log (leases) and lease type/status - it seems to follow that more or less. My wifes phone is marked as active/offline - so is also my mobile - but I am still home, my wife is not.... And I have been home all the time, no entries in the log so to speak.

larhedse commented 1 year ago

Ahh.. And there, just like that, I was away all of a sudden - and I am still at home, in UniFi and well IRL....

larhedse commented 1 year ago

I have spent some time to try to understand how Presence Detection works in the py code - now I am NOT a python developer (although I do develop software, but well that is another ballpark so to speak), so some misunderstanding might happen....

First I have been using OPNsense Diagnostics to see how the ARP table in OPNsense updates. It does not. Or rather it does, but the ARP table is emptied very frequent, and it is no way usable as it works in my OPNsense installation (I am using Unbound DNS with DoT - DNS over TLS, and that might be part of the challenge here?). Our mobiles, for example will be droped from that ARP table in NO time at all. I can't even say for how short duration it might appear - but it is for sure to short time frame.

And since I consider the python code to be 100% correct for ARP use, the ARP table in it self from OPNsense is not usable.

What I have not figured out - yet - is how to combine ARP table with for example DHCP v4 Leases, or something. So I will be back when and if I get to a "new algorithm" for this to work. Something is off with the ARP table for the moment at least.

travisghansen commented 1 year ago

The arp table entry for the devices selected by this integration are purposely removed every scan interval. The rest of the arp table should remain. That is what makes this integration quite unique is the ability to not only read the arp table but also remove the entry.

Removing the entry ensures the data is not stale. The device must communicate with the firewall (ie: something on the internet) at least every scan interval (excluding the stay home logic) otherwise the device is no longer home. Most modern mobile devices are doing all kinds of background things that would trigger the kind of required activity.

larhedse commented 1 year ago

Well, it does not work for me - my wife or myself is going in and out all the time. I see how you think, and in theory I concur, but in reality I can not use OPNsens Presence Detection - it simply does not fulfill my requirement. Think if you use presence where one is going in/out every 5 minutes - any lamp depending on this will flash every 5 minutes. While I am still at home that is.

I will most likely remove OPNsense integration, since well I have to have it on the LAN interface (where it is not supposed to sit) and well Presence is not up to my standard.

travisghansen commented 1 year ago

OK, not sure how I can help you and why it doesn't work for you. The logic was discussed ad nauseam on the forums and ultimately landed where it does working remarkably well for all who used/tested it...especially comparing those trying detection of similar means using unifi/ping/nmap/etc.

I'm also not sure what to tell you about the LAN interface. The integration doesn't really care where the OPNsense ui/api is running as long as it is available to the hass instance. This can be on the local network or across the internet..it doesn't care.

larhedse commented 1 year ago

LAN: That was the original posting here in this thread. If I have it on another subnet, it will never detect presence. It only finds devices from time to time when it is sitting on the same subnet as the OPNsense firewall (first interface installed, normaly LAN). At least not in my installation. Why I do not know. I guess it is some thing, but yea not for me to discover.

larhedse commented 1 year ago

There is no need to further investigate into this.