search

home-assistant / operating-system

:beginner: Home Assistant Operating System
Apache License 2.0
4.82k stars 963 forks source link

Hostname conflict, changing published hostname from 'homeassistantxxx' to 'homeassistantxxy' #1163

Closed cben0ist closed 2 years ago

cben0ist commented 3 years ago

Hardware Environment

Home Assistant OS release:

System Health

version 2021.1.1
installation_type Home Assistant OS
dev false
hassio true
docker true
virtualenv false
python_version 3.8.7
os_name Linux
os_version 5.4.86
arch x86_64
timezone America/Toronto
Home Assistant Community Store GitHub API | ok -- | -- Github API Calls Remaining | 4771 Installed Version | 1.9.0 Stage | running Available Repositories | 778 Installed Repositories | 38
Home Assistant Cloud logged_in | true -- | -- subscription_expiration | January 29, 2021, 7:00 PM relayer_connected | true remote_enabled | true remote_connected | true alexa_enabled | true google_enabled | true can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Hass.io host_os | Home Assistant OS 5.10 -- | -- update_channel | stable supervisor_version | 2020.12.7 docker_version | 19.03.13 disk_total | 30.8 GB disk_used | 25.1 GB healthy | true supported | true board | ova supervisor_api | ok version_api | ok installed_addons | Node-RED (7.2.11), InfluxDB (3.7.9), Grafana (5.3.6), Home Assistant Google Drive Backup (0.103.0), Visual Studio Code (2.9.1), Assistant Relay (0.7.4), Mosquitto broker (5.1), Network UPS Tools (0.3.1), OpenZWave (0.8.0), chrony (1.1.3), TasmoAdmin (0.13.1), deCONZ (6.6.2), WireGuard (0.4.0), MariaDB (2.2.1), Terminal & SSH (8.10.0), Portainer (1.3.0), Glances (0.9.1)
Lovelace dashboards | 4 -- | -- mode | storage views | 5 resources | 23
Spotify api_endpoint_reachable | ok -- | --

Journal logs:

Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1742.local IN A 10.1.40.179
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Hostname conflict, changing published hostname from 'homeassistant1742' to 'homeassistant1744'.
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1744\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN TXT ""
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1744\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN SRV 0 0 0 homeassistant1744.local
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1744.local IN A 10.1.40.179
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Hostname conflict, changing published hostname from 'homeassistant1744' to 'homeassistant1752'.
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1752\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN TXT ""
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1752\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN SRV 0 0 0 homeassistant1752.local
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1752.local IN A 10.1.40.179
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Hostname conflict, changing published hostname from 'homeassistant1752' to 'homeassistant1757'.
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1757.local IN A 10.1.40.179
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Hostname conflict, changing published hostname from 'homeassistant1757' to 'homeassistant1759'.
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1757\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN TXT ""
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1757\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN SRV 0 0 0 homeassistant1757.local
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1759.local IN A 10.1.40.179
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Hostname conflict, changing published hostname from 'homeassistant1759' to 'homeassistant1766'.
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1759\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN TXT ""
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1759\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN SRV 0 0 0 homeassistant1759.local
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1766\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN SRV 0 0 0 homeassistant1766.local
Jan 10 15:38:56 homeassistant systemd-resolved[952844]: Detected conflict on homeassistant1766\032\091f82e32391f07425383dd5fbb3a6beafa\093._workstation._tcp.local IN TXT ""

Kernel logs: https://pastebin.com/Ah9rHgSE

Description of problem: I switched to Home Assistant OS in December. I believe this was a 5.6 fresh install at this point and I restored my whole configuration from my previous supervised installation. Since then I have been updating to each update until 5.10 yesterday. I noticed high CPU issues early after the install but I cannot say if that was since the 5.6 initial install or after an update to 5.7 or 5.8. After digging more into the problem I reealized I had a problem specifically with systemd-resolved and all these "Hostname conflict" messages. When I stop this service, goes down by 50% to 80%. Now I'm kind of stuck and not sure where to look at. I'm tempted to recreate a fresh install and migrate again but I'd rather give a shot at finding the actual issue and fixing it. Any help is appreciated. Thank you!

agners commented 3 years ago

I have noticed such issues, but I have several test devices running with hostname homeassistant :see_no_evil: . I once installed a isolated instance, and did not observe the issue, so I was assuming this is something triggered by my messy test setup.

Do you have/had a second instance running in the same network maybe?

cben0ist commented 3 years ago

No, I do not, at least not that I know of :) also seems like such case would be resolved by ha adding that random number in the name, no ? Would that issue come from my network configuration. Has anyone had to do something special on their Ubiquiti Dream Machine Pro to get Home Assistant working properly ? Thanks

cben0ist commented 3 years ago

@agners the issue was not occurring as often recently and it does again on every restart after I have updated to Home Assistant OS 5.12. I have tried renaming my hostname to something else but the problem still persists. I really do not know where to look at now... If you have any idea please let me know. Thanks

cben0ist commented 3 years ago

OK, I have disabled multicast DNS on my UDM pro and that error has stopped right away. I am actually using https://hub.docker.com/r/scyto/multicast-relay so I should have disabled multicast DNS before. That does not explain why it was working with HassOS 5.11 though. I will reopen if that error comes back again. Thank you

TheJulianJES commented 3 years ago

This also only happened after one of the versions at the beginning of this year (or end of last year). I had the same issue with the mDNS Reflector (not repeater) on my UniFi Security Gateway (USG). This really seems like a bug in Home Assistant OS though (or at least something that should be worked around if it's not directly a bug, as many people need to use this, to get Chromecasts working across multiple networks for example).

klenaers commented 3 years ago

I eventually blocked the mDNS-traffic on the OS. I have a Ubiquiti EdgeRouter X-firewall with mDNS repeater active and multiple VLAN's.

For now only following iptables-command stops the CPU-load and memory usage of the systemd-resolved-process.

iptables -A INPUT -p udp -m udp --dport 5353 -j DROP

And afterwards you only see that there is a naming-conflic but not the negative effect of the loop. This command shows that the mDNS-packets are blocked.

iptables -L -nvx

This is not a real solution but helps until there a real solution on disableling the mDNS-broadcast of homeassistant.local But hopefully this helps you until it is resolved!

I also tried to save the iptables-rule with iptables-save and it looks to be persistant between reboots.

TheJulianJES commented 3 years ago

Is it perhaps possible to reopen this issue? Seems like quite a few setups are affected. Many users probably don't even notice.

klenaers commented 3 years ago

@TheJulianJES : indeed. I did some more testing and my iptables-scenario doesn't work. After the rule is activated chromecast doesn't work anymore. I noticed this issue when after some time the system went unresponsive and I needed to reboot everything so it worked again. First I was looking at the addons but eventually I noticed these messages in the log.

klenaers commented 3 years ago

Hi, I tried something else and now also chromecast seems to work again! No conflict-messages either in the log.

I went to the folder /mnt/overlay/etc/NetworkManager/system-connections and edited this file Supervisor eth0.nmconnection and changed the line mdns=2 to mdns=1. What I read in the documentation of NetworkManager is this: "resolve" (1) do not register hostname but allow resolving of mDNS.

I rebooted my system several times and the CPU-load stays low and no messages anymore in the log of conflicting names. I'll let you know in a few days if i notice any issues.

klenaers commented 3 years ago

After changing the mdns-parameter, I noticed issues with my esphomes that aren't resolved anymore(what seems logical). I checked the edgerouter-config and noticed that the mdns-reflector & -repeater where active. I disabled mdns reflector. This morning, I enabled mdns=2 in the NetworkManager. I saw something in the log that the mdns-info is old and it will be refreshed(or something like that). Since then I don't see any host conflicts anymore in the log!! CPU-load is back to normal. This issue is fixed now for me!

I tried to replicate the real issue(rebooting everthing, enable the mdns-reflector on edderouter, ....) and I couldn't replicate the issue anymore.

What's your experience? This issue was bugging me for a long time without really knowing what was the source. Only when I started monitoring the free memory and processor use, I noticed this.

- platform: systemmonitor
  resources:
    - type: disk_use_percent
      arg: /config
    - type: memory_free
    - type: disk_use_percent
    - type: processor_use
    - type: last_boot
TheJulianJES commented 3 years ago

Disabling mDNS Reflector on the USG/UDM stops this issue from occurring (as noted above). However, sometimes users need these features and many probably have it enabled and just don't notice Home Assistant OS using CPU. Did you try restarting HassOS after having re-enabled the reflector?

Also, are you using a "domain name" on your EdgeRouter? (Something like local or localdomain)

klenaers commented 3 years ago

@TheJulianJES I disabled the mDNS reflector but the mDNS repeater stayed active all the time. So chromecast, ... that use mDNS kept working through the firewall in the other VLANS.

After rectivating the mDNS reflector I didn't notice negative impact. I rebooted everything(systemd-resolved, home assistant core, HassOS and edgerouter) and check with every restarted process if the conflict messages reappeared(but they didn't).

I use a seperate domain ".home.me". So no local or localdomain.

I also tried to set the parameter in the NetworkManager-file mdns=1 and back to mdns=2 to see if I got the same message that the mdns-info was old and it needed to be refreshed. But I didn't see it in the logs this time.

Everything is still working without isseus. So no real idea what fixed this issue eventually.

agners commented 3 years ago

Keep in mind that this is intended behavior if there is a name conflict in the network.. So could it be that a new test installation or similar has been run at one point?

klenaers commented 3 years ago

@agners I booted the raspberry pi with a the 2de home assistant-environment. Everything kept working. I changed the edgerouter-setting withe the reflector active and nothing changed. I renamed the hostname a few times and one moment there was a loop on the 2de system. I never saw any issues on my production server. Most of the time there where a few conflict messages but they stopped very fast when the following line appeared: homeassistant systemd[1]: systemd-hostnamed.service: Succeeded.

This is the message I got the one time when there where a lot of conflict-messages:

Jun 07 17:36:09 homeassistant systemd[1]: Stopped Network Name Resolution.
Jun 07 17:36:09 homeassistant systemd[1]: Starting Network Name Resolution...
Jun 07 17:36:09 homeassistant systemd-resolved[46552]: Positive Trust Anchors:
Jun 07 17:36:09 homeassistant systemd-resolved[46552]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 07 17:36:09 homeassistant systemd-resolved[46552]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jun 07 17:36:09 homeassistant systemd-resolved[46552]: Using system hostname 'homeassistant'.
Jun 07 17:36:09 homeassistant systemd-resolved[46552]: mDNS-IPv4: There appears to be another mDNS responder running, or previously systemd-resolved crashed with some outstanding transfers.
Jun 07 17:36:09 homeassistant systemd[1]: Started Network Name Resolution.

I can't replicate the original issue. But I beleave this was the message I got when everything started working again.

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

DanaGoyette commented 2 years ago

I'm still seeing continuous hostname conflicts if I use the avahi reflector on my OpenWRT router (needed for the sake of VLANs). Is systemd-resolved not smart enough to release that all those addresses are its own addresses?

Apr 04 19:00:12 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:12 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant' to 'homeassistant2'.
Apr 04 19:00:12 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant2.local IN AAAA fe80::5054:ff:fee7:c71
Apr 04 19:00:12 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant2' to 'homeassistant6'.
Apr 04 19:00:12 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant6.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:12 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant6' to 'homeassistant11'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant11.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant11' to 'homeassistant14'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant14.local IN AAAA fe80::5054:ff:fee7:c71
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant14' to 'homeassistant16'.
Apr 04 19:00:13 homeassistant.local ghcr.io/esphome/esphome-hassio-aarch64:2022.3.2/addon_5c53de3b_esphome[357]: 2022-04-04 12:00:13,192 INFO 200 GET /devices (0.0.0.0) 16.74ms
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant16.local IN AAAA 2600:xxxx:7a00:4b40::492
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant16' to 'homeassistant21'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant21.local IN A 192.168.3.140
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant21' to 'homeassistant23'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant23.local IN AAAA fe80::5054:ff:fee7:c71
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant23' to 'homeassistant26'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant26.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant26' to 'homeassistant31'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant31.local IN AAAA fe80::5054:ff:fee7:c71
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant31' to 'homeassistant40'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant40.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant40' to 'homeassistant42'.
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant42.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:13 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant42' to 'homeassistant52'.
Apr 04 19:00:14 homeassistant.local systemd-resolved[307]: Detected conflict on homeassistant52.local IN AAAA fd98:dd33:656a:40::492
Apr 04 19:00:14 homeassistant.local systemd-resolved[307]: Hostname conflict, changing published hostname from 'homeassistant52' to 'homeassistant57'.
agners commented 2 years ago

So is Home Assistant OS multi-homed in that setting (in multiple VLANs)?

Could be that systemd-resolved is not smart enough. I guess that should be fairly straight forward to test with a systemd based Linux distribution.

DanaGoyette commented 2 years ago

In my case, the homeassistant is in a VM attached to a bridge on the host, with none of the on-wire VLANs configured on host or guest. I'm running the avahi reflector on my router itself.

I'm currently trying using the reflection-filters option to filter which services get reflected, and that seems to be helping... but then I have to list each service I want shown between VLANs. I wonder what service systemd-resolved is using to decide it's seeing a duplicate?

Really, the bug looks more like a systemd-resolved bug than a Home Assistant bug, so somebody (or I) should file it upstream.

agners commented 2 years ago

In my case, the homeassistant is in a VM attached to a bridge on the host, with none of the on-wire VLANs configured on host or guest. I'm running the avahi reflector on my router itself.

If your system is not dual-homed, why is the reflector sending back the same mDNS messages on that network? From how I understand how reflectors should work is they should forward from one network to the other (and vice-versa)...

Really, the bug looks more like a systemd-resolved bug than a Home Assistant bug, so somebody (or I) should file it upstream.

I guess it can be debated if that is a bug in first place. But I agree, it is a problem with systemd-resolved itself. Ideally you reproduce it with a different distribution (ideally one which is used by systemd folks like Fedara), and open an issue in their repository.

andrelung commented 1 year ago

Had the same issue with a pfsense router. I initially suspected a problem with my second LAN interface, but simply disabling "Enable reflection - Repeat mdns packets across subnets" in the avahi-package solved the problem.

image

Thanks!

noci2012 commented 1 year ago

The real problem is a reflector sending packets back to the originating network imho. With a multi vlan network, one needs some repeater formarding mDNS packets to all other vlans.

fresnoboy commented 1 year ago

I think this is the underlying problem with https://github.com/esphome/issues/issues/4326 . I have multiple VLANs (I believe in the principle of least privilege), and use PFsense to enable mdns forwarding for chromecasts across multiple VLAN's. Because of needing to have HA have direct access to one of the IOT VLAN's in addition to the main server VLAN (supporting some Tuya air sensors that won't talk to the integration locally unless HA is on the same subnet as the devices, I have had HA multihomed.

Something appears to have changed with HAOS behavior in the last couple of releases, because now esphome shows device reachability going in and out, as well as editing of configs abruptly closing without saving, and I see numerous of these hostname conflict error messages in the log.

Removing the 2nd VLAN interface from HA restores stability (it runs in an ESXi guest VM). Interestingly, simply unchecking the network interface in the networking config doesn't fix it, you have to remove the interface, so I suspect something may be going on in the base HAOS config and not the container.

I can't turn off the multicast processing in PFSense because I have about 5 VLAN's where I need to make mDNS work across to deal with chromecasts, SONOS, and various user devices (servers, adult and kid devices are all segmented on different VLANs), and I don't want to have to make HAOS try and do the briding by putting it directly on all those VLANs. But I can't talk to those Tuya devices without a direct interface on that network.

Everything was stable and running fine a couple months ago, so something changed in HA to cause this problem as the PFSense config has not been changed nor the HA config for many months.

agners commented 1 year ago

and use PFsense to enable mdns forwarding for chromecasts across multiple VLAN's.

and

Removing the 2nd VLAN interface from HA restores stability (it runs in an ESXi guest VM).

If you do both, make HA dual homed and at the same time forward mDNS packages between the two network, HA will receive mDNS announcements twice, doesn't it? How should it know which one to trust/how can it de-duplicate? There isn't really a way... The way you are setting up your network doesn't really work.

Everything was stable and running fine a couple months ago, so something changed in HA to cause this problem as the PFSense config has not been changed nor the HA config for many months.

HAOS gets new software versions every now and then. New software versions might behave slightly different, especially in edge cases (e.g. receiving the same mDNS announcements twice through two different network interface).

phiberoptick commented 1 year ago

If you do both, make HA dual homed and at the same time forward mDNS packages between the two network, HA will receive mDNS announcements twice, doesn't it? How should it know which one to trust/how can it de-duplicate? There isn't really a way... The way you are setting up your network doesn't really work.

Yes, you're correct. However, some devices, such as Samsung TVs will not allow another device to connect to them and control them from another subnet. Even with mDNS. I've ran into this with a few integrations and actually run home assistant with a second interface alias on another VLAN.

While this is the quickest way, one day I might actually get around to either manually setting rules on my Unifi equipment to block mDNS packets on only the interface alias that's on the IoT VLAN or modify the systemd networking to disable it on just that interface. I worry that an update would negate the changes so will probably go with rules in the router instead.

agners commented 1 year ago

mDNS is built on link-local multicast. Link-local multicast is meant to be, surprise, link local. Any mDNS reflector or other "link-local multicast extension to another subnet" is a hack at the end of the day.

Since nowadays many IoT protocols embrace protocols built on link-local multicast (like mDNS/DNS-SD), VLAN is kinda a bad option.

IMHO, instead of spending time making VLAN work, rather embrace zero trust networking: Just make sure all your devices are secured by default (no anonymous logins, safe passwords, encrypted communication etc.). Consider your network Internet. It's quite freeing :smile:

If you have legacy stuff or stuff you can't secure that way, you can still use a VLAN to move that particular device off of your main LAN.

phiberoptick commented 1 year ago

mDNS is built on link-local multicast. Link-local multicast is meant to be, surprise, link local. Any mDNS reflector or other "link-local multicast extension to another subnet" is a hack at the end of the day.

Since nowadays many IoT protocols embrace protocols built on link-local multicast (like mDNS/DNS-SD), VLAN is kinda a bad option.

IMHO, instead of spending time making VLAN work, rather embrace zero trust networking: Just make sure all your devices are secured by default (no anonymous logins, safe passwords, encrypted communication etc.). Consider your network Internet. It's quite freeing :smile:

If you have legacy stuff or stuff you can't secure that way, you can still use a VLAN to move that particular device off of your main LAN.

The problem with that is that there are no IoT devices that are secure. The biggest security risk on a network are the IoT devices that you have no control of. Especially the ones that contact anything in the cloud. Look at Eufy, Ring, etc.

The only reason my iot devices can communicate with anything on my network is so that I can cast to them or otherwise control them like for instance the TV or sound bar etc. And in that case, I would prefer that they are not allowed to talk to anything else other than the specific hosts/ports that I want them to talk to. I also block them from calling home except for the hosts that absolutely stop them from working. The number one blocked DNS queries on my network are all tracking hosts at Microsoft, Amazon, Samsung etc.

In a perfect world, they wouldn't talk to the cloud at all and have absolutely no internet access and only be allowed to speak to the specific hosts and ports that I want them to.

Unfortunately, that would require the manufacturers to give up all that add tracking money which they will never do. Or it would end up doubling, tripling or more the price of the devices.

Until then, I'll use whatever means I need to to completely segregate them from anything and everything else on my network that I don't need them to be able to talk to. And right now that's usually either vlans or a whole lot of manual firewall rules. Any of my IoT devices that do not absolutely need internet are not even given a gateway when DHCP assigns them their address which is static. If there's an update that's needed and no way for me to push it to the device locally I allow an exception to the specific host that they need to talk to for the update and only for the amount of time it takes for them to update.

I also segment all mobile devices like cell phones and tablets etc to their own VLAN that cannot communicate to any of the other devices or vlans unless specifically allowed via rules. I don't need one of the kids phones able to see all of the devices or possibly control them. That includes casting to them without my knowledge and subsequently rules to allow it.

Any five-year-old can download a script off the internet and totally own 90% of people's networks because they just put stuff on them and have no idea about it and rely on the manufacturer to actually secure them, update them, patch vulnerabilities etc. When the majority of hardware released is lucky if it's supported for a few years when people end up using it for much longer. There comes a point when it's completely called for to sacrifice ease of use for security. Especially for those people that have no idea. Which is the majority of the world unfortunately. I'm sure everybody knows friends or family that are still running Windows 7 and postponing updates because they don't want to be bothered with reboots etc. That's all it takes to have all of your personal information, finance information etc stolen. And when that happens it's the fault of the administrator or person in charge of the network who just relied on the manufacturer and trusted that they would do what they needed to do.

Just putting a device on your network and allowing it to do whatever it wants is almost as bad as just posting your configs and passwords on the internet.

fresnoboy commented 1 year ago

mDNS is built on link-local multicast. Link-local multicast is meant to be, surprise, link local. Any mDNS reflector or other "link-local multicast extension to another subnet" is a hack at the end of the day. Since nowadays many IoT protocols embrace protocols built on link-local multicast (like mDNS/DNS-SD), VLAN is kinda a bad option. IMHO, instead of spending time making VLAN work, rather embrace zero trust networking: Just make sure all your devices are secured by default (no anonymous logins, safe passwords, encrypted communication etc.). Consider your network Internet. It's quite freeing 😄 If you have legacy stuff or stuff you can't secure that way, you can still use a VLAN to move that particular device off of your main LAN.

The problem with that is that there are no IoT devices that are secure. The biggest security risk on a network are the IoT devices that you have no control of. Especially the ones that contact anything in the cloud. Look at Eufy, Ring, etc.

The only reason my iot devices can communicate with anything on my network is so that I can cast to them or otherwise control them like for instance the TV or sound bar etc. And in that case, I would prefer that they are not allowed to talk to anything else other than the specific hosts/ports that I want them to talk to. I also block them from calling home except for the hosts that absolutely stop them from working. The number one blocked DNS queries on my network are all tracking hosts at Microsoft, Amazon, Samsung etc.

In a perfect world, they wouldn't talk to the cloud at all and have absolutely no internet access and only be allowed to speak to the specific hosts and ports that I want them to.

Unfortunately, that would require the manufacturers to give up all that add tracking money which they will never do. Or it would end up doubling, tripling or more the price of the devices.

Until then, I'll use whatever means I need to to completely segregate them from anything and everything else on my network that I don't need them to be able to talk to. And right now that's usually either vlans or a whole lot of manual firewall rules. Any of my IoT devices that do not absolutely need internet are not even given a gateway when DHCP assigns them their address which is static. If there's an update that's needed and no way for me to push it to the device locally I allow an exception to the specific host that they need to talk to for the update and only for the amount of time it takes for them to update.

Just putting a device on your network and allowing it to do whatever it wants is almost as bad as just posting your configs and passwords on the internet.

+1 Totally agree. Zero trust requires proper endpoint management and security, but in the IOT space endpoints are the problem for all the reasons just mentioned. We don't have the option of using only secure IOT devices because they don't seem to make those.

And this is only going to get worse with Matter. At least with zigbee the cheap chinese etc... devices don't have IP access to anything, and if I trust the zigbee coordinator code (I use the open source zigbee2mqtt), you have limited security exposure. But with matter everything is IP connected, and that's a real problem because I don't trust any of those devices.

VLAN's are going to be even more important with Matter and will require even more sophisticated network management to safely integrate them in our homes.

agners commented 1 year ago

Let's stay on topic here.

In the end I stay to my stance: Use standards the way they are engineered. Link-local multicast is built for link-local.... If you bork your network that this no longer the case, then don't blame developers. It is on you, you abuse the standard.

dsander commented 1 year ago

To me this all seems on topic. Yes this might not be how the standard was developed, but as far as I know there also often isn't a good way to avoid having a mDNS repeater in the network. Like others said "zero trust" does only work in one direction, without network segregation one, well, has a hard time separating devices from each other.

Having HA only in one network certainly works, but then all traffic has to go through the firewall. Giving it direct access to all networks made my experience better (and removed the need for the firewall to be up for HA to work properly).

What I am doing on my supervised installation is telling systemd-resolve to disable mdns on all but the first interface:

for i in {1..4}; do /usr/bin/systemd-resolve --set-mdns=no --interface=eth$i; done

Because something resets it (never figured out what does this) it is running every couple of minutes in a cron tab. Certainly not pretty but it works.

It would be awesome if we could somehow tell the HA supervisor on which interfaces systemd-resolve should be using mdns.

agners commented 1 year ago

To me this all seems on topic. Yes this might not be how the standard was developed, but as far as I know there also often isn't a good way to avoid having a mDNS repeater in the network. Like others said "zero trust" does only work in one direction, without network segregation one, well, has a hard time separating devices from each other.

In my experience, after enabling mDNS repeater you run into more problems: How do you deal with names/services announced using IPv6 link local addresses? You can't simply forward those packets, as the link local address won't be valid on the other network. And Matter being IPv6 based does make use of link-local addresses....

Also IPv6 Neighborhood Discovery and RIO, required for Thread Border Routers, are protocols which are designed to work link-local.

In the end, VLANs are also not ideal: You still "group" devices, and devices within a VLAN can talk to each other.

IMHO, nicer would be to have all the devices in one LAN, and filter traffic per device (based on port/MAC, e.g. using OpenFlow). These type of filtering and device specific routing is definitely done in data center SDNs, but not sure if one ever attempted to implement such networking architectures in home networks. :sweat_smile:

What I am doing on my supervised installation is telling systemd-resolve to disable mdns on all but the first interface:

for i in {1..4}; do /usr/bin/systemd-resolve --set-mdns=no --interface=eth$i; done

Because something resets it (never figured out what does this) it is running every couple of minutes in a cron tab. Certainly not pretty but it works.

Yeah this is definitely a hack. Afaik, systemd-resolve is socket activated, so that is probably the reason you have to run it all the time.

It should be possible to influence mDNS settings per interface. it could be part of the Supervisor networking. Patch welcome :wink:

dsander commented 1 year ago

In my experience, after enabling mDNS repeater you run into more problems: How do you deal with names/services announced using IPv6 link local addresses?

Easy, I don't have IPv6 in the local network. Having two WANs, I figured there would be no gain of NATing IPv6 to the LAN. 😄

IMHO, nicer would be to have all the devices in one LAN, and filter traffic per device (based on port/MAC, e.g. using OpenFlow). These type of filtering and device specific routing is definitely done in data center SDNs, but not sure if one ever attempted to implement such networking architectures in home networks. 😅

Have you implemented that approach? It sounds like a lot of configuration work (which would be manual for me because tplink omada isn't easily automateable). How do you deal with additions to the guest WLAN? Having to add the IPs/MACs of every new guest to the configuration when someone asks for the wifi password sounds a bit impractical 😛

It should be possible to influence mDNS settings per interface. it could be part of the Supervisor networking. Patch welcome 😉

Great, I will see if I can find the right places to make it configurable. If/when it makes it to the UI will would be a checkbox in this area?

image
phiberoptick commented 1 year ago

In my experience, after enabling mDNS repeater you run into more problems: How do you deal with names/services announced using IPv6 link local addresses?

Easy, I don't have IPv6 in the local network. Having two WANs, I figured there would be no gain of NATing IPv6 to the LAN. 😄

That tracks with my network as well. I explicitly disable IPv6 support on all my devices and hosts .

IMHO, nicer would be to have all the devices in one LAN, and filter traffic per device (based on port/MAC, e.g. using OpenFlow). These type of filtering and device specific routing is definitely done in data center SDNs, but not sure if one ever attempted to implement such networking architectures in home networks. 😅

Have you implemented that approach? It sounds like a lot of configuration work (which would be manual for me because tplink omada isn't easily automateable). How do you deal with additions to the guest WLAN? Having to add the IPs/MACs of every new guest to the configuration when someone asks for the wifi password sounds a bit impractical 😛

I agree with @agners on this one. I also use MAC-based ACLs on all the networks. So yes, anyone that connects to my Wi-Fi, I have to manually add their MAC address to the network. As well as make sure their device isn't using random MAC addresses. It really isn't that bad. That could also be because I don't let just anyone who asks use the Wi-Fi. I don't run a guest network. If you don't live here come over more than once every few weeks, you don't need access to my network. This includes wired ports/hosts as well.

It should be possible to influence mDNS settings per interface. it could be part of the Supervisor networking. Patch welcome 😉

Great, I will see if I can find the right places to make it configurable. If/when it makes it to the UI will would be a checkbox in this area?

image

That would be awesome. However, I wouldn't complain even if it was only accessible via console or SSH using " ha". In fact, that's how I put my home assistant on multiple VLANs back before it was even an option in the GUI.

phiberoptick commented 1 year ago

Just wanted to drop another comment. I went in to my homeassistant instance and manually disabled mdns replies. Just on a single interface it will still resolve but it won't reply so the other interface gets the reply. It's literally four words in a resolvectl command.

While it won't help anyone with out root access to their instance, it does prove a concept. Seems that adding that command into any of the existing network configuration options would allow people to manually select whether that interface should ignore mDNS or not...

The journal just scrolling and complaining like mad...

Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082570.local IN A 192.168.0.10                
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082570' to 'homeassistant3082574'.
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082574.local IN A 192.168.0.10                
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082574' to 'homeassistant3082578'.
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082578.local IN A 192.168.0.10
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082578' to 'homeassistant3082582'.                                                                                                                  
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082582\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN TXT ""                                                                                                
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082582\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN SRV 0 0 0 homeassistant3082582.local
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082582.local IN A 192.168.0.10
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082582' to 'homeassistant3082586'.
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082586\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN TXT ""
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082586\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN SRV 0 0 0 homeassistant3082586.local
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082586.local IN A 192.168.0.10
Jul 28 19:26:44 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082586' to 'homeassistant3082593'.
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082593\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN TXT ""
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082593\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN SRV 0 0 0 homeassistant3082593.local
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082593.local IN A 192.168.0.10
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082593' to 'homeassistant3082596'.
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082596.local IN A 192.168.0.10
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082596' to 'homeassistant3082601'.

Verify the interfaces.

# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ed:05:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.10/23 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1699:563b:5aae:73c4/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
# ip addr show wlan0
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether dc:a6:32:ed:05:a6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.55.135/24 brd 192.168.55.255 scope global noprefixroute wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::9c1f:334d:2c0c:7434/64 scope link noprefixrout
       valid_lft forever preferred_lft forever
# resolvectl mdns
Global: yes
Link 2 (eth0): yes
Link 3 (wlan0): yes
Link 4 (docker0): no
Link 5 (hassio): no
Link 7 (veth47a5d5f): no
Link 9 (veth38b87d4): no
Link 11 (veth1fe34bd): no
Link 13 (veth2d21e8d): no
Link 15 (veth35a554d): no
Link 17 (veth446b161): no
Link 19 (veth739f2be): no
Link 21 (vethb0956cf): no
Link 23 (veth3ccc358): no
Link 25 (veth9a4fc96): no
Link 27 (veth651a900): no
Link 29 (vethbd7b69a): no
Link 31 (veth8835fec): no
Link 33 (vethf40dfca): no
Link 35 (vethb4aa6ee): no
Link 37 (veth2c1b60e): no
Link 39 (veth662913e): no
Link 41 (veth80e81a1): no

Setting the Ethernet interface to only resolve mDNS and not reply/respond.

# resolvectl mdns 2 resolve
# resolvectl mdns
Global: yes
Link 2 (eth0): resolve
Link 3 (wlan0): yes
Link 4 (docker0): no
Link 5 (hassio): no
Link 7 (veth47a5d5f): no
Link 9 (veth38b87d4): no
Link 11 (veth1fe34bd): no
Link 13 (veth2d21e8d): no
Link 15 (veth35a554d): no
Link 17 (veth446b161): no
Link 19 (veth739f2be): no
Link 21 (vethb0956cf): no
Link 23 (veth3ccc358): no
Link 25 (veth9a4fc96): no
Link 27 (veth651a900): no
Link 29 (vethbd7b69a): no
Link 31 (veth8835fec): no
Link 33 (vethf40dfca): no
Link 35 (vethb4aa6ee): no
Link 37 (veth2c1b60e): no
Link 39 (veth662913e): no
Link 41 (veth80e81a1): no

The logs as I made the change:

Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082601\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN TXT ""
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082601\032\0918089f76a381a40598714f1a85cc2a97a\093._workstation._tcp.local IN SRV 0 0 0 homeassistant3082601.local                                                                  Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082601.local IN A 192.168.0.10                Jul 28 19:26:45 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082601' to 'homeassistant3082605'.                                                                                                                  Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082605.local IN A 192.168.0.10                Jul 28 19:26:45 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082605' to 'homeassistant3082608'.
Jul 28 19:26:45 homeassistant systemd-resolved[450]: Detected conflict on homeassistant3082608.local IN A 192.168.0.10                Jul 28 19:26:45 homeassistant systemd-resolved[450]: Hostname conflict, changing published hostname from 'homeassistant3082608' to 'homeassistant3082612'.
Jul 28 19:26:45 homeassistant systemd-resolved[450]: eth0: Bus client set MulticastDNS setting: resolve                               Jul 28 19:26:47 homeassistant addon_core_mosquitto[590]: error: received null username, clientid or topic, or access is equal or less than 0 for acl check
Jul 28 19:26:47 homeassistant addon_core_mosquitto[590]: error: received null username, clientid or topic, or access is equal or less than 0 for acl check
Jul 28 19:26:47 homeassistant addon_core_mosquitto[590]: error: received null username, clientid or topic, or access is equal or less than 0 for acl check
Jul 28 19:26:47 homeassistant addon_core_mosquitto[590]: error: received null username, clientid or topic, or access is equal or less than 0 for acl check
Jul 28 19:26:47 homeassistant addon_core_mosquitto[590]: error: received null username, clientid or topic, or access is equal or less than 0 for acl check
Jul 28 19:26:47 homeassistant addon_core_mosquitto[590]: error: received null username, clientid or topic, or access is equal or less than 0 for acl check

No more complaining. I'm going to run it for a little while like this just to see integration stop working or other gotchas pop up. But assuming they don't, it can be done with the command or with a drop-in config, (like are already being used in home assistant systemd), that with you the same thing.

Thoughts?

t3hk0d3 commented 10 months ago

I had simillar symptoms:

  1. Constant spam about hostname conflict.
  2. High CPU usage by systemd-resolved and mdnsd: Screenshot_20231117_195218
  3. resolvectl mdns returned on mode for all active interfaces. Turning them off by resolvectl mdns 2 resolve/off was fruitless, because NetworkManager turns mdns on immediately.

I've got it solved following way:

  1. Set all connections to resolve-only mode (you might leave one of connections active, to keep homeassistant.local working) 
    > nmcli connection modify <connection> connection.mdns resolve
  2. Restart Network Manager 
    > systemctl restart NetworkManager.service
  3. Confirm it working 
    > resolvectl mdns
Global: yes
Link 2 (eth0): resolve
Link 3 (wlan0): no
Link 4 (usr0): resolve
Link 5 (iot0): resolve
Link 8 (docker_gwbridge): no
Link 9 (hassio): no

After that a constant stream of conflicts got resolved, and no more CPU hogging by systemd-resolved and mdnsd.

Screenshot_20231117_205836

jhansche commented 9 months ago

@t3hk0d3 These commands do resolve the mdns issues for multi-homed interfaces, but I see that even after running these commands (using either nmcli or resolvectl, and even setting mdns=1 in /etc/NetworkManager/system-connections for the "Supervisor" connection, it still gets reset and turned back on at some point - appears to be when the machine is rebooted, as I just restarted HA and it remained set to resolve. But after a reboot it gets reset back to on/yes.

Did you find a way to make the mdns=resolve config persistent?

NOTE: I'm not using HAOS, but a supported supervised installation on debian

agners commented 9 months ago

Constant spam about hostname conflict.

Did you run two Home Assistant instances with the same name in the network? Or maybe multi-homed installation with a mDNS reflector on your switch?

In general, systsemd-resolved should be responsible to announce the hostname via mDNS (so that http://homeassistanat.local works).

t3hk0d3 commented 9 months ago

@agners This issues happens when there is multiple network interfaces (for example for different VLANs), and router (for example UniFi) proxies/reflects multicast traffic between networks.

Solution is here

agners commented 9 months ago

Ok. We won't change that by default though as people need the hostname announcement via mDNS.

justfalter commented 8 months ago

If you've got NextDNS installed on your unifi gateway and are experiencing this issue: NextDNS might be consuming a lot of CPU and Memory on your gateway due to the constant stream of new hostnames https://github.com/nextdns/nextdns/issues/886.

The best workaround I could come up with involved configuring the Unifi gateway to block all multicast DNS traffic coming from the Home Assistant network interfaces, preventing unifi from forwarding its mdns queries which caused HA/systemd-resolved to believe there was a conflict. My workaround should work regardless of whether you're using NextDNS, or not: https://github.com/nextdns/nextdns/issues/886#issuecomment-1890793441

pcmike commented 5 months ago

@t3hk0d3 These commands do resolve the mdns issues for multi-homed interfaces, but I see that even after running these commands (using either nmcli or resolvectl, and even setting mdns=1 in /etc/NetworkManager/system-connections for the "Supervisor" connection, it still gets reset and turned back on at some point - appears to be when the machine is rebooted, as I just restarted HA and it remained set to resolve. But after a reboot it gets reset back to on/yes.

Did you find a way to make the mdns=resolve config persistent?

NOTE: I'm not using HAOS, but a supported supervised installation on debian

Same here. This change is not a permanent solution as it's reverted right back upon a system reboot. How do you get this change to be sticky? @t3hk0d3

kkhalme commented 4 months ago

I also run into this issue today. I have a multi-home HAOS on Rpi5. The main issue is the inability to persistently save the NetworkManager configs. They get reset after each host reboot.

Stan-Gobien commented 4 months ago

I have multi-homed HAOS. All my IOT devices are in a separate VLAN. On my PC in my normal LAN the hostname.local addresses didn't work. That was annoying since ESPhome doesn't show the IP of the device.

So I specifically enabled mDNS repeater on my OPNsense router and it worked perfectly. I could browse to hostname.local addresses that are in a separate VLAN.

But however now I noticed this hostname conflict. I have disabled mDNS repeater for now. But this is a step backwards again.

giannoug commented 3 months ago

Same issue here with UDM SE and HAOS being in two VLANs. I had to disable "IoT Auto-Discovery" which made the issue go away but now I can't resolve IoT .local domains (e.g. ESPHome) from my main network.

mattsch commented 1 month ago

Setting the interface's IPv4 configuration to static and then making the manual NetworkManager change does seem to persist across reboots. At least for a wired interface. It would be really nice to expose that configuration somewhere in the stack as well. I would have expected the Configure which network adapters integrations will use option to do the trick at least.