Matter-over-Thread device unresponsive

[djandrew2005]

The problem

When adding a Matter device to Home Assistant the device becomes unresponsive

What version of Home Assistant Core has the issue?

core-2023.4.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Matter

Link to integration documentation on our website

https://www.home-assistant.io/integrations/matter/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

I use a Google Nest Hub as a Thread Border Router.

I tried the Home Assistant OS 10.0.rc3 which has a potential fix but the problem still exists

[home-assistant[bot]]

Hey there @home-assistant/matter, mind taking a look at this issue as it has been labeled with an integration (matter) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `matter` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign matter` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

matter documentation matter source _{^{(message by IssueLinks)}}

[cburghardt]

Same thing here, OS 10.0.rc4 and HA is used as a border router.

[marcelveldt]

If a device becomes unresponsive (so the state changes to unavailable), the cause is in 99% of the cases network related, more specifically multicast DNS related. Take a look at your network, no inter vlan routing is supported (yet).

Also myself and others have confirmed that even having a multicast responder active on your network is enough to have thread go completely mad. For example on Unifi networks this option is known as "multicast DNS". Disable it, even if your thread border routers are on the same (v)lan.

Also bad implementations of IGMP snooping can cause issues. Even the Linux NetworkManager had issues handling the routing table from thread BR's, hence the patch in HAOS 10.

[cburghardt]

@marcelveldt I don't use vlans and I don't have any mDNS responders on my network (afaik). How can I verify if something is wrong there? My router is a fritz box.

[djandrew2005]

I turned off multicast completely and the issue persists

[cburghardt]

In my case the device is available most of the time. But several times per day it becomes unresponsive for about 10 minutes and then recovers. As mentioned I don't have any mdns repeaters active, no vlans or anything in that area. The only router is a fritz box, the thread Radio is a sky connect stick so I really don't see why my network would cause issues.

[marcelveldt]

@cburghardt Well, Matter is an IPv6 based protocol and even Thread based devices use IP communication and multicast for discovery so it has everything to do with networking ;-) So even a bad implementation of IGMP on a switch can cause it to go havoc.

Check if you can discover something that happens on the same time as your device becoming unresponsive. Also make sure that you update to the 3.3.1 version of the Matter add-on because we fixed a few bugs yesterday.

@djandrew2005 I think your issue (the device becomes unavailable after adding it) should be fixed with add-on version 3.3.1

[cburghardt]

@marcelveldt bare with me if that is a different issue but what I found in the logs when the device becomes unresponsive (version 3.3.1, this morning) is the following:

2023-04-19 05:30:38 core-matter-server chip.DMG[126] ERROR Subscription Liveness timeout with SubscriptionID = 0xae9351c4, Peer = 01:0000000000000001 2023-04-19 05:30:48 core-matter-server chip.clusters.Attribute[126] ERROR 'No Schema found for Attribute 1/319486977/319422467' Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/chip/clusters/Attribute.py", line 750, in _handleReportEnd attribute_path = TypedAttributePath(Path=change) File "/usr/local/lib/python3.10/site-packages/chip/clusters/Attribute.py", line 150, in __init__ raise KeyError(f"No Schema found for Attribute {Path}") KeyError: 'No Schema found for Attribute 1/319486977/319422467'

That no schema thing repeats until the device becomes available again.

[marcelveldt]

@cburghardt the problems are related but strictly taken not the same. We were aware of an issue that the device becomes unresponsive (or just displaying the incorrect state) right after adding it. That is fixed in addon version 3.3.1. by working around that schema not found error (it is now just logged but not crashing)

Your situation is a bit different though as it looks like the device is sending some data the SDK can't parse. Strangely enough normally speaking these Schema not found errors can be ignored because its manufacturer specific data and should not be needed for the primary use.

So, the device gets state "unavailable" in HA or is the state set to something else? You are 100% certain the issue is exactly the same after updating to add-on version 3.3.1 ?

EDIT: I now carefully read your message and spot the first log line: "ERROR Subscription Liveness timeout with SubscriptionID" - meaning the subscription timed out. So back at the device or network related cause again.

[jusicgn]

Don't know if it helps someone: I also had issues with Matter over Thread devices becoming unavailable. I even disabled the OTBR on my multiprotocol stick because I thought this could be the cause of the problems as I also have an HomePod mini running in my network for the Apple stuff.

Then I read about the IGMP problems. And I also have a FritzBox (model 4060) in my network. Googled a bit and found that the FritzBox uses IGMP for "optimizing Wi-Fi traffic for live TV". I disabled the option in the Wi-Fi settings (in German): "WLAN-Übertragung für Live TV optimieren".

Can't say it for sure but my feeling is that this increased the stability a lot. Of course together with the new Matter version 3.3.1. I even re-enabled my OTBR and until now the new found Thread stability persists :) Even with devices that are also "shared" with HomeKit.

[cburghardt]

I am not using the WLAN from Fritz but this is for sure helpful for others. My WLAN is using TP-Link Omada stuff. I have already contacted them as I can't find any elements of IGMP Snooping or Multicast issues. The part that I don't understand: mDNS definitely works as it is used for many other things (also for HA) and I can find the border routers via the meshcop part for example. So how can I find out what happens several times per day so that everything breaks for 10 mins? Could IPV6 addresses be the issue, maybe they change?

[marcelveldt]

On Unifi there is a feature called "multicast enhancements" - as soon as I enable that my Thread devices go haywire. So I assume that TP Link has a similar feature ? Check for anything that is supposedly for optimizing multicast traffic, and try to disable (or enable) it.

[cburghardt]

Omada has something similar to convert multicast to unicast. I have disabled it but the problem persists. Is there any chance to trace what happens when the device becomes unresponsive? Like discovery fails or routes don't work or something.

[marcelveldt]

Yes, you can try to ping(6) it from the HA console.

[cburghardt]

OK, so final stupid question to get started - how do I find the IP address? I tried a service browser and the _matter._tcp. lists 3 entries. When I get something like this I can't ping it from the HA console (SSH addon): fd3d:4fc7:5d9e:1:30b1:f9ef:4b15:4de9:5540

If the last portion is the port, then at least this should work but I can't ping it (although the device is available currently): fd3d:4fc7:5d9e:1:30b1:f9ef:4b15:4de9

I looked at the diagnostics information from the device but at least could not immediately spot it. Sorry for all those questions, maybe it could be helpful for others as well.

Matter is HECK complicated ...

[cayossarian]

This past week I took my 15 or so eve devices over to Matter. I was able to put each Eve device into pairing mode (for multi-fabric) and assign them to HA Matter integration but then this configuration made every Eve device drop off line (on both the Apple side and HA side) nearly like clock work every 90 seconds or so and then come back after a couple seconds. This pattern is clearly visible in the history of each device. HomePods, AppleTV (2nd gen), and HA are all on the same VLAN and no snooping, etc. Once I removed HA Matter from the equation things went back to normal on the Apple HomeKit side with the devices staying on-line. My HA is running on a supervised Debian Linux setup and I don't have a SkyConnect (yet). The thread integration on HA says "no preferred network" but shows the HomePods.

As a second test I took an Eve Matter sensor and instead of putting it in Apple Home first I paired it directly on HA with the Companion App and exposed the sensor to Apple Home via HomeKit Controller. So far that sensor looks stable. Could just be the multi-fabric is problematic in my setup. Same intermittent behavior.

[cburghardt]

Just out of curiosity, is this somehow related: Default: mDNSPlatformSendUDP got error 99 (Cannot assign requested address) sending packet to ff02::fb on interface fe80::e4e8:9cff:feed:3f1a/veth94f46bd/21 I get this in the logs of the Skyconnect.

[djandrew2005]

2 tips that helped me resolve the issue.

1. Disable any DNS especially on ipv6 DHCP
2. Select "Allow All" on Promiscuous Mode (VM Settings -> Network -> Advanced)

[marcelveldt]

Please let uw know if this issue can be closed with the new Matter add-on 4.4.x

[cayossarian]

I posted earlier that my Eve devices were becoming unavailable intermittently in a Debian Supervised install but I was not the OP for this issue. For me the 4.4.0 and core 2023.5.3 updates resolved my issues as HA Matter was upgraded to 1.1 for sleepy devices like Eve door/window and regular Eve Energy (Matter). I have HomePods as Border routers but the devices are steady now even with multi-fabric (via both HomeKit and HA combined pairing).

Very much appreciated @marcelveldt !!!

As an aside for others that may experience problems in the supervised install it's useful to watch the firewall logs for mDNS and broadcast blocking around port 5353, 5540, 5580 and broadcast to 224.0.0.251. When I first put the Matter bits to test I had to make some changes there to even get the bits to connect.

[cburghardt]

For me the Eve plug still becomes unresponsive every 6 hours or so since the update to 4.4.0

[mbhforum]

I cannot even pair my Eve Motion sensor devices. I do have my HA server running on a different VLAN than my thread border routers. I tried disabling mDNS in my UniFi setup but didn’t help. Not sure what else I can try.

[cayossarian]

I cannot even pair my Eve Motion sensor devices. I do have my HA server running on a different VLAN than my thread border routers. I tried disabling mDNS in my UniFi setup but didn’t help. Not sure what else I can try.

I have Eve energy, motion, and door/window running reliably with Apple HomePod border routers. When HA boots it take up to a minute or so for sleepy Eve Door/Window devices to respond but after that they are immediate. I paired first with HomeKit and then use the accessory code provided by the paired accessory within HomeKit to pair to HA companion app. If you don't use HomeKit, you can just pair with the HA companion app.

The HA documentation says the HA and border routers need to be on the same VLAN (note use of link-local communication requirements).
mDNS is required for Thread on the WiFi/Ethernet link, e.g., to the border router(s).

Normally mDNS won't cross a VLAN boundary because it's link-local traffic although there are solutions like Avahi that bridge mDNS across subnets/VLANS but there are some who say that extending mDNS across subnets disrupts Thread. Jury is out but anyway...

IPv6 must be allowed to pass on subnets even if you don't assign IPv6 addresses in your network.

To get my install to a working/stable condition I surgically allowed traffic in the router and host firewalls as such (all UDP unless noted):

Between (from/to) IPv6 link-local fe80::/10 addresses, source port any, destination port 5353 (mDNS) and port 5540 (Matter Operational Discovery and Communication) From IPv6 fe80::/10 (link-local) to ff02::fb (multicast mDNS scope), source port any, destination port 5353 From IPv6 fd00::/8 (ULA) to ff02::fb (multicast mDNS scope), destination port 5353

IPv4 subnet addresses, src port any, destination port 5353 IPv4 subnet addresses to 224.0.0.251 (multicast mDNS scope) destination port 5353

For the border routers, at least Apple HomePods, TCP traffic must pass from/to their unprivileged ports and between devices (link-local and subnet), and HA on unprivileged ports (1024:65535)

In a supervised install the host firewall must be off or above traffic allowed to pass in the firewall rules. I used the HA supervised host firewall and monitored the logs, opening ports as necessary (all the above). If you use a firewall on theHA host the hassio network interface must also allow docker containers to talk to each other, e.g., 172.30.0.0/16 and any other add-on requirements.

[mbhforum]

Thanks for this feedback. One thing I also failed to mention is I disabled the firewall rule that was preventing traffic into my Core network (my ATV's are on this subnet) to eliminate any possible firewall/acl issues and it did not allow me to pair my Eve motion sensors to HA (they are paired fine to Apple Homekit just like yours as the primary).

I also made sure mDNS is working through my IoT (where HA is) and my Core subnets. I also realized that I never blocked off any ipV6 traffic, so that was allowed between IoT and Core anyway.

Putting my HA server on the same subnet as my ATV's isn't really desirable for me from a security perspective.

Any other ideas?

[mbhforum]

Shortly after typing this, I decided to put my HA server on my Core network where Thread Border Routers are (ATV, HPM) and it now works! I guess we can treat Thread like we treat Sonos = Not easy to segregate.

[Hupro]

I think I maybe having this problem with Matter-over-Thread devices becoming unresponsive but I'm not sure if it's the same thing or not because my problem is slightly different. I have 3 Nanoleaf Essential Matter bulbs (on latest firmware) and I paired them with Matter first with Google Home and then shared it to Home Assistant since we can't do the other way yet. Randomly throughout the day a random one of the bulbs will become unresponsive in Home Assistant, but it continues to work fine in Google Home without any problems. From reading through this issue if my problem was with mdns then the bulb should have also become unresponsive in Google Home but that is not the case. Any ideas of what may be causing that? I understand thread and matter are still very new so I also just wanted to see if this was a problem anyone else had too.

[cayossarian]

Seems that one sided good behavior was the same in HomeKit vs Google); Matter/Thread was rock solid in HomeKit, it was only HA that was having the random unavailability. Then I fixed the mdns/link-local routes/rules.

[djandrew2005]

I think I maybe having this problem with Matter-over-Thread devices becoming unresponsive but I'm not sure if it's the same thing or not because my problem is slightly different. I have 3 Nanoleaf Essential Matter bulbs (on latest firmware) and I paired them with Matter first with Google Home and then shared it to Home Assistant since we can't do the other way yet. Randomly throughout the day a random one of the bulbs will become unresponsive in Home Assistant, but it continues to work fine in Google Home without any problems. From reading through this issue if my problem was with mdns then the bulb should have also become unresponsive in Google Home but that is not the case. Any ideas of what may be causing that? I understand thread and matter are still very new so I also just wanted to see if this was a problem anyone else had too.

That’s exactly my problem too.

[djandrew2005]

@cayossarian How did you fix them?

[cayossarian]

I left a post earlier in this thread that explains the subnet, mDNS queries, multicas mDNS, and link-local addresses I ensured passed over the HA Ethernet and docker network interfaces to and from Devices and the border routers. For me this meant 1) ensuring there were rules in the router that allowed the described addresses to talk to and from HA, 2) the firewall on the HA host was allowing these addresses/ports. I used a firewall log to detect the traffic. Maybe your HA host doesn’t have a firewall. Finally, 3) ensure HA is in the same subnet/VLAN as the Thread border Router (Apple in my case).

[Edit] It’s also worth noting that when HA starts some good portion of the Eve Thread Matter devices take about 2-3 minutes to become available but once they do they are stable and responsive. Startup is a minor nit.

[cburghardt]

I can declare success since I moved all Matter devices to the Homepod, shared it with HA and then also disabled the pi hole.

[Hupro]

Hmm interesting, unfortunately in my case all my matter devices are on the same network with no separation or anything like that. I will have to try to look into what @cayossarian mentioned but I don't have a lot of knowledge about how to look into those things so will have to figure it out some other time. For now I'll just continue using Google Home to control them since it doesn't have any problems there

[marcelveldt]

@Hupro do you have the ability to run Home Assistant OS (maybe on a VM or something) ? Would be good to know if that does not has the issue.

[Hupro]

Ah sorry I forgot to mention how I have my stuff setup. I am currently running Home Assistant OS in a VM so the Matter add-on was installed automatically from the official add-on store. My current version numbers are Home Assistant 2023.6.1, Supervisor 2023.06.1, Operating System 10.2, Frontend 20230608.0 - latest. For my Thread routers I have two Nest Hubs, one in the room with the lights and one farther away (hoping to get a skyconnect soon not sure if that would change anything). I just found the logs tab for the Matter Server add-on so I will try to see what shows up in there the next time the problem happens.

[Hupro]

So I was able to look at the log entry when it became unavailable. Not sure if that provides any useful information or not but if there is something else I should look at when it becomes unavailable please let me know.

2023-06-10 21:08:42 core-matter-server chip.EM[125] INFO <<< [E:7514i S:8601 M:78286786] (S) Msg TX to 1:0000000000000002 [E710] --- Type 0001:08 (IM:InvokeCommandRequest)
2023-06-10 21:08:42 core-matter-server chip.IN[125] INFO (S) Sending msg 78286786 on secure session with LSID: 8601
2023-06-10 21:08:42 core-matter-server chip.DMG[125] ERROR Subscription Liveness timeout with SubscriptionID = 0xc75175fe, Peer = 01:0000000000000002
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO SecureSession[0x7f7a3c2de8c0]: Moving from state 'kActive' --> 'kDefunct'
2023-06-10 21:08:42 core-matter-server chip.DMG[125] INFO Will try to resubscribe to 01:0000000000000002 at retry index 0 after 0ms due to error src/app/ReadClient.cpp:894: CHIP Error 0x00000032: Timeout
2023-06-10 21:08:42 core-matter-server chip.DMG[125] INFO OnResubscribeTimerCallback: ForceCASE = 1
2023-06-10 21:08:42 core-matter-server chip.DMG[125] INFO Trying to establish a CASE session
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO UDP:[fdc1:1a9d:bae8:1:a83:b246:7d18:ffc6%enp0s3]:5540: new best score: 4
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Checking node lookup status after 102 ms
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Keeping DNSSD lookup active
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO UDP:[fdc1:1a9d:bae8:1:a83:b246:7d18:ffc6%enp0s3]:5540: score has not improved: 4
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Checking node lookup status after 104 ms
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Keeping DNSSD lookup active
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO UDP:[fdc1:1a9d:bae8:1:a83:b246:7d18:ffc6%enp0s3]:5540: score has not improved: 4
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Checking node lookup status after 105 ms
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Keeping DNSSD lookup active
2023-06-10 21:08:42 core-matter-server chip.DIS[125] INFO Checking node lookup status after 200 ms
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO Evicting a slot for session with LSID: 8602, type: 2
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO Sorted sessions for eviction...
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO Candidate Session[0x7f7a3c207240] - Attempting to evict...
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO SecureSession[0x7f7a3c207240]: Moving from state 'kDefunct' --> 'kPendingEviction'
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO Successfully evicted a session!
2023-06-10 21:08:42 core-matter-server chip.SC[125] INFO Initiating session on local FabricIndex 1 from 0x000000000001B669 -> 0x0000000000000002
2023-06-10 21:08:42 core-matter-server chip.EM[125] INFO <<< [E:7515i S:0 M:118257162] (U) Msg TX to 0:0000000000000000 [0000] --- Type 0000:30 (SecureChannel:CASE_Sigma1)

[marcelveldt]

Nanoleaf devices are reported unstable by multiple users and apparently this is a Nanoleaf issue which should be fixed in a future firmware update. For example Eve devices stay available on the same thread network. As you can see very clearly in the log, the subscription to the device fails (it basically is no longer responsive to the matter server).

Let's wait a bit for further updates. We saw the issue in early Eve firmware, and now with the latest (3.2?) firmware fixed.

[hoppel118]

Hi @cayossarian ,

thanks for documenting your findings. I don't have Matter (over Thread) devices in a different VLAN at the moment. But I want to be safe for future use.

I have a main vlan, where all my Apple devices and my HAOS (vm) reside and I also have an IoT vlan, where devices reside that aren't Matter capable (untrustable cloud native devices).

I am unsure, if I understand all your information right. But let me describe what I have now:

I have Eve energy, motion, and door/window running reliably with Apple HomePod border routers. When HA boots it take up to a minute or so for sleepy Eve Door/Window devices to respond but after that they are immediate. I paired first with HomeKit and then use the accessory code provided by the paired accessory within HomeKit to pair to HA companion app. If you don't use HomeKit, you can just pair with the HA companion app.

This sounds fantastic. When I reboot my HA it takes up to 15 or even 30 minutes for all devices to be recognized by HAOS and to respond. It also takes that long when I reboot my AppleTV 4k, the only Thread Border Router in my network.

The HA documentation says the HA and border routers need to be on the same VLAN (note use of link-local communication requirements). mDNS is required for Thread on the WiFi/Ethernet link, e.g., to the border router(s).

I found that information and was already in contact with marcelveldt who told me the same. So at the moment all relevant devices are in my main vlan.

Normally mDNS won't cross a VLAN boundary because it's link-local traffic although there are solutions like Avahi that bridge mDNS across subnets/VLANS but there are some who say that extending mDNS across subnets disrupts Thread. Jury is out but anyway...

OK, at the moment I use an mDNS service in my Unifi network to make my IoT devices (without Matter/HomeKit over Thread support) in the IoT vlan discoverable in the main vlan.

IPv6 must be allowed to pass on subnets even if you don't assign IPv6 addresses in your network.

its allowed.

To get my install to a working/stable condition I surgically allowed traffic in the router and host firewalls as such (all UDP unless noted):

Between (from/to) IPv6 link-local fe80::/10 addresses, source port any, destination port 5353 (mDNS) and port 5540 (Matter Operational Discovery and Communication) From IPv6 fe80::/10 (link-local) to ff02::fb (multicast mDNS scope), source port any, destination port 5353 From IPv6 fd00::/8 (ULA) to ff02::fb (multicast mDNS scope), destination port 5353

If I understand you correctly, I need the following rules for the destination port 5353 (mDNS) and port 5540 (Matter Operational Discovery and Communication).

So, I configured one port group to accomplish this. It looks as follows:

---

My ipv6 rules look as follows:

---

Do these rules need to be "LAN v6 In" or "LAN v6 Local" or do I need rules for both types?

IPv4 subnet addresses, src port any, destination port 5353 IPv4 subnet addresses to 224.0.0.251 (multicast mDNS scope) destination port 5353

My ipv4 rules look as follows:

---

My IPv4 Adress Group is configured as follows:

---

For the border routers, at least Apple HomePods, TCP traffic must pass from/to their unprivileged ports and between devices (link-local and subnet), and HA on unprivileged ports (1024:65535)

This is something I still have to do.

In a supervised install the host firewall must be off or above traffic allowed to pass in the firewall rules. I used the HA supervised host firewall and monitored the logs, opening ports as necessary (all the above). If you use a firewall on theHA host the hassio network interface must also allow docker containers to talk to each other, e.g., 172.30.0.0/16 and any other add-on requirements.

This is also something I still have to do/check. I am using Openmediavault 6 (Debian) as a KVM Hypervisor. My HAOS is installed as a KVM. I am unsure, if need to change something.

• the physical network interface of my Openmediavault machine is configured as a trunk port
• after that I configured logical network interfaces for every relevant vlan on that trunk port
• after that I configured network bridges for all logical vlan network interface

My HAOS vm gets an ipv4 and different ipv6 address from the network bridge device of my main vlan. This looks good to my eyes. and I can ping6 ipv6 destinations.

What do you think about the things I already have accomplished? Did I understand you correct or am I missing something?

Thanks and regards Hoppel

[cayossarian]

@hoppel118 If your devices are working I'd say you maybe you could even narrow your rules as they seem overly broad. I'll DM you so as not to pollute this thread related to a supposed defect. [Edit: I can't figure out how to DM you on GitHub but I've posted an explanation on the HA community forum at this thread and if you need further comment DM me there.

Cheers

[marcelveldt]

Closing this as the original issue report has been reported solved.

For anyone else: see our updated readme in the matter server repo with tips and tricks and especially do not try to route the mdns/matter traffic over multiple vlans, you will be on your own getting support.