Xiaomi aqara gateway unavailable after some time

[groggemans]

Home Assistant release with the issue:

0.71.0, 0.70.1

Last working Home Assistant release (if known): unknown

Operating environment (Hass.io/Docker/Windows/etc.):

Docker (homeassistant/home-assistant:0.71.0)

Component/platform:

https://www.home-assistant.io/components/xiaomi_aqara/

Description of problem: All sensor data is available when home-assistant starts up, but after some time all sensors connected to the xiaomi aqara gateway become unavailable. Restarting home-assistant fixes the issue

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

xiaomi_aqara:
  gateways:
    - key: *****************
      mac: ****************
      host: ****************

Traceback (if applicable):

Additional information: There are no aqara/gateway related error messages.

[luci-11]

I have more or less the same issue here. I've installed HA on Raspbian following this https://www.home-assistant.io/docs/installation/raspberry-pi/. In my case the gateway remains online while some sensors become unavailable.

Restarting HA --> I have all sensors back but I need to add that no heartbeat is received for the sensors that were previously unavailable (confirmed by wireshark), so after 150min they go back to unavailable again (and that is the expected behaviour as far as I've read here https://github.com/home-assistant/home-assistant/pull/11631 )

I would like to add that forcing poll with self._should_poll = True in components/binary_sensor/xiaomi_aqara.py seem to avoid the unavailable issue but (apart from gas sensor that doesn't have a battery) I don't think this should be the correct way, even because it generates a lot of traffic and I need to check as soon as I come back home that the sensor effectively changes its state when kept alive with polling.

Question: Is there any way to trigger a poll to a sensor from a script for example? I'd like to check whether, if polled when unavailable, the sensor goes back to available status.

Thanks in advance, If you need any tests just let me know.

[luci-11]

Update from my side: (Premice: After the following tests I've noticed that the sensors I was testing were offline in mi home too, so I got in that conclusions)

TEST 1: restarting HA, I have all sensors back to available but in reality they do not change status (tested with water leak sensor). I suppose this is because the initial list is created from gateway reply to list command and that the gateway returns the latest status stored before the sensor went offline.

TEST 2: polling the device keeps it available but the returned status is fake and not updated in real time. I suppose in this case too the gateway returns the latest status before the sensor went offline (tested with a door sensor)

So....Is there any way to know in advance from gateway which sensors are offline in mi home too, so that to know in advance which devices have an available status even if it is fake (because it is not current status but last saved status in gateway) ? I think this is another real issue.

[groggemans]

Last time I checked my sensors where still online in mi-home, but I'll try to confirm this.

@luci-11 which firmware/hardware version of the gateway are you using?

[PsycoS81]

hello to everyone, I have the same problem, all my 4 sensors (pir, temp, door and button) are no more available on ha (0.71). Currently I can not make a clean restart because the system stops at this point: 2018-06-14 18:15:51 INFO (Thread-13) [homeassistant.components.xiaomi_aqara] Shutting down Xiaomi Hub 2018-06-14 18:15:51 INFO (Thread-13) [xiaomi_gateway] Closing socket 2018-06-14 18:15:51 INFO (Thread-13) [xiaomi_gateway] Closing multisocket

I tried the following attempts without any good result: 1) downgrade up to 0.67 2) downgrade of pycryptodome to version 3.3.1 3) remove and reconfigure the gateway and sensors from the application mi-home

after all these attempts the situation is this, I can see the values ​​of the sensors after the restart but these are not updated anymore and after 150min they go back to unavailable again

@luci-11 have you solved in some way?

Thanks in advance

[arsaboo]

I have slightly different (and potentially related) issue. All my sensors appear offline on the Mi Home app, but they appear fine in HA.

[PsycoS81]

@groggemans i have xiomi gateway v3, firmware version 1.4.1_155.0143

[groggemans]

I'm also running 1.4.1_155.0143 with hw version 3.

@PsycoS81 does it stop while exiting? or while starting again?

[PsycoS81]

@groggemans while exiting

[captnbp]

Same issue for me : Home Assistant release with the issue: 0.71.0

Last working Home Assistant release (if known): unknown

Operating environment (Hass.io/Docker/Windows/etc.): Docker (homeassistant/home-assistant:0.71.0)

Component/platform: https://www.home-assistant.io/components/xiaomi_aqara/

Aqara version 1.4.1_155.0143 with hw version 3

It only gets the sensors state/values at HA start then nothing updates until all Aqara sensors become unavailable.

[captnbp]

I just installed https://github.com/monster1025/aqara-mqtt with the aqara-mqtt container bound to the host network interface and it works like a charm :-)

[syssi]

Can somebody provide debug logs? I will provide some instructions if needed.

[PsycoS81]

@syssi which log files do you need?

[groggemans]

I can confirm @captnbp's observation that the sensors are only updated once when HA starts.

Currently running in debug mode, hopefully this results in some usable logs.

[groggemans]

@syssi any (preferably not public) way to send you the log? (its 1800 lines, from the start of home assistant til just after sensors become unavailable)

[syssi]

Just drop me a mail: basti@linkt.de

[syssi]

@groggemans I've checked your logs. Your Home Assistant instance is unable to receive multicast traffic. Could you describe your network setup in detail? Do you use this setting:

If you are using Home Assistant in Docker, make sure to use --net=host.

[groggemans]

I'm running in docker with --net=host (using nomad container orchestrator/scheduler).

The server has a virtual interface on a vpn which is used for web interface access (subnet 128 and 132) and a physical interface connected in the network the gateway is in (subnet 0).

Aren't the chromecasts and yeeelights also using multicast traffic? These work just fine, so I didn't suspect this to be the problem.

[syssi]

The Yeelight communication is UDP and no multicast traffic. Trust me. The multicast messages of the xiaomi gateway doesn't arrive at the HA instance. Please install "ngrep" and try to dump the traffic:

$ ngrep port 9898 
interface: eth0 (192.168.132.0/255.255.255.0)
filter: (ip or ip6) and ( port 9898 )
#
U 192.168.132.3:4321 -> 224.0.0.50:9898
  {"cmd":"heartbeat","model":"gateway","sid":"34ce0088db0b","short_id":"0","token":"LSWByhjQ4jzYHu74","data":"{\"ip\":\"192.168.132.3\"}"}                                     
#
U 192.168.132.3:4321 -> 224.0.0.50:9898
  {"cmd":"heartbeat","model":"gateway","sid":"34ce0088db0b","short_id":"0","token":"tFmVBr9rLaOVr3Zq","data":"{\"ip\":\"192.168.132.3\"}"}                                     
#
U 192.168.132.3:4321 -> 224.0.0.50:9898
  {"cmd":"heartbeat","model":"gateway","sid":"34ce0088db0b","short_id":"0","token":"LQv4ZVWmLdgX12iW","data":"{\"ip\":\"192.168.132.3\"}"}                                     
#
U 192.168.132.3:4321 -> 224.0.0.50:9898
  {"cmd":"heartbeat","model":"gateway","sid":"34ce0088db0b","short_id":"0","token":"M3FGzutTA2SPZ0V6","data":"{\"ip\":\"192.168.132.3\"}"}                                     
#
U 192.168.132.3:4321 -> 224.0.0.50:9898
  {"cmd":"heartbeat","model":"gateway","sid":"34ce0088db0b","short_id":"0","token":"BE1yKIdhLMXdf4JL","data":"{\"ip\":\"192.168.132.3\"}"}

If you execute "ngrep" in your docker container there will be silence. Try other hosts on your network.

[groggemans]

Ok, I'll check so we are sure. Any idea what the problem could be? As multicast with --net=host should work.. ?

[groggemans]

@syssi I can see the multicast traffic inside the container, but it seems it takes the virtual vpn interface as the default interface. If I use ngrep on the correct physical interface traffic comes in as expected. Home assistant probably doesn't listen on all interfaces by default?

[syssi]

Per default "any" interface is used:

https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/xiaomi_aqara.py#L100

If you use the interface parameter a specific interface can be selected. You could try something like:

xiaomi_aqara:
  interface: '192.168.0.XXX'
  gateways:
    - mac: xxxxxxxxxxxx
      key: xxxxxxxxxxxxxxxx

[groggemans]

any should contain the physical interface shouldn't it? I've added the ip of the physical interface with this option, but it doesn't seem to make a difference.

Any other things I should try?

[syssi]

What's your host (operation) system? This is the code which doesn't work properly for your setup: https://github.com/Danielhiversen/PyXiaomiGateway/blob/master/xiaomi_gateway/__init__.py#L118-L137

[groggemans]

The host is running ubuntu 16.04, but this should not be important, as the code runs inside a docker container. (which is running debian, stretch I believe)

[groggemans]

I've used https://github.com/monster1025/aqara-mqtt in the past on this setup, and that worked as @captnbp reported. We might find the problem by comparing both implementations. Although I'm not really seeing any differences in the case any is used as the interface (and the code is not running on windows).

https://github.com/monster1025/aqara-mqtt/blob/master/src/xiaomihub.py#L179-L185

[groggemans]

@syssi Seems like my firewall was interfering. For me the problem can be considered fixed. (didn't think of this, as ngrep showed traffic without problems)

Still have some trouble with a few sensors, but I'll open a new issue if I can't get it fixed after some investigation/debugging.

Thanks for your help and time!

[groggemans]

@luci-11 @PsycoS81 @captnbp

You should all check if:

• multicast messages are receivable/received (ngrep port 9898)
• firewall allows traffic on port 9898
• docker uses --net=host

[luci-11]

in my case, I receive all multicasts except heartbeat of unavailable sensors. Now i've moved my gateway closer to gas and water detector and they are correctly online/working in case of alarm. Doing this now I have other sensors unreachables (offline in mihome) and that appear online in HA at restart. So in my case it's a zigbee range problem but the first state in HA is still fake until unavailable

[syssi]

Correct. A device is marked as unavailable if unseen for a couple of minutes. We could invert the behaviour: All sensors will be initialized as unavailable and gets available on the first update.

[luci-11]

Question thinking about inverted behaviour (i don't remember this thing): if a sensor changes state while unavailable in HA, you should receive a report in any case right? .. and is report message triggering status change back from unavailable to available? or just heartbeat?

[syssi]

A event (window open/close) is published as a report. On network level it's a UDP multicast message to 224.0.0.50:9898:

$ ngrep port 9898
#
U 192.168.130.29:4321 -> 224.0.0.50:9898
  {"cmd":"report","model":"magnet","sid":"158d000116dff3","short_id":35590,"data":"{\"status\":\"close\"}"}                                                                                                                                 
#
U 192.168.130.29:4321 -> 224.0.0.50:9898
  {"cmd":"report","model":"magnet","sid":"158d000116dff3","short_id":35590,"data":"{\"status\":\"open\"}"}                                                                                                                                  
#

$ tail -f home-assistant.log
2018-07-03 07:55:38 DEBUG (Thread-23) [xiaomi_gateway] MCAST (report) << {'cmd': 'report', 'model': 'magnet', 'sid': '158d000116dff3', 'data': '{"status":"close"}', 'short_id': 35590}
2018-07-03 07:55:38 DEBUG (MainThread) [homeassistant.components.xiaomi_aqara] PUSH >> <Entity Door Window Sensor_158d000116dff3: on>: {'status': 'close'}

2018-07-03 07:55:41 DEBUG (Thread-23) [xiaomi_gateway] MCAST (report) << {'cmd': 'report', 'model': 'magnet', 'sid': '158d000116dff3', 'data': '{"status":"open"}', 'short_id': 35590}
2018-07-03 07:55:41 DEBUG (MainThread) [homeassistant.components.xiaomi_aqara] PUSH >> <Entity Door Window Sensor_158d000116dff3: off>: {'status': 'open'}

[balloobbot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

[groggemans]

I'll close the issue as it seems to be fixed for most people here. Best to open a new ticket if you still have problems.