Multiple Xiaomi Gateways via Integration page not working

YoinkZed commented 4 years ago

The problem

Recent update to 0.112 introduced an error when setting up multiple Xiaomi Aqara Gateways. This was doable through the configuration.yaml file in 0.111.4. The error comes when trying to add the second gateway using either (any, internal ip of Home Assistant, or the gateway ip):

... and then the error:

Multiple reboots on both Home Assistant and the Gateway doesn't solve the issue.

Environment

arch	x86_64
chassis	embedded
dev	false
docker	true
docker_version	19.03.8
hassio	true
host_os	HassOS 4.10
installation_type	Home Assistant OS
os_name	Linux
os_version	5.4.44
python_version	3.7.7
supervisor	227
timezone	Europe/Copenhagen
version	0.112.0
virtualenv	false

Home Assistant Core release with the issue: 0.112
Last working Home Assistant Core release (if known): 0.111.4
Operating environment (OS/Container/Supervised/Core): Home Assistant OS
Integration causing this issue: MI XIAOMI AQARA
Link to integration documentation on our website: https://www.home-assistant.io/integrations/xiaomi_aqara

Problem-relevant `configuration.yaml`

Latest working configuration.yaml code.

xiaomi_aqara:
  discovery_retry: 20
  gateways:
    - key: !secret xiaomi_key1
      host: 192.168.1.91
      mac: !secret xiaomi_mac1
    - key: !secret xiaomi_key2
      host: 192.168.1.92
      mac: !secret xiaomi_mac2

Traceback/Error logs

Additional information

fliphess commented 4 years ago

I'm not sure about this path: wouldn't it be better to just revert the change made back to what it was in 0.111.4 with the mindset of "Do not change when it's not broken"?

I prefer have the old yaml configuration which was always working perfectly for my aqara gateways and it worked for years and years never resulting in any problems with it.....

resoai commented 4 years ago

Why are you slow to fix the error? Return everything back and correct as much as you want. How many people still have to express their displeasure to you?

You should be ashamed of yourself for posting this. People are donating their own time to an open source project and they don't owe anything to you. Without this HomeAssistant wouldn't even exist.

dshokouhi commented 4 years ago

Guys you can rollback HA as well or be patient and wait for the issue to get fixed. You can see they are trying to get it fixed, don't forget people contribute to this project on their own free time so be respectful. True bugs may show up but your can rollback HA and choose not to upgrade until the issue is resolved. Not everyone has this issue either. The best way you can help is to test the changes proposed or wait for someone to test it. Keep in mind the changes require the knowledge of knowing where your system files are located.

@starkillerOG would you be able to create a custom component with the proposed library changes for more users to test?

fliphess commented 4 years ago

I fully agree about the rejection of rude responses and people that are being impatient, this is not the way we should work/come together in a software community like this one.

If you experience this issue, just rollback and wait until it's fixed if you don't want to participate or otherwise if you can find some spare time, offer your help if you want things to go faster and make things better.

One thing that itches me a bit though is that I really don't know why this huge change was done. I do not want to be rude, impatient or an ass about it or nag the people that are doing the much appreciated hard work of trying to make home assistant better, but just for my own information: What were you trying to improve on this component that has been working properly for the last years, that resulted in these issues?

Just to create some more understanding for those who are heavily relying on the xiaomi_aqara component and were not able to turn their lights on.

artbrayko commented 4 years ago

Вам должно быть стыдно за то, что опубликовали это. Люди жертвуют свое время проекту с открытым исходным кодом, и они ничего не должны вам. Без этого HomeAssistant даже не существовало бы.

Thank you for your comment. I forgot where I am. I'm sorry to offend anyone. I had a hard day

dshokouhi commented 4 years ago

@fliphess the integration was converted to a config flow which enables many many new features. Like adding devices to an area using device automations. Managing the devices from the UI. An easier way for new users that don't know yaml to setup the integration. All integrations are being converted to this new setup. Yaml was also deprecated for integration setup if you recall a few months ago (https://www.home-assistant.io/blog/2020/04/14/the-future-of-yaml/). There are just some lingering bugs that need to be fixed and tested.

ekt- commented 4 years ago

I really don't mean to be rude or anything and "be gentle with the coder, not with the code" always applies. but in the real world you don't leave a core feature broken for few releases. You roll back, fix it, and deploy again.

Worst problem of the world right now? absolutely not. but, strange as it may seem, some of use relies on HA to be functioning and not consider it a toy anymore.

Rolling back HA has its own issue. with 0.112 the db schema has changed. it means you need to delete your now migrated db and lose all the data collected so far. That's what I'll do anyway. Again, not to be an ass, just speaking frankly

YoinkZed commented 4 years ago

I really don't mean to be rude or anything and "be gentle with the coder, not with the code" always applies. but in the real world you don't leave a core feature broken for few releases. You roll back, fix it, and deploy again.

Worst problem of the world right now? absolutely not. but, strange as it may seem, some of use relies on HA to be functioning and not consider it a toy anymore.

Rolling back HA has its own issue. with 0.112 the db schema has changed. it means you need to delete your now migrated db and lose all the data collected so far. That's what I'll do anyway. Again, not to be an ass, just speaking frankly

Well I gotta tell you that this might happen in the perfect world, but unfortunately this is not something even the big companies are following.

I do also rely on Home Assistant and my better half also looks funny at me, when it is night and the lights does not turn on, but the only person I can blame is myself for evolving into a project where tons of people dedicate much of their sparetime, but with no payment or obligations whatsoever. But the fact so many people are a part of this, we also get way more and better integrations than any other product (in my opinion) is able to deliver. So I think we should learn to be more patience and perhaps a habit of using backup even greater might be a good solution, so a roll back wont be such a pain for people. Maybe several versions back like I do myself in case of something like this happens.

So Thumbs up for @starkillerOG for gathering as much information as possible, so we can continue to develop this!

starkillerOG commented 4 years ago

@artbrayko Good luck learning python, because I'm not helping you anymore.

For the rest of the people that can appreciate the time I am putting into resolving an issue that I do not have, can't reproduce and is probably caused by the gateway beeing on some seprate subnet, a VPN blocking communication or some other stupid network issue:

I will look into how the old code managed to setup the gateway withouth knowing all information gathered during discovery.

Am I correct that all of you specified both the IP-adress and MAC of the gateway in your config.yaml in the old integration?

Because withouth knowing both the MAC (sid) and IP I do not see a way you could setup the integration withouth the discovery step....

resoai commented 4 years ago

@starkillerOG yes, that is correct, I specified both IP and MAC, if I remember correctly, MAC was mandatory for 2+ gateways but I might be wrong.

starkillerOG commented 4 years ago

Alright, I made a bunch of changes fixing some additional bugs and also including a way to specify the IP and MAC manually so you can setup the gateway if SSDP fails and Auto-discovery also fails.

I am almost certain this will fix the issues mentioned here, have tested it myself extensively.

Now I am just wainting on @Danielhiversen or @syssi to merge https://github.com/Danielhiversen/PyXiaomiGateway/pull/172

resoai commented 4 years ago

@starkillerOG fantastic! Perhaps this could even make it to the hotfix release if we are all lucky.

meichthys commented 4 years ago

I also manually specified the up address and Mac of the gateway. My network is very locked down and multicast was having issues navigating the (probably overly complex) network. @starkillerOG thanks for your work on this!

YoinkZed commented 4 years ago

Am I correct that all of you specified both the IP-adress and MAC of the gateway in your config.yaml in the old integration?

In my case yes. I had both the IP, Mac and Network-Key of course :)

starkillerOG commented 4 years ago

The PR has merged an PyXiaomiGateway has released version 0.13.0 on Github. However PyPi is still showing 0.12.4 as the newest version so we can not yet install it using pip. I think @Danielhiversen still needs to upload the latest release to PyPi, but honestly I have no idea how releasing something on PyPi works....

Almost there!

resoai commented 4 years ago

Releasing updates on PyPi is easy and straightforward so hopefully it won't take long.

starkillerOG commented 4 years ago

Does anyone know why the travis build fails here: https://github.com/Danielhiversen/PyXiaomiGateway/pull/177?

meichthys commented 4 years ago

@starkillerOG This build is now passing thanks to @syssi and v0.13.2 of PyXiaomiGateway is now updated on pypi.

maury77 commented 4 years ago

I tried to install the component but I have the same problem:

status installation

python3.8 -m pip install PyXiaomiGateway --user homeassistant Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Requirement already satisfied: PyXiaomiGateway in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (0.13.2) Requirement already satisfied: homeassistant in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (0.113.1) Requirement already satisfied: cryptography>=2.1.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from PyXiaomiGateway) (2.9.2) Requirement already satisfied: ruamel.yaml==0.15.100 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (0.15.100) Requirement already satisfied: bcrypt==3.1.7 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (3.1.7) Requirement already satisfied: async-timeout==3.0.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (3.0.1) Requirement already satisfied: certifi>=2020.4.5.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (2020.4.5.2) Requirement already satisfied: ciso8601==2.1.3 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (2.1.3) Requirement already satisfied: voluptuous==0.11.7 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (0.11.7) Requirement already satisfied: requests==2.24.0 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (2.24.0) Requirement already satisfied: python-slugify==4.0.0 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (4.0.0) Requirement already satisfied: attrs==19.3.0 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (19.3.0) Requirement already satisfied: jinja2>=2.11.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (2.11.2) Requirement already satisfied: astral==1.10.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (1.10.1) Requirement already satisfied: PyJWT==1.7.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (1.7.1) Requirement already satisfied: pip>=8.0.3 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (20.1.1) Requirement already satisfied: voluptuous-serialize==2.4.0 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (2.4.0) Requirement already satisfied: aiohttp==3.6.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (3.6.1) Requirement already satisfied: pyyaml==5.3.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (5.3.1) Requirement already satisfied: pytz>=2020.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from homeassistant) (2020.1) Requirement already satisfied: six>=1.4.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from cryptography>=2.1.1->PyXiaomiGateway) (1.15.0) Requirement already satisfied: cffi!=1.11.3,>=1.8 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from cryptography>=2.1.1->PyXiaomiGateway) (1.14.0) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from requests==2.24.0->homeassistant) (1.25.9) Requirement already satisfied: chardet<4,>=3.0.2 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from requests==2.24.0->homeassistant) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from requests==2.24.0->homeassistant) (2.9) Requirement already satisfied: text-unidecode>=1.3 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from python-slugify==4.0.0->homeassistant) (1.3) Requirement already satisfied: MarkupSafe>=0.23 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from jinja2>=2.11.1->homeassistant) (1.1.1) Requirement already satisfied: multidict<5.0,>=4.5 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from aiohttp==3.6.1->homeassistant) (4.7.6) Requirement already satisfied: yarl<2.0,>=1.0 in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from aiohttp==3.6.1->homeassistant) (1.4.2) Requirement already satisfied: pycparser in /home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.1.1->PyXiaomiGateway) (2.20)

Error

tail -f /mnt/dietpi_userdata/homeassistant/home-assistant.log result = await self._async_handle_step(flow, cur_step["step_id"], user_input) File "/home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages/homeassistant/data_entry_flow.py", line 201, in _async_handle_step result: Dict = await getattr(flow, method)(user_input) File "/home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages/homeassistant/components/xiaomi_aqara/config_flow.py", line 61, in async_step_user await self.hass.async_add_executor_job(xiaomi.discover_gateways) File "/home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/homeassistant/.pyenv/versions/3.8.0/lib/python3.8/site-packages/xiaomi_gateway/init.py", line 76, in discover_gateways _socket.bind((self._interface, 0)) OSError: [Errno 99] Cannot assign requested address

thanks

meichthys commented 4 years ago

@maury77 You'll need to wait until Homeasistant version 0.113.2 is released. That version should have these fixes. I would recommend waiting for the fix, but If you want to use the fix sooner you'll need to download the fixed version of the component from github and add it to your custom_components directory (I also had to modify the manifest.json file to use the latest version of PyXiaomiGateway (v0.13.2).

dshokouhi commented 4 years ago

The PR was not originally tagged for the milestone of 0.113.2 I went ahead and tagged it but can't make promises it will be included. It also still needs to be merged so it can be included in any release.

meichthys commented 4 years ago

@maury77 FYI - the fix got bumped from the 0.113.2 milestone. It should be in an upcoming release though!

YoinkZed commented 4 years ago

So if we are lucky it will be included in the next release - fingers crossed and thumbs up for the good work!

Quick question, with the update, do we then just try adding the second, third and son on with the 'Network interface to use' as Any?

meichthys commented 4 years ago

After the update you'll be able to setup multiple gateways either using any as the interface, or if you have issues using any you can enter the ip and mac of the gateway to add. After adding one gateway just setup the integration again to add a second, third.. etc.

One thing to note here: When entering a mac address, you'll want to include the :'s (In the old integration - pre v0.112.0 - we needed to exclude the :'s in the yaml.)

maury77 commented 4 years ago

Does the release contain no update?

i tried but same error but i wouldn't want it wrong having installed manually

meichthys commented 4 years ago

@maury77 As mentioned above, the fix won't be in the current 0.113.2 release, it will be in a future release (Hopefully 0.113.3)

starkillerOG commented 4 years ago

One thing to note here: When entering a mac address, you'll want to include the :'s (In the old integration - pre v0.112.0 - we needed to exclude the :'s in the yaml.)

actually does not matter if you use : it gets automatically reformatted. You can use ABCDEFGHIJKL AB:CD:EF:GH:IJ:KL AB-CD-EF-GH-IJ-KL or ABCD.EFGH.IJKL

Edit: I checked and this was actually not the case, but just made an commit that makes sure you now can use any of the above formats.

meichthys commented 4 years ago

@starkillerOG Nice improvement 🚀

maury77 commented 4 years ago

is included in 0.113.3?

starkillerOG commented 4 years ago

is included in 0.113.3?

No, just check the PR, that first needs to be merged. It is ready, but one of the HomeAssistant maintainers needs to check it and merge it...

starkillerOG commented 4 years ago

See the status of https://github.com/home-assistant/core/pull/37469

yecine06 commented 4 years ago

I have tested the latest prerelease which include the PR but it still not working on my side.

I have added the mac adress and the ip of the gateway but i still have the error :

Failed to discover a Xiaomi Aqara Gateway, try using the IP of the device running HomeAssistant as interface

starkillerOG commented 4 years ago

@yecine06 so you do get the option to specify the MAC and IP? That is really strange

starkillerOG commented 4 years ago

@yecine06 just looked at the code, the error you are seeing is because you did not specify the MAC and IP during the config flow, or you are not on the 0.114.0b2 release.

Take a look at the code: https://github.com/home-assistant/core/blob/b626368b6a65ad9aab4dd4cc0abb554a7db3fd97/homeassistant/components/xiaomi_aqara/config_flow.py#L69-L115

You are getting the error of line 114, while if you specify both the MAC and IP (you need to fill in both!), at line 78 the host, mac and sid will be set such that at line 87 a direct connection to the gateway is made, then there are 3 possiblities, or you get a succesfull setup, or you get an "invalid_host" or "invalid_mac" error which are diffrent than the error you are having....

starkillerOG commented 4 years ago

@yecine06 did you initiate the config flow through manually hitting the + icon or did you use the zeroconf discovered instance?

starkillerOG commented 4 years ago

@yecine06 I just made some very small tweaks to the code: https://github.com/home-assistant/core/pull/38622 Could you test those and see if that works for you?

zacharynguyeenx commented 4 years ago

This is what I got after auto discovery and also after manually entering the IP and MAC address.

omerbitas commented 4 years ago

This is what I got after auto discovery and also after manually entering the IP and MAC address.

Same for me

YoinkZed commented 4 years ago

Was this addressed with the latest 114.0 release?

zacharynguyeenx commented 4 years ago

This is what I got after auto discovery and also after manually entering the IP and MAC address.

Turned out my problem had nothing to do with Home Assistant, the gateway was having its port 9898 closed thus the integration can't complete. Followed the last point under Troubleshooting of Xiaomi Gateway (Aqara) Integration page and was able to add the second gateway now.

The error message could be improved with more troubleshooting steps though. Also noted the cryptic [%key:common::config_flow::data::ip%] string is still there on 0.114.1, possibly localisation done wrong?

Zoritto commented 4 years ago

Leave all fields blank and just click SUBMIT! enjoy)

starkillerOG commented 4 years ago

@zacharynguyeenx thanks for the feedback, I added a link to that section in the error message in this PR: https://github.com/home-assistant/core/pull/39617 The wrong translation was already fixed by someone else (sorry for that, that was my fault)

home-assistant / core