search

bwssytems / ha-bridge

Home automation bridge that emulates a Philips Hue light system and can control other systems such as a Vera, Harmony Hub, Nest, MiLight bulbs or any other system that has an http/https/tcp/udp interface. This is a compact impl to run on small format computers. This is impl started from this project https://github.com/armzilla/amazon-echo-ha-bridge.
Apache License 2.0
1.45k stars 198 forks source link

HA-Bridge stopped - Alexa still finds devices #1216

Open CPM-luvr opened 4 years ago

CPM-luvr commented 4 years ago

How is this happening? Maybe it is related to having duplicate devices in Alexa list.

ssd65 commented 4 years ago

I noticed this issue(feature?) of Alexa picking up stored(?) devices from other alexa/echo devices. So it doesn't matter if your bridges are off.

One workaround is to keep deleting "one-by-one" (remove-all doesn't seem to work) the devices. Then detect, clear again. It will clear the detected devices in the other alexa devices.

You can also experiment by shutting down the other alexa devices in your home and then re-detect.

AndreasGaus commented 4 years ago

Same issue here with HA-Bridge and > 10 Amazon Echo devices. A caching mechanism seems to store the devices e.g. on Echo with integrated Zigbee Hub or maybe on other Echo devices or in the Amazon cloud. I stopped HA-Bridge and the devices "Philips Electronics intelligentes Gerät" are found again and again.

AndreasGaus commented 4 years ago

One workaround is to keep deleting "one-by-one" (remove-all doesn't seem to work) the devices. Then detect, clear again. It will clear the detected devices in the other alexa devices.

My issue: cannot delete HA-Bridge hosted pseudo Hue devices (and the many double named entries), while the real Philips Hue (Hue-Bridge) hosted ones work as expected in general. "There are more than one devices having the same name" sentence is the most heard one in our house (and probably our neighbourhood).

For short: Deleting one-by-one is better, but no workaround for me. If I have to click more than 400 times to fulfil Amazon's own idea of how Web-APIs should work: OK, I would do so. But let's see what happened to me in a very brutal detail:

0) My configuration: > 10 Echo devices (1 having an integrated Zigbee Hub), 150 phsyical or logical Hue devices, Philips Hue Bridge, HA-Bridge 5.3.0, FHEM.

1) Disable the "Philips Hue" skill (which is sometimes in conflict with the built-in Zigbee skill of some Echo device integrated Zigbee hubs).

2) Power off Hue Bridge and HA-Bridge.

3) Delete in amazon.alexa.de all devices (which are found again and again) one by one by clicking the delete button for each device instead of the button "delete all" at the very end of the device list. I have more than 400 devices, so I decided to start alphabetically sorted with A, the B, then C... and the test these letters only. After that I used the "delete all" button and searched devices again:

4) Doubles of devices are partly not found again by using one-by-one deletion, but some A, B, C devices are displayed again!

5) Power down also all my (> 10) Echo devices (also Amazon Fire Tab). HA-Bridge related devices are not not found again. The list is now very small (< 20 devices from originally > 400) ;-) So the caching seems to be done in house (not in the cloud)!

Now is the time to power on step by step my devices and hubs. I decided this order:

6) Power on my Echo with included Zigbee Hub (which I am not using explicitly, because I bought an original Philips Hue Bridge instead): Magically some Zombie "Royal Philips Electronics" devices reappeared (having the Umlaut issue "Küche" also known widely). So the integrated Zigbee Hub is part of the issue.

7) I decided now to delete the currently < 50 devices list one-by-one and search again (having only the Echo with integrated Zigbee Hub enabled): now 200 devices, mainly HA-Bridge hosted devices are found via integrated Hub (also some double named devices): The caching seems to be alive (because HA-Bridge is powered down). The workaround "ony-by-one" deletion seems not to work for the integrated Hub's caching. Power-down the integrated Hub and after "delete all" no HA-Bridge related devices are found. Q.E.D.

8) I decided now to enable additionally the original Philips Hue bridge and reactivate Philips Hue Skill and leave the Echo device with integrated Zigbee hub power off as long as possible (BTW: there are some complaints 2020/07, that latest Philips Firmware updates causes many issues with double device names): it worked fine as expected! No doubles, but HA-Bridge hosted devices.

9) Now I powered-on all Echo devices except the device which has an included Zigbee hub: in contrast to my expectation all "Royal Philips Electronics" HA-Bridge hosted devices are found again (also some double named devices)! So also some other Echo devices seem to cache them because they now reappear step by step (HA-Bridge is not running yet, caching seems to be distributed over all Echo devices). > 400 device Zombies are now alive again, also previously deleted A, B, C devices deleted one-by-one earlier! I restarted 2) and all the on-by-one deleted devices reappeared!

10) Somewhat in panic, I tried now to delete only the double named devices. "Device successfully removed" as Toast info from Amazon ist very slow, so maybe let the web services some time to do its job! Then I searched again: 300 devices, OK. Some doubles seem to be removed. But after pressing F5, 350 devices and doubles are back again. Really painful.

11) Maybe I should delete one-by-one of all non-unique device names and not leave one, e.g. DeviceA, DeviceA, DeviceA -> delete all three names? I finally ended up by clicking 400 times delete and wait for the Toast "The device is successfully removed". But I left one device alive (my "red lamp" in the living room, which is hosted by HA-Bridge). HA-Bridge ist kept powered off! I checked also in the Android App, that only my "red lamp" is displayed and searched for new devices again: Red lamp only, but after a while, F5 shows again 300 lamps, and many double names: Multiple devices as "Royal Philips Electronics intelligentes Gerät" and skill related names.

12) I enabled now the HA-Bridge again zu check, if the connection to HA-Bridge will probably solve the issue to delete the devices hosted originally by HA-Bridge in a durable way. But I detected no behavior change.

Finally I ended up without full success, even with the workaround to delete single devices. But I reduced the devices from 400 to 200 :-) After the end of June 2020 updates from Philips and maybe also Amazon updates (HA-Bridge version ha-bridge-5.3.0) this issue is back again (a half year rule?).

=> Amazon local echo devices (regardless if a Zigbee hub is present) seem to cache parts of the smart home device list. The button "delete all" neither deletes the devices from the Amazon Cloud nor from the in-house caches permanently. => Double devices of real Philips Hue hosted devices may be removed best using the one-by-one deletion workaround. This seems to work in most cases, but not in all. => The HA-Bridge mapping seems to lack the unique identification scheme which Amazon uses (probably vendor dependent) and Philips uses (for Hue / Zigbee). => Only the HA-Bridge has additional issues with German Umlaut (this unresolved Unicode issue: . See also https://github.com/bwssytems/ha-bridge/issues/1151#issuecomment-562870788))

So a combination of weak Amazon user interface / device list behavior, Philips software updates (which are known to be not stable for years) and incomplete or wrong HA-Bridge behavior can drive you crazy. IMHO there are two options: a) delete, wait, plug and pray b) change to a directly supported FHEM Alexa mapper.

HA-Bridge helped me for years to make my home smart. Sorry, but I must say that this fine software probably will not or cannot fix these complex issues in the future in behalf of a weak software architecture of Amazon and Philips.

AndreasGaus commented 4 years ago

Because the caching issue seems to be related to the duplicate names, please find here this related story in detail: https://github.com/bwssytems/ha-bridge/issues/1180#issuecomment-640122295

AndreasGaus commented 4 years ago

The workaround described by Superczar https://github.com/bwssytems/ha-bridge/issues/1180 worked fine for me. The details I checked so far:

1) Power off all physical Amazon devices (also software devices like Amazon App on mobile phones / desktop PCs; only shut down devices, installation kept). 2) Delete all devices in the Web UI of https://alexa.amazon.de (for Germany) 3) DO NOT search for devices in the Web UI but enable exactly one Amazon Echo device (I took the oldest one I own). 4) Check in the Web UI: Wow! No doubles / duplicated device names found and the single Amazon Echo works fine (with some restrictions, see my comment at the end of this post). 5) Enable the other Echo devices step by step and check using F5 in the Web UI if the list changes. Check if speech recognition and on / off commands work.

Yeah!

For me no further devices doubles are found. However, I did not dare a device search again (currently I am tired of deleting devices again and again as you may unterstand).

Remember that my system works fine for half a year, when I firstly detected the caching issues after software updates. The current June 2020 update of Philips probably triggered the revival of the cached zombies.

Maybe the great HA-Bridge team may find a holistic solution for caching issues which also fixes other annoying issues by reviewing the ID handling? Numbering / ID handling of devices is IMHO not stable in HA-Bridge and may also cause the caching errors even if HA-Bridge is not running!

popy2k14 commented 4 years ago

Same issue here. FIRST @ devs: Thanks fur such a great piece of software! Sadly i have to say i think the issue is in habridge. Why i am saying this?

I have another third party hue bridge (deconz/phoscon) and from there i have never duplicate devices.

Hoping the devs find the culprit.

Keep up the good work thx

popy2k14 commented 4 years ago

I have a workaround/solution, at least for me on RPI 4B with Raspbian buster.

I have the following echo devices:

The trick is to identify the problematic devices which are caching/sending the duplicates. I have done this, this way:

Do this for every alexa:

So now you have a list with BAD and GOOD devices and all echos should have NO power! In my case they are the following:

!! DONT PLUG IN ALL ECHOS; AND DO NOT SEARCH FOR DEVICES NOW !!

Now comes the fun part with ufw to block the UPNP traffic of the BAD guys. I am using ufw because its easier than fiddling around with iptables. First install it:

sudo apt install ufw

Now allow all incomming traffic (default is deny!!!!) sudo nano /etc/default/ufw

Change the DEFAULT_INPUT_POLICY to ACCEPT: DEFAULT_INPUT_POLICY="ACCEPT"

Save the file with CTRL-X and y + Enter Reopen and ensure that ACCEPT was saved! Otherwise you could lock yourself out of the RPI (ssh)!

Enable the ufw: sudo ufw enable

Status should look like:

pi@rfhem-pi:/etc/ufw $ sudo ufw status verbose
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), disabled (routed)
New profiles: skip

Disable ufw for now: sudo ufw disable

edit the "/etc/ufw/before.rules" files with nano: sudo nano /etc/ufw/before.rules

Now comment these lines out:

# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
#-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT

This will prevent that every UPNP discovery request will be accepted.

Add the following block with the correct IP for every GOOD device to accept UPNP:

#Accept UPNP discovery from this devices!
-A ufw-before-input -s 192.168.0.230/32 -p tcp --dport 80 -j ACCEPT
-A ufw-before-input -p udp -s 192.168.0.230/32 -d 239.255.255.250 --dport 1900 -j ACCEPT

And for every BAD device to block UPNP (dont forget to correct the IP):

#Block UPNP Discovery from this devices (they are bad -> duplicate devices in HA Bridge)
-A ufw-before-input -s 192.168.0.240/32 -p tcp --dport 80 -j ACCEPT
-A ufw-before-input -p udp -s 192.168.0.240/32 -d 239.255.255.250 --dport 1900 -j DROP

And before the COMMIT line, add the following. Just to be sure we block UPNP for new devices/echos which will be online in the future on the same network:

#drop all other maybe new echos
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j DROP

Be aware that the COMMIT line has to be the last line. Otherwise your changes may not take effect.

Save the file with CTRL-X and y + Enter and restart ufw with: sudo ufw reload

NOW connect every echo back to power and do a device search. This solved the duplicate devices issue for me!!

Thanks a lot to all people from https://github.com/bwssytems/ha-bridge/issues/806, from there i had the idea.

Hope this is helping you and others to eliminate the issue.

AndreasGaus commented 4 years ago

Uff! @popy2k14 : good workaround, but too complicated to be really accepted by the folks that have the double name / umlaut issue. I really do not understand why Amazon's Alexa software does not enhance the list of devices in the Alexa portal to finally remove a device permanently (also in the cached lists). The issues with duplicated devices are also present in other environments without HA-Bridge, maybe having the same source of a weird design flaw.

popy2k14 commented 4 years ago

Yeah, for sure it takes long to find the problematic devices and its complicated. I am with you: amazon has to fix this, but when would this be...

I mean just an idea for an workaround: Could HA-Bridge (or others) not just compare the names (or ID of the hue device) and prevent the duplicates in such a way?

Sure i am not deep in the hue-api and dont know much about it. Just an idea from "outside" :-)

pOpY

AndreasGaus commented 4 years ago

You are right. I am really frustrated, because of the "duplicated device" and "umlaut" issues arising again and again for years. A complex infrastructure with several companies involved needs a basic and stable model (based on a simple list of devices identified using unique ID and name; grouped by rooms as an option). I cannot believe that this simple model around a complex technical communication network is in general the issue #1!

tbclark3 commented 3 years ago

In the Alexa app, Settings/Device Discovery, there is an option to "Show available devices on my wifi network...". Turning it off will cause Alexa to forget the "inappropriately remembered" devices. You can then turn it back on and start with a clean slate.

popy2k14 commented 3 years ago

Thx for the hint. I don't find the option in my German alexa app. Can you please exactly tell us where we find the option?

tbclark3 commented 3 years ago

Open Settings, then Device Discovery (in the Preferences subsection). It is the only option under Device Discovery. There is another setting that might have potential. Under Settings/Alex Privacy there is a menu item called "Manage Smart Home Devices History. Under it, there is a single option called "Deleve Smart Home Devices History. I haven't tried it.

Screenshot_20201124-125805 Screenshot_20201124-130344

popy2k14 commented 3 years ago

Sadly i dont find the setting you descibed here on my "german" alexa app on android 10 (Oneplus 6T).

When i go to "More" (Mehr) -> "Settings" (Einstellungen) -> there is no option named "Device discovery" (Geräte Suche)

My Alexa App Version are:

Which Versions do you have und Info in the App? From which country you are?

thx

tbclark3 commented 3 years ago

I am in the US--only a few miles from Amazon. Maybe I'm getting special treatment. :) Client version: 1.24.208006.0 App version: 2.2.373840.0

audiofreak9 commented 3 years ago

@popy2k14 Don't feel too left out, I do not see this option either, iOS: Client version: 1.24.208013.0 App version: 2.2.386342

popy2k14 commented 3 years ago

@tbclark3 thx for the info. Seems you have a newer App version than me here in Austria. Hope the update drops soon here.

Does it work for you so you don't get duplicates from 'Philips Royale...'?

@audiofreak9 that's good so I am not alone 🙂

tbclark3 commented 3 years ago

I can't say whether it would eliminate all duplicates from Philips Royale. I can, however, give you a summary of my experience. During my attempts to get Alexa to discover habridge devices, I resorted to renumbering all habridge devices, power cycling all of the Alexa devices, and trying a device discovery. I was surprised that Alexa discovered 17 of the approximately 50 devices in habridge, and even more surprised that she "discovered" the old, no longer active, device numbers. I could not figure out the reason for those particular 17 devices. Obviously Amazon is caching this information on their servers. I started looking for settings and discovered that the "Device Discovery" setting made Amazon forget the incorrect device numbers. Afterward, Alexa discovered 0 devices. One could debate whether that was an improvement. That is when I decided to route all of the devices through Smartthings, and that has worked flawlessly and without any duplicates showing up. If Alexa ever directly discovers the habridge devices, then all of them will be duplicated.

I could send you the apk from my phone, but I doubt if it would give you the device discovery option if it doesn't already exist on whatever European server you are using. If you want to try it, we will need to figure out a way of transferring it privately.

popy2k14 commented 3 years ago

Thanks for the info. I think it will not work to use your US apk in the EU. We will have to wait and see :-)

mol7os commented 3 years ago

Don´t forget the FireTV´s ... these devices and services are not made for sustainable usage ... Doublicated devices mess up my installation from time to time and me to the brink of madness.

steviehs commented 2 years ago

If I hadn't now standing around 6 of this s***boxes in my house, I would stop using them, but I really love using them for switching lights....

Now maybe it would helpful for some of you, I wrote a small piece of robotframework code which automagically does the "delete every device one by one" part. Slow but working. You can find it on my gitlab misc page: https://gitlab.com/steviehs/stevemisc

steviehs commented 2 years ago

@popy2k14 wow, I ignored your one hint to remove the device from each echo while connected... this one Echo 2nd gen. does not forget anything even after a factory reset. So, yes, the way to go is really to delete each device step by step with this echo set up....

popy2k14 commented 2 years ago

@steviehs It's bad that amazon just dont fix such major issues! Yes, it's time consuming but i hope you got your setup working?

steviehs commented 2 years ago

Haha, atm there are no double or old devices shown... still, I see error messages like 2021-11-29 10:49:22 ERROR (MainThread) [homeassistant.components.emulated_hue.hue_api] Unknown entity number: 105 not found in emulated_hue_ids.jsonin homeassistants log showing that some sh*tbox is asking for old entities... but may be it forgets, when it gets older :-) But I am really happy with my robot script for deleting automatically... had to use it already 4 times :-) (development time vs. manual work is now break even :-)

popy2k14 commented 2 years ago

That's good! Sorry, dont know much about homeassistant.

Thanks for your script, maybe someone else (or i, when i change my setup) can profit of it. Currently in my case i have'nt changed anything since my guide above, and i have no double devices. But also not added any new alexa to my setup.

jayjupdhig commented 2 years ago

Maybe related with that? now i have a smiliar situation as described here... https://github.com/bwssytems/ha-bridge/issues/1351

steviehs commented 2 years ago

@jayjupdhig I would recommend to try: delete every single device in Alexa Web manually, this is important! If you see, that you have to do this more than once, try the robot script above. Then factory reset every gen >=2 Echo in da house. After that, I started ha-bridge 2.4.1rc1 from scratch again with an empty db, setting everything up once again is not so complicated as you could copy the calls from the old db into the webui of ha bridge. Then a rescan should give you the devices also with a 3rd gen Echo. I personally went for the solution described by @popy2k14 above, bought a used Gen 1 and blocked all the others.

All this is still such a crap... but hey, I am so used to not use my light switches manually anymore...

jayjupdhig commented 2 years ago

No difference between delete all and delete once... i see device they don't exist on that HA-Bridge instance...

1.) I've done a HW reset (then it was an access point again)

2.) I also deleted the device cache / history (that's "global", for all device in the account) in Amazon Alexa Web GUI

3.) Re-Numbering the IDs does also not help so i think, beacuse also our smart home system (which includes HA-Bridge) is a completely new installation...

That [these facts] all make ABSOLUTELY NO SENSE - NO LOGIC BEHIND THAT S**T...

popy2k14 commented 2 years ago

Do exactly the steps above https://github.com/bwssytems/ha-bridge/issues/1216#issuecomment-663559163 including block the bad ones in firewall. That worked 100% for me. The cruicial part is to find the bad alexas and block them. Again, do it exactly like i was writing it down. Otherwise you can get duplicates again.

I have dropped HA bridge because of that issue and using fhem connector without it. SOrry, cant support anymore.

jayjupdhig commented 2 years ago

ok thx :-)

(the thing is that i only have 1 active device in my account, the other 3 ones are turned off, so offline...)

popy2k14 commented 2 years ago

oh, that's bad. Can you use replace the one with an 1st generation? The first generation one's hadn't the issue for me.

jayjupdhig commented 2 years ago

We don't have any Gen. 1 devices here - thank you anyway :-)

jayjupdhig commented 2 years ago

Nobody knows about that issue...?