jcallaghan / home-assistant-config

My Home Assistant configuration & documentation.
https://www.jcallaghan.com/
MIT License
175 stars 8 forks source link

Move from deCONZ over to ZHA #167

Closed jcallaghan closed 4 years ago

jcallaghan commented 4 years ago

deCONZ has been absolutely rock solid for the last year but recently I've had two occasions where I've lost connectivity with my Zigbee network and one of these I had to rebuild the integration with Home Assistant (#109). While I appreciate it's reliability up until this point I'm not so forgiving over the two issues and plan to move to Zigbee Home Automation (ZHA) which leverages Zigpy.

I will need support for:

Questions:

References

Zigbee network

This is my Zigbee network minus some devices I have paired since the last issue. I have just over 150 Zigbee devices.

image

deCONZ

https://github.com/dresden-elektronik/deconz-rest-plugin/issues/2875 https://github.com/dresden-elektronik/deconz-rest-plugin/issues/2863

jcallaghan commented 4 years ago

Observations

🟢 I have no side-by-side comparison but ZHA seems way more responsive than I found deCONZ.

🟢 ~I miss the separation of devices that deCONZ has where it split lights, sensors, and switches into separate UI screens.~ The new ZHA devices page in 0.112 allows you to sort by device type and is in fairness much better than the device pages that deCONZ provided.

🟢 No more flashing all the lights when a sensor is paired. deCONZ always flashed every light three times when I joining a new sensor.

🟢 My Smartthings multi-sensors no longer report a tampering event every hour. @ebaauw via dresden-elektronik/deconz-rest-plugin#2498 worked through this but I never benefited from his contribution sadly. Now I can enable my door knock #159, you have mail #158 and some other tamper based automations once again. This was worth moving to #ZHA for entirely.

🔴 Previously I wasn't getting accurate results for some devices where their battery level always reported 100%. It will be interesting to see if this changes at all with ZHA?

🟢 Standardisation of entity names would be helpful. deCONZ among other integrations create battery level entities with a suffix like _battery_level whereas ZHA it is just _power. This means every device must be renamed to create consistency. Particularly with so many of us using automation and templates to manage device battery levels. This is is just my OCD and a standard I have. Maybe one day I will update my code and not worry about this.

🟢 Know your device codes. For example, an outdoor Hue motion sensor is SML002 and the indoor is SML001. Knowing these helps identify the device a little more easily. See the table below for more information.

🟢 ~I found the joining page became hard to use and identify when new sensors had joined or even if ZHA was still searching for new devices once 2-3 devices had joined. So after adding a few devices I went back to the main ZHA page and returned back to add more devices.~ The new ZHA joining page in 0.112 when renaming a device all the entities get renamed too.

🟢 The bonus was my Hue sensors (indoor and outdoor) plus all the four-button remotes all joined without me needing to go round and manually join them. Sadly the same can't be said for my Aqara or Smartthings senors.

🟢 When joining all my devices I found it much quicker to move around the house room-by-room with my laptop and join the devices in that room. It was much quicker to give each device a temporary device name in ZHA. This allowed me to differentiate new devices vs one's I knew about. Once completed I was able to sit down with a glass of wine and rename all of the entities.

🟢 Joining devices alongside using the events developer tool listening for zha_events was key to my success. * I was quickly able to add devices and then by causing the device to fire an event (such as opening a door or window) I was able to determine the address of the device. I could then use the address to identify the right device before naming it.

🟢 ~I have a very fixed naming standard that I follow. This involved me renaming every single entity (some sensors have 4+ entities). Renaming the device previously renamed each of the devices entities. Renaming must be achieved through a string match as it would not work on the out of the box names but after renaming each of the entities it worked. Right now it seems this is very much a manual task but it would be great to see the entities of a device rename when the host device changes.~ Now ZHA has moved over to an integration in 0.112 now when renaming a device all the entities get renamed too.

References

https://github.com/dresden-elektronik/deconz-rest-plugin/issues/1261

jcallaghan commented 4 years ago

Pairing tools of choice

[[picture]]

jcallaghan commented 4 years ago

Device codes

Device Code/Quirk Battery Join
Hue indoor motion sensor Philips SML001 SML001
Hue outdoor motion sensor Philips SML002 SML002
Hue 4 button remote Philips RWL021 zhaquirks.philips.rwl021.PhilipsRWL021
Hue Smart Button Philips ROM001 zhaquirks.philips.rom001.PhilipsROM001 CR2032 Press and hold the button for 3 seconds to activate the device
Huw outdoor light strip
Aqara Temperature Sensor LUMI lumi.weather zhaquirks.xiaomi.aqara.weather.Weather CR2032 Hold pairing button for 5 seconds
Aqara door LUMI lumi.sensor_magnet.aq2 zhaquirks.xiaomi.aqara.magnet_aq2.MagnetAQ2 CR1632 3 seconds press
Aqara button v1
Aqara button v2 LUMI lumi.remote.b1acn01 zhaquirks.xiaomi.aqara.sensor_switch_aq3.SwitchAQ3B CR2032 Hold for 3 seconds
Aqara water LUMI lumi.sensor_wleak.aq1 zhaquirks.xiaomi.aqara.wleak_aq1.LeakAQ1 CR2032
Aqara motion lumi.sensor_motion.aq2
Aqara cube #188 lumi.sensor_cube zhaquirks.xiaomi.aqara.cube.Cube CR2450 5 second press
Smartthings button Samjin Samjin button zhaquirks.samjin.button.SamjinButton 10 seconds
Smartthings door Samjin multi zhaquirks.samjin.multi2.SmartthingsMultiPurposeSensor2019 10 seconds
Ikea Wireless Dimmer #184 CR2032
jcallaghan commented 4 years ago

Groups

image

jcallaghan commented 4 years ago

🟢 Smartthings sensors unavailable

See solution below.

My Smartthings sensors have become unavailable after restarting Home Assistant. Aqara and Hue devices seem fine. It seems those with the unk_manufacturermanufacturer ID are those that become unavailable. After looking at the device information nothing was know about these devices.

image

I noticed when I tried pairing one again the device LED wasn't very bright. After changing out the battery the light was much brighter and it joined with the correct device information. I don't think this is related to the battery more that these devices have lost connection.

Before...

image

After...

image

Solution

I removed these devices from ZHA and paired them again. When they joined I checked the device was recognised correctly before continuing on to another device.

jcallaghan commented 4 years ago

🟢 No hold or double press events for Hue remotes

See solution below.

I've lost the ability to use hold and double press events. I have some dope automations that triggered when with a button long hold event as well as when there was a double press. Seems the Hue remote is limited in what button presses it reports on. This is key for me and even after all the hard work to move to ZHA is a deal-breaker so I'm keen to support and help to address this.

trigger:
  platform: event
  event_type: deconz_event
  event_data:
    id: bathroom_light_switch

Example

I don't mind having to use the device_ieee but it would be nice to use the device name and it would be perfect if command listed 1001 like these examples.

  - platform: event
    event_type: deconz_event
    event_data:
      id: bedroom_button_switch
      event: 1002

vs

  - platform: event
    event_type: zha_event
    event_data:
      device_ieee: "00:15:8d:00:02:90:4c:2d"
      command: "single"

https://github.com/zigpy/zha-device-handlers/issues/86#issuecomment-647680161 https://github.com/zigpy/zha-device-handlers/issues/212

Solution

@dmulcahey very kindly updated the created a PR that adds support for press, hold, short release and long release in https://github.com/zigpy/zha-device-handlers/pull/388. I have also changed my automations to use device triggers rather than events.

trigger:
  - platform: device
    domain: zha
    device_id: ec715141c2564ed6945e5f4157e830aa
    subtype: turn_on
    type: remote_button_short_press
dmulcahey commented 4 years ago

🚨 Smartthings sensors unavailable

My Smartthings sensors have become unavailable after restarting Home Assistant. Aqara and Hue devices seem fine. It seems those with the unk_manufacturermanufacturer ID are those that become unavailable. After looking at the device information nothing was know about these devices.

image

I noticed when I tried pairing one again the device LED wasn't very bright. After changing out the battery the light was much brighter and it joined with the correct device information. I don't think this is related to the battery more that these devices have lost connection.

Before...

image

After...

image

@Adminiuga this is something we should look in to. The "unknown" manufacturer / model stuff still seems to be an intermittent issue.

jcallaghan commented 4 years ago

🚨 Aqara Vibration Sensor

The angle, xyz post and strength attributes were previously available as attributes to the entity when using deCONZ. These don't appear to be available in ZHA which will render some automations which leverage these attributes broken such as #96.

dmulcahey commented 4 years ago

🚨 Aqara Vibration Sensor

The angle, xyz post and strength attributes were previously available as attributes to the entity when using deCONZ. These don't appear to be available in ZHA which will render some automations which leverage these attributes broken such as #96.

We publish the tilt information as events. Can you explain what you mean a bit more by strength? We can explore supporting the same functionality. I have several of these so getting them working similarly shouldn’t be tough.

jcallaghan commented 4 years ago

Conbee II stick unavailable - timeline and notes

24/06/2020 12:30

  1. I lost power to the house and backup power which resulted in total power-out.
  2. Home Assistant reported issues accessing the Conbee II device and I had no control of my Zigbee devices.
  3. When my Synology NAS came back up it appears the deCONZ docker container started. I had only turned it off and not disabled it from auto-starting while I started to test out ZHA.
  4. The Conbee II device is visible on the NAS. It can be seen with lsusb and has the same device path.
  5. Turned off deCONZ and disabled auto-restart in Docker.
  6. Restarted Home Assistant Docker container several times and it still would not see the Conbee II device.
  7. Restarted the Docker service on my Synology NAS. Issue persisted.
  8. Turned off Home Assistant and ran the deCONZ container. The device was available and reported the firmware correctly. I reloaded the last known network config through the hidden network menu..
  9. Turned off deCONZ and brought Home Assistant back up. Issue persisted.
  10. Cleared the Home Assistant container and pulled down the image again. Conbee II device available again.
  11. Devices are available and show as such in Home Assistant but they cannot be controlled. Possible devices still reconnecting.

24/06/2020 16:00

  1. I went around the house and switched off all my powered lights for 30 seconds before turning them back on.
  2. Devices seem to be coming back online. @dmulcahey suggested to give it some time to allow things to come back online.
  3. I am now actively removing devices that were unavailable (those without a quirk if I am correct). Seems by removing them and pairing them again it seems to allow ZHA to discover them correctly.

24/06/2020 21:00

  1. After doing some family stuff for a few hours all my sensors that were unavailable are all back.
  2. Interestingly my lights seem more responsive. Something to keep monitoring but I've seen/had some bad behaviour where I would turn a light group off and only some of the lights would turn off.
  3. Seems removing devices with no quirk and pairing them again is doing the trick. I've almost got all my devices over to ZHA and working.
Adminiuga commented 4 years ago

hard to say without knowing the previous good network configuration and comparing the new one. I think the backup of the configuration is done by the phoscon, and depending on how long you've been running ZHA, the network config may have changed since the last backup.

  1. has zigpy_deconz started successfuly?
  2. Do you get attribute report from the devices?
  3. what firmware are you running on?
adonno commented 4 years ago

Oh man wat an adventure.

jcallaghan commented 4 years ago

Hanging ZHA add device page

Few times I have been on the add device page and performed a number of scans. This results in the log text area becoming quite heavy and after a period the page becomes slow and eventually, the browser (Edge) suggest to kill the tab.

Fixed by adding a few devices at a time. The new UI in 0.113 has also made the joining experience slicker and the log appears to be hidden or not available by default which also avoids the page becoming heavy.

Adminiuga commented 4 years ago

how many scans have you performed? Usually the intention was to join one or a couple of devices and then go back. But with 150 devices on the network, yeah, you are going to get a lot of noise from the other devices. Unless you need the realtime feedback when joining devices, you could also call zha.permit service and join devices, that would work in the background, without any realtime logging in the web ui

jcallaghan commented 4 years ago

how many scans have you performed? Usually the intention was to join one or a couple of devices and then go back. But with 150 devices on the network, yeah, you are going to get a lot of noise from the other devices. Unless you need the realtime feedback when joining devices, you could also call zha.permit service and join devices, that would work in the background, without any realtime logging in the web ui

Thanks for the pointers on the zha.permit service. You've described exactly what I've done where I join a few devices loop back and join some more. I've had a handful of devices that have refused to join which is where I've experienced this hanging issue. This is most likely from repeated join attempts while also receiving tons of noise.

jcallaghan commented 4 years ago

Two oddities

  1. One of my Smartthings Sensors has come through as "on/off" as opposed to "open/closed". Curious how the device class is determined and is this something I can fix or is it a device remove/join again?

Seems after rebooting Home Assistant/ZHA this sensor reported correctly. Thank you to @dmulcahey for helping me this one.

image

  1. One of my Aqara temperature sensors has come through with no battery level entity. Suspect this is due to the missing quirk.

This has been fixed with a new quirk. https://github.com/zigpy/zha-device-handlers/issues/385

image

Adminiuga commented 4 years ago

For aqara battery a quirk is required. Either it wasn't discovered completely or it is a different model and does not match the current signature. Post Zigbee information for it.

jcallaghan commented 4 years ago

@Adminiuga Aqara temperature sensor with missing battery information.

{
  "node_descriptor": "<Optional byte1=2 byte2=64 mac_capability_flags=128 manufacturer_code=4151 maximum_buffer_size=127 maximum_incoming_transfer_size=100 server_mask=0 maximum_outgoing_transfer_size=100 descriptor_capability_field=0>",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0302",
      "in_clusters": [
        "0x0000",
        "0x0003",
        "0x0402",
        "0x0403",
        "0x0405",
        "0xffff"
      ],
      "out_clusters": [
        "0x0000",
        "0x0004",
        "0xffff"
      ]
    }
  },
  "manufacturer": "LUMI",
  "model": "lumi.weather",
  "class": "zigpy.device.Device"
}
jcallaghan commented 4 years ago

Smartthings reporting normally

{
  "node_descriptor": "<Optional byte1=2 byte2=64 mac_capability_flags=128 manufacturer_code=4673 maximum_buffer_size=82 maximum_incoming_transfer_size=82 server_mask=11264 maximum_outgoing_transfer_size=82 descriptor_capability_field=0>",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0402",
      "in_clusters": [
        "0x0000",
        "0x0001",
        "0x0003",
        "0x0020",
        "0x0402",
        "0x0500",
        "0x0b05",
        "0xfc02"
      ],
      "out_clusters": [
        "0x0003",
        "0x0019"
      ]
    }
  },
  "manufacturer": "Samjin",
  "model": "multi",
  "class": "zhaquirks.samjin.multi2.SmartthingsMultiPurposeSensor2019"
}

and Smartthings reporting with the wrong device class.

{
  "node_descriptor": "<Optional byte1=2 byte2=64 mac_capability_flags=128 manufacturer_code=4673 maximum_buffer_size=82 maximum_incoming_transfer_size=82 server_mask=11264 maximum_outgoing_transfer_size=82 descriptor_capability_field=0>",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0402",
      "in_clusters": [
        "0x0000",
        "0x0001",
        "0x0003",
        "0x0020",
        "0x0402",
        "0x0500",
        "0x0b05",
        "0xfc02"
      ],
      "out_clusters": [
        "0x0003",
        "0x0019"
      ]
    }
  },
  "manufacturer": "Samjin",
  "model": "multi",
  "class": "zhaquirks.samjin.multi2.SmartthingsMultiPurposeSensor2019"
}

I also have this global customisation in play.

"binary_sensor.*_window*contact":
  icon: mdi:window-open-variant

image

dmulcahey commented 4 years ago

Hanging ZHA add device page

Few times I have been on the add device page and performed a number of scans. This results in the log text area becoming quite heavy and after a period the page becomes slow and eventually, the browser (Edge) suggest to kill the tab.

The new version of this page has the log hidden by default. I haven't peeked too much at what Bram did but I am sure it will be way more responsive. Definitely something we can look at in more detail to see what can be done to make this better.

Adminiuga commented 4 years ago

Your aqara has a different device type. Must be a different revision. Needs a new quirk to match signature.

For smartthings sensor, read the zone_type attribute of the IASZone cluster on both devices and compare the results. That value control the device class, but I don't remember if smartthings gives out the correct information or not. And since you are using templating, make sure your entity_id matches.

dmulcahey commented 4 years ago

Your aqara has a different device type. Must be a different revision. Needs a new quirk to match signature.

For smartthings sensor, read the zone_type attribute of the IASZone cluster on both devices and compare the results. That value control the device class, but I don't remember if smartthings gives out the correct information or not. And since you are using templating, make sure your entity_id matches.

I had him do that :) Great minds... lol Waiting for the results after a restart of HA.

jcallaghan commented 4 years ago

@dmulcahey ps the log is super useful still. AS busy as it is for me when pairing devices I was able to see device handles flashing through which was a good clue it had discovered the device I was pairing.

And as throughout this process @dmulcahey @Adminiuga I really appreciate your support. This has provided me with a ton of reassurance that this was the right thing to do. No regrets whatsoever. Thank you 🙏

jcallaghan commented 4 years ago

Further share on Twitter.

jcallaghan commented 4 years ago

Tim Messerschmidt @SeraAndroid

What's** the experience like in comparison to deconz? How long does it take for the network to be ready after a restart?

Having ZHA within Home Assistant is great...just the one place to manage things is always a benefit. And one less Docker image! I've noticed an improvement in the latency with light on/off events for sure.

I thought at first it would cause problems with the amount of time I bounce Home Assistant due to dev work but ZHA is up in no time at all and it has fixed a few devices I've had issues with. I'd say ZHA is up before Home Assistant has finished loading all the components.

I'm working with @dmulcahey to see if we can create a ZHA ready like an event but from our conversation, it is already up before Home Assistant is started. I'd love an event that I can use to report if there are issues.

jcallaghan commented 4 years ago

Martin Pugh @thepugmiester

I'm just on the verge of starting the ZigBee rollercoaster. I have a Conbee 2 and a stack of IKEA lights and remotes already taking to ha using the ones hub. I've just got deCONZ running but as I'm starting from scratch anyway, world you recommended going the LHA route?

Rollercoaster...Dreamliner! A great choice for your gateway and lights are key for a stronger mesh. I have 56 lights and when I moved this over from my Hue bridge my Zigbee game changes massively! Knowing what I know now, I wish I started off with ZHA 8 months ago. ⭐⭐⭐⭐⭐

There are still some kinks to work out but you only need to see the positive engagement I've already received to see ZHA is a serious contender.

adonno commented 4 years ago

I've Tried adding a Xiaomi Mi Open Close but I'm only getting image

dmulcahey commented 4 years ago

@adonno It didn’t pair correctly. Remove it via the UI. Press the button on the device while doing this. Wait a minute and rejoin it. Once it starts to pair you need to press the button on it every couple seconds until it’s paired.

adonno commented 4 years ago

@dmulcahey how can I remove it via the UI I have no button for this specific device on the ui

Adminiuga commented 4 years ago

@adonno which version? If on 0.111, do you have device in config panel -> zha -> device drop down? I assumed the screen shot was from there

adonno commented 4 years ago

image

Home Assistant 0.112.0b2 supervised on ubuntu

dmulcahey commented 4 years ago

@adonno you’re in the OOTB device screen... go to configuration-> Zigbee Home Automation

adonno commented 4 years ago

maybe I'm doing something wrong but ZHA has moved

zigbee

dmulcahey commented 4 years ago

@adonno whoops, missed that you were on the beta! In the beta it’s exactly where you are. I have no idea why the buttons aren’t there though. Try just resetting the device and re-pairing it again with the instructions from above without removing it.

adonno commented 4 years ago

Thanks it worked. I noticed that i wasn’t able to add devices via a „repeater“ There is a button „add via this device“ but it seemed impossible. Only way was to move in reach to the stick.

jcallaghan commented 4 years ago

Device codes

See my Wiki ZHA Devices page https://github.com/jcallaghan/home-assistant-config/wiki/ZHA-Devices

GitHub
jcallaghan/home-assistant-config
My Home Assistant configuration & documentation. Contribute to jcallaghan/home-assistant-config development by creating an account on GitHub.
jcallaghan commented 4 years ago

I think my move and journey to ZHA is firmly over. I have a handful of dev devices still in a box that I won't join now until I need them. Thanks again to @dmulcahey and @Adminiuga for their amazing support during this process. Your hard work and dedication to the project and the wider Home Assistant community are much appreciated, thank you!

image

basnijholt commented 3 years ago

Thanks a lot for documenting this publicly! It's quite helpful.

donnib commented 3 years ago

@jcallaghan what kind of routers do you use ? I have about 70 devices where most of them were GU10 Ikea Trådfri and had so many issues with them locking up and requiring power cycle so i have started switching to Hue GU10 and seems much more stable (have not had a Hue GU10 requiring power cycle yet and it's more than a week that i did the change). I use deConz and RaspBee but i would really like to move to ZHA to get rid of the deConz not becaus i don't like it but i have to maintain the rPI and i am afraid of SD card issues. At the moment i am testing with Sonoff Zigbee bridge and ZHA. You are using Conbee ?

Also do you have any IKEA battery trådfri products such as remotes ? Another issue i have seen with Xiaomi battery sensors (i can see you use Xiaomi) is that they drop out and needed repairing if their parent got offline e.g their router dissapeared because power cycle. have you seen issues like that ?

basnijholt commented 3 years ago

BTW, I moved all my 50+ devices over to ZHA from deCONZ two weeks ago and was really happy with the improved event handling in automations. Unfortunately, ZHA proved to be extremely unreliable for me (see https://github.com/zigpy/zigpy-deconz/issues/140) and therefore I moved everything back to deCONZ this weekend. Apparently, having only Hue lights that can act as routers isn't optimal for ZHA. There are many others that do not experience these problems, I just want to leave a warning somewhere ;)

donnib commented 3 years ago

@basnijholt thx, i currently have 17 GU10 Hue but i do have few other routers such as IKEA tradfri but i want to get rid of them.

MrMoosieMan commented 3 years ago

BTW, I moved all my 50+ devices over to ZHA from deCONZ two weeks ago and was really happy with the improved event handling in automations. Unfortunately, ZHA proved to be extremely unreliable for me (see zigpy/zigpy-deconz#140) and therefore I moved everything back to deCONZ this weekend. Apparently, having only Hue lights that can act as routers isn't optimal for ZHA. There are many others that do not experience these problems, I just want to leave a warning somewhere ;)

Well, darn. I'm in the middle of figuring out which technology to go with and your review is having me second guess my approach. I have about 50 Hue lights and was thinking I would use ZHA - maybe this is not a good idea. Thanks!

jcallaghan commented 3 years ago

BTW, I moved all my 50+ devices over to ZHA from deCONZ two weeks ago and was really happy with the improved event handling in automations. Unfortunately, ZHA proved to be extremely unreliable for me (see zigpy/zigpy-deconz#140) and therefore I moved everything back to deCONZ this weekend. Apparently, having only Hue lights that can act as routers isn't optimal for ZHA. There are many others that do not experience these problems, I just want to leave a warning somewhere ;)

Well, darn. I'm in the middle of figuring out which technology to go with and your review is having me second guess my approach. I have about 50 Hue lights and was thinking I would use ZHA - maybe this is not a good idea. Thanks!

It certainly is the right idea. I would not go back - both Home Assistant and ZHA have continued to grow and while I might have had some troubles migrating I've not looked back! ZHA is absolutely rock solid and hasn't given me any trouble in a long time. I now have circa 180 devices connected to ZHA - mostly Hue bulbs, motion sensors, and remotes along with Smartthings plugs and contact sensors and a ton of Aqara sensors (water, contact, temperature).