[REQUEST] Support for Silicon Labs EFR32 (Mighty Gecko family) based Zigbee radio adapters using newer EZSP serial protocol?

[Hedda]

Can the bellows library be made to support newer EZSP serial protocol such as EZSP v6, v7, and v8 when used with ZNet 6.x on Silicon Labs EFR32 (Mighty Gecko family) based Zigbee radio adapters?

See ex. #242 as that hardware is based on this same family and series of chips from Silicon Labs.

Zigbee EmberZNet version 6.7.0.0 has breaking change (with EZSP Protocol Version 8 and Secure EZSP Protocol Version 2) as both EZSP and Secure EZSP have adopted a new frame format with the following changes: (1) the fields of "Frame Control" and "Frame ID" are now two bytes; (2) no longer use "Legacy Frame ID"; (3) consume two bits of "Frame Control" to indicate the frame format version which is version 1 now.

• https://www.silabs.com/documents/public/release-notes/emberznet-release-notes-6.7.6.0.pdf

Perhaps newer modules will dethrone Nortek HUSBZB-1 for ZHA as the must-have Zigbee adapter?

[MattWestb]

@tube0013 Its also something strange with EZSP 6.6.4.0 that looks like yours phenomen. Was deleting 2 of 3 paired devices and was not enable pairing the a gen. Deleting the EZSP and the not deleted router was there then installing EZSP its was initiating the network (Was having pan and key) but not able paring the other 2 devices that i have deleted and reseted (have no pan and key). Repeating it 2 more time and all time the same. Then deleting the EZSP, stopping HA, deleting the zigbee.db. starting HA. Installing EZSP. And not possible paring any devices. Then deleting EZSP, stopping HA, Deleting zigbee.db. starting HA and installing TI-CC2531 as coordinator (That I was having before starting testing EZSP). Paring with all 3 devices working well and can controlling them. Then deleting the TI-CC2531 integration and restarting HA (must because of docker comports) and installing the EZSP and all devices its repairing !! But some cant being controlled. So deleting one Ikea bulb and was not possible paring it a gen. My feeling its that its being problems with the paring / rejoining in 6.6.4.0. But I have also doing successful binding with EZSP (with the "TI-CC2531" zigbee.db) and controlling one bulb with the Ikea On/Off dimmer switch. Very likely have silabs making changings in the protocol that its not catched up yet. I think its can being good for the developer to knowing so not getting more PITA then thinking its new things in V8. I have not trying with the dev branch then running my HA in docker. @Adminiuga If you need help with logging or more testing just say.

[tube0013]

I have another bridge (I bought 2 in case I bricked one or one came bricked), that I successfully flashed the 6.5.5 ota firmware this morning. I will test with both to see if I get issues. I did this multiple times - remove all devices one at a time, remove the zha integration, delete zigbee.db restart HA. Add ZHA - I could always pair a Tuya Temp/Humid sensor, but not Konke PIR, Aqara Lumination or Innr bulb. I tried repeating this and also using bellows command line with the leave command and aways the same. Also upgrading to new HA 0.113 with has the new bellows release in it.

[MattWestb]

First 4 god things !

1. You have verified that the 6..5.5 OTA its working that its great for the community (post it in Sonoff thread so other it's knowing it pleas) .
2. The other its that the problems its coming after 6.5.5 and before 6.6.4 so only one major release.
3. You can paring and repairing " normal" devices (Xiaomi its zigbee only to the name). What i have herig its normally the HUE bulbs the best routers and Ikea normally OK but not together with all end devices and Osram its one of the worse.
4. Great hope the HA 0.113 its released in docker soon.

Is it possible using bellows command line in docker ? I like trying changing PAN, KEY and CH and see how its working and don't getting all devices rejoining then then re installing the EZSP.

[tube0013]

here are a bunch of logs for v8 with 0.113.0 - I removed all four devices, then went through trying to re-pair 3 of them. The Konke PIR and Innr Bulb would not pair. the Tuya sensor does.

https://paste.ubuntu.com/p/9cYhvzt2yV/

[MattWestb]

Thanks for the Sonoff posting it was making some waves :-)))) I have HOMA LED driver that is Z30 and inside its one TI-CC2530 and its being very easy paring / rejoining (and also very bad radio). The Ikea its EZSP stack and Z30. Can it being that the Z30 security polices its blocking then it's different security settings ?
Only some brain storming and some PITA ;-)

[MattWestb]

I think You have named the problem @tube0013 !! In 6.6.4.0 release notes 2.1 Plugin Changes: its have being changes Trust Center Network Key Update Periodic. If setting the EZSP with old parameters that its out of range for the TC then all pairing and rejoining it's more or less broken. The rest looks like LL and other more cosmetics things.

[MattWestb]

HA 0.113.0 its in and running. With "CC2531 db" all 3 devices is coming back but not working. Deleting HOOMA and repairing it working OK. Deleting my Ikea On/Off Dimmer Switch and it joined the network but not passing the initializing .

Device 0x3eaf (14:b4:57:ff:fe:70:c4:bb) joined the network
Device 14:b4:57:ff:fe:70:c4:bb changed id (0x9fc9 => 0x3eaf)
[0x3eaf] Requesting 'Node Descriptor'
Tries remaining: 2
[0x3eaf] Extending timeout for 0xa7 request
Device 0x3eaf (14:b4:57:ff:fe:70:c4:bb) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:70:c4:bb
Device 0x3eaf (14:b4:57:ff:fe:70:c4:bb) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:70:c4:bb
[0x3eaf] Delivery error for seq # 0xa7, on endpoint id 0 cluster 0x0002: message send failure
Tries remaining: 1
[0x3eaf] Extending timeout for 0xa9 request
Device 0x12e1 (14:b4:57:ff:fe:70:c4:bb) joined the network
Device 14:b4:57:ff:fe:70:c4:bb changed id (0x3eaf => 0x12e1)
Canceling old initialize call
[0x12e1] Requesting 'Node Descriptor'
Tries remaining: 2
[0x12e1] Extending timeout for 0xab request
Device 0x12e1 (14:b4:57:ff:fe:70:c4:bb) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:70:c4:bb
Device 0x12e1 (14:b4:57:ff:fe:70:c4:bb) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:70:c4:bb
[0x12e1] Delivery error for seq # 0xab, on endpoint id 0 cluster 0x0002: message send failure
Tries remaining: 1
[0x12e1] Extending timeout for 0xad request
[0x12e1] Delivery error for seq # 0xad, on endpoint id 0 cluster 0x0002: message send failure
[0x12e1] Requesting Node Descriptor failed
[0x12e1] Discovering endpoints
Tries remaining: 3
[0x12e1] Extending timeout for 0xaf request

The paring LED its shining all the time until its timing out so the device its not sleeping.

One curious question @tube0013 have you also trying the 6.7.4.0 OTA firmware on the TSB ? Perhaps better waiting so you have the 6.5.5 and can do testing with it if not working flashing it.

[Adminiuga]

I was experiencing something very similar. I still trying to pin point the root cause, but ATM I want to split EZSP version handling into different submodules.

Can you try changing https://github.com/zigpy/bellows/blob/2af82b78044655453040c30e083efc63884de373/bellows/zigbee/application.py#L244 to be t.EzspDecisionId.DENY_APP_KEY_REQUESTS ?

restart EZSP and try again. If it doesn't help, try increasing key table size. in configuration.yaml add:

zha:
  zigpy_config:
    ezsp_config:
      CONFIG_KEY_TABLE_SIZE: 8

restart EZSP. Post results here

[MattWestb]

I trying then i have find the bellows files that HA have hidden very well :-(

Was finding something interesting than my HOMA its 100% paring and Ikeas 100% not pairing then having cleaned HA.

From Sonoff Zigbee bridge:

s-hadinger commented 25 days ago Finally!!! The Xiaomi and Blitzwolf are pairing correctly. Thanks to all.

Using a Zigbee sniffer, I could make sure to mimic what the CC2530 was doing. Lesson 1: the device refuses non-encrypted networks. Lesson 2: the device refuses network key sent in the clear. The devices expect to receive the network key encrypted with the well-known key.

The right setting is the following:

Link Key set to well-known key Disable sending network key in clear

[Adminiuga]

i think it is something about joining ZB3 devices (end devices) directly to coordinator. Currently it is failing on an empty network with only one another HA1.2 end device on the network. This is on EZSP 5.8.10 which is EZSP protocol version 4.

[Adminiuga]

In this particular case it is a ZB3 end-device, and I was able to join it successfully only after adding a transient link key ZigBeeAlliance09 for the IEEE of the joining device. 🤔

[MattWestb]

My HOMA LED driver its (on the paper) Z30 and router its paring OK (inside its TI-CC2530) My Ikea Bulb its also router and Z30 and not pairing. My Ikea dimmer switch its also Z30 and not pairing. So for My its not router / end device more different Zigbee stacks / versions. I think the Z30 its more restricted and not accepting to low security and the newer stacks its made more secure. I have 2 new Xiaomi window / door contacts and one Temp/hum/preas I can trying if they working

[MattWestb]

MCCGQ11LM = Win/door 0xf4d8: completed initialization

WSDCGQ11LM = Tem/hum/Prea Not parring

Tha last can have newer FW that its seeing the last months that is "more Z30"

[MattWestb]

The Temp/hum/prea was successful then paring thrus the HOMA !!! But not possible thru NCP !!

But the Ikea Of/Off switch not true HOMA. [0x8726] Requesting 'Node Descriptor' Tries remaining: 2 [0x8726] Extending timeout for 0x17 request

The Ikea Z30 its leaving the networks because of to high / low security in the NCP.

[MattWestb]

Then have pared the Ikea (Z30) devices is falling back to Z21 with the CC-2531 (Z21) the security its OK for Z21 - Z21. Then changing NCP to EZSP with z30 the Ikea (Z30) devices trying re joining but leaving the network. The devices have all networks information but the Z30 - Z30 its blocking the rejoining the network. And my HOMA its very likely falling back from Z30 to Z21 so no problem with Z21 - Z30.

[Adminiuga]

then changing NCP to EZSP with z30 the Ikea (Z30) devices trying joining but leaving the network

is this with EZSP?

[MattWestb]

Yes its with EZSP 6.6.4.0 = Ver 7 I think

[Adminiuga]

yeah, I kind of experiencing the same with Ver 4 on EmberZnet 5.8.10

[MattWestb]

Zigbee 3 security its blocking the zigbee devices (end/routers) rejoining the network and then its very understanding that the same devices not can paring the network because the Z30 security its blocking it.

[Adminiuga]

but it you are resetting the device, is it still considered rejoining? I can join fine, only if I add transientLinkKey either with the key from install code or ZigBeeAlliance09

[MattWestb]

I was not resetting the devices then changing NCP from CC2531 to EZSP. So all the devices have all network security saved in memory (PAN, Network key and so on). The HOMA its working after the switch of the NCP as it was a normal restart of HA = Z21(or bad Z30 security its OK Ikea Z30 - EZSP Z30 security then the Ikea devices leaving the network,

Then resetting the IKEA devices they trying pairing but failing like this:

0x8726] Requesting 'Node Descriptor' Tries remaining: 2 [0x8726] Extending timeout for 0x17 request

Time out = The device have trying joining the but leaving the network then the Z30 security its not OK

[MattWestb]

From the Tasmota Sonoff Zigbee bridge:

s-hadinger commented 25 days ago Finally!!! The Xiaomi and Blitzwolf are pairing correctly. Thanks to all.

Using a Zigbee sniffer, I could make sure to mimic what the CC2530 was doing. Lesson 1: the device refuses non-encrypted networks. Lesson 2: the device refuses network key sent in the clear. The devices expect to receive the network key encrypted with the well-known key.

The right setting is the following:

Link Key set to well-known key Disable sending network key in clear

[Adminiuga]

Link Key set to well-known key Disable sending network key in clear

But that's what bellows was doing for ages

[Adminiuga]

although Opple which is ZB3 joins just fine without any issues...

[Adminiuga]

and Konke also joins and rejoins just fine and stays connected. This is all on EZSP version 4

[MattWestb]

Then its something else that its doing the ghost things ;-( I can trying firmware downgrade one Ikea On/Off switch to LL2 (with deCONZ) tomorrow and testing if it can join.

Its one of the fue ZB certed devices from Xiaomi but its depends of the Zigbee stack security setting and I think Xiaomi its using the NXP stack.

[Adminiuga]

I think at least my experience may have to do something with the devices. Opple remote and Konke motion sensor are joining just fine and both are ZB3 (supposedly), but Aqara no-neutral wall switch (end device) joins and leaves.

[MattWestb]

Different implantation of Z30 and versions of the Z30 implantation (like EZSP ver 4 and 6) and TI and NXP versions and so on. Its very spooky mess what I can see.

Do You have HA ore LL ver 2 devices that you can testing ? Was some of this devises working before with belloes that not working now ?

[MattWestb]

Ikeas old motion sensor and RGBWW bulb its not updated to ZB3. The problem its we not knowing it the Z30 devices its falling back Z21 and can joining the network or if its real zigbee 3 in both ends. The strange thing was my Xiaomi weather that was not pairing with the EZSP but OK thru HOIMA.

[MattWestb]

I have making some more interesting testing. First 2 bugs that its not yours but in HA. 1 If deleting one device in deCONZ and restarting deCONZ and HA and then adding it in ZHA HA its saying its one "deCONZ" device(I think it's only cosmetic). The second its then ZHA have trouble then paring its close the paring window (60 sec), if clicking "try again" the old paring process its still running (60 sec its too short for paring then problem in the pairing process) in the log window and needs ca 30 sec more time for ending the process.

Now to device testing: "Procedure" Deleting ZHA in HA. Stopping HA. Deleting Zigbee.db and zigbee.storage. Starting HA. Adding ZHA with EZSP. That Is the cleanest i can do (what i knowing) without deleting all. Adding HOMA (ZB3 CC2530) its paring very fast without timeouts (ca 5 sec) Adding one Ikea motion sensor (old model ZB2 LL deleted in deCONZ) OK. (ca 10 seconds) Adding one Ikea motion sensor (new model ZB3 deleted in deCONZ) OK but taking long time because timeouts (around 30 sec). Adding Lumi.magnet.aq2 (Z?) paring OK but talking little longer time and with medium timeouts (more than 20 seconds). Adding limi.weather (Z?) paring OK but talking long time and with little timeouts (more than 30 seconds). Trying adding the other Ikea ZB3 devices resulting in device leaving and rejoining and the coordinator getting timeouts all the time and in the end not pared. Thats was the first round. The second round I was not pairing the HOMA = no router only end device. All its the same as the first round only that i was not able paring the new Ikea motion sensor agen (have trying over 10 time and all negative). I have redone the above scenario 6 times as in the first round and with the same results.

Its opens for one more scenario then i seeing more or less timeouts in the parring log and the HOMA has nearly no timeouts and Xiaomi and old Ikeas have moderate timeouts and Ikea ZB3 very / too much of it. Ikea devices (silabs stack) its known that getting lock in the firmware then sending multiple commands to then too fast (from deCONZ). That can being one thing way the leaving (rebooting ?) and rejoining all the time then trying to paring them. Is it possible for you putting some smale delay between commands so not "stalling" some bad acting devices ? Z2M its on the hunt for pairing issues and its trying putting delay in the TI firmware for the NCP then its working then the firmware have debug enabled and have problem then its disabled. The largest strange thing it's the new Ikea motion sensor (ZB3) that was only first time possible to paring. Can HA making some strange things here or is it zigpy / belloes process "isolated " from HA / ZHA ? Keep working all things its (nearly) possible to do :-)))

Update: Have trying CC2531 NCP and all 7 devices it's possible to paring but must holding them very near the NCP then doing it (very week radio) and the process its not so fast as with EZSP.

[s-hadinger]

@MattWestb I tried to pair an IKEA bulb to EZSP with Tasmota, it worked fine. However since it's actually a router, there are strange things happening.

When EZSP and IKEA bulb are paired:

• If I reboot the EZSP, the bulb becomes unreachable. The packets are correctly sent (checked with sniffer), but the bulb does not react.
• Then, if I power off/on the IKEA bulb, it sends a Device_announce and everything is working again.

Does anybody knows if I need to do something special on EZSP side when rebooting to have routers connected again?

[MattWestb]

I have new flashed Tasmota dev and EZSP 6.7.4.0 installed. I cant testing it then it was not possible getting my Ikea test bulb paired. I was doing the same test serie as the post above and the result was the same in Tasmota as in ZHA.

HOMA (Labeled ZB3 LED driver with one CC2530 module inside) paired OK and looks OK after restart (not trying sending commands). Old Ikea motion sensor (ZB2 LL) paired OK but not handled of tasmota. New Ikea motion sensor (ZB3) it's trying pairing but only first stage with: 14:33:49 MQT: tele/tasmota_B3C82A/RESULT = {"ZbState":{"Status":34,"IEEEAddr":"0x14B457FFFE70C4BB","ShortAddr":"0xFB6D","ParentNetwork":"0x0000","Status":1,"Decision":0}} n the web log (It's in the network but not configured). Ikea New bulb (With new FW ZB3) Same as New motion sensor. Ikea On/Off Dimmer Switch (With new FW ZB3) Same as New motion sensor. lumi.weather (Aqara ZB ??) Pairing OK and sending OK reports. lumi.sensor_magnet.aq2 (Aqara door / window sensor ZB ??) Pairing OK and sending OK reports.

The interesting is what firmware is in your Ikea bulb ZB2 LL or ZB3 then you can paring it and i can't with all Ikea ZB3 products i have trying (Only first time of 15 time with the new motion sensor was paring OK). I only owning one Ikea bulb with ZB2 LL firmware (The old RGBWW) but i cant removing it from my production system ;-(

[MattWestb]

CMD: ZbStatus2 its giving in MQTT:

{"ZbStatus2":[{"Device":"0xFB6D","IEEEAddr":"0x14B457FFFE70C4BB","Endpoints":[]},{"Device":"0xFEFB","IEEEAddr":"0x00124B001A346376","ModelId":"HOMA1008","Manufacturer":"ShenZhen_Homa","Endpoints":["0x0B","0x0D"]},{"Device":"0xA7B4","IEEEAddr":"0x90FD9FFFFE59BB73","ModelId":"TRADFRI motion sensor","Manufacturer":"IKEA of Sweden","Endpoints":["0x01"]},{"Device":"0x95F0","IEEEAddr":"0x14B457FFFED4D031","Endpoints":[]},{"Device":"0xA472","IEEEAddr":"0x14B457FFFE8094A0","Endpoints":[]},{"Device":"0x881B","IEEEAddr":"0x00158D00045CDCE2","ModelId":"lumi.sensor_magnet.aq2","Manufacturer":"LUMI","Endpoints":["0x01"]},{"Device":"0xAC89","IEEEAddr":"0x00158D00053FD050","ModelId":"lumi.weather","Manufacturer":"LUMI","Endpoints":["0x01"]}]}

MAC 0x14B457XXXX its (new) Ikea devices.

[MattWestb]

@s-hadinger I can confirming you problem after restart. Was sending toggle command to the HOMA and its was working OK. Repower tasmota and toggle coman not working (sending OK but the device was not executing it) Repower the HOMA and the command is working. So this thing it's probably not only Ikea.

[Adminiuga]

@NilsBohr do you have EZSP ver 8 firmwares for ELU013 and ELR023 ? Couldn't find one on https://elelabs.com/products/elelabs-usb-adapter.html

PS: Thanks for sending the RaspiShield 👍

[MattWestb]

@Adminiuga @s-hadinger little interesting reading and explaining of upgrade ZB3 from ZB HA https://github.com/Koenkk/zigbee2mqtt/issues/1820#issue-477188389. Also linked to TIs "What's New in Zigbee 3.0" https://www.ti.com/lit/an/swra615a/swra615a.pdf Explaining TC, Child Device Management, Child Device Management. Install code and so on. Tasmota was not running in ZB3 mode before (CC253X ZB1.2 firmware) so its little more to considering, and i don't how much of the ZB3 is implemented in belloes and if the firmware before was not ZB3 enabled it was no problems but then have ZB3 firmware in the coordinator the ZB3 devices trying using true ZB3 connection and not falling back to ZB 1/2 from both sides (NCP and devices). In my test set up its looks like all the TB3 devices is leaving the network direct after joining it and can't being configured.

Only some uncreative brainstorming.

[Hedda]

There is now is a guide by @digiblur for flashing Sonoff ZBBridge with new Tasmota-ZBBridge firmware and its Zigbee chip with an older EmberZNet firmware version that support EZSP v4 so can use it today as a network-attached Zigbee coordinator via its serial bridge with current bellows from ZHA in Home Assistant using a socket connection

https://www.digiblur.com/2020/07/how-to-use-sonoff-zigbee-bridge-with.html

Alternative way of flashing Tasmota and Silabs Ember firmware on it and other tips can be found in this other article by @NotEnoughTech

https://notenoughtech.com/home-automation/flashing-tasmota-on-sonoff-zigbee-bridge/

Can’t wait for EZSP v8 support in bellows so can use latest EmberZNet firmware on the Sonoff Zigbee Bridge

[Adminiuga]

Anyone has Zigbee Pro 2017 R22 and ZCL v7 specs?

[s-hadinger]

Thanks for the links, I'll check with a Zigbee sniffer what is going on.

[MattWestb]

Only for reference in witch EZSP (ZNet) version then the stack protocol versions was updated. V4 Its pre ZNet 5.9 V5 in ZNet 5.9 V6 in ZNet 6.0 V7 in ZNet 6.4.0 V8 in ZNet 6.7.0 and not backwards compatible with < V8.

From: https://www.silabs.com/community/wireless/zigbee-and-thread/forum.topic.html/release_notes_ofezs-bydL

[Adminiuga]

I'm going by the last EmberZNet for specific version, so v4 is up to date with docs in 5.8.10 v5 is up to date with 5.10.2 v6 is up to date with 6.3.3 v7 is up to date with 6.6.5 v8 for now is up to date with 6.7.6.0

But it means that 5.8.10 is much newer than the default one coming with HUSBZB-1, which is something like 5.4 IIRC

[MattWestb]

Great hope the V4 not making problem with the old hardware then you jumping forward in the spec versions ! From wat protocol version can my 6.6.4 going down to for helping you testing (only 7 or lower) ? And then you are ready enough and like getting help with alpha / beta testing of more device hardware only give my one cal and i do so much i can for you !

[Adminiuga]

the device is not completely saved into database. No endpoints, no clusters. Doh, wrong thread

[NilsBohr]

@Adminiuga

Happy to announce that we have released a Firmware Update utility for our products as well as EZSP v8 firmware.

Feel free to check out and comment. https://github.com/Elelabs/elelabs-zigbee-ezsp-utility

You can switch between v6 and v8 without any issues.

[Hedda]

@Adminiuga @dmulcahey @puddly @MattWestb would it not be awesome if users from ZHA GUI in Home Assistant could use bellows and this Elelabs utility script to update the firmware on their Silicon Labs based adapters? It would be very user-friendly if EFR32 firmware updates could be done via bellows and HA’s ZHA UI

[Adminiuga]

@NilsBohr the firmware flashing is awesome.

But I have a question: could you share the default and upper bounds of the configuration values baked into your firmware? For example was trying to set the size of Source Routes table to 32 and it fails with an error. IIRC default NCP firmware image was allowing this modifications.

[NilsBohr]

@Adminiuga That's a great point. In the current firmware, it is set to 7.

We will update the repository to add metadata for all the firmwares. Also, we will release an updated firmware version with Source Routes increased to 32.

[Adminiuga]

@NilsBohr is usb-to-uart chip wired for hardware flow control on ELU013? Can you let me know what pins on MG1 chip are used for rx/tx? I'd like to play with some customizations. If this is not something you'd like to share publicably, then email or dm on home assistant discord server

[NilsBohr]

@Adminiuga Hardware flow control is not wired to USB-UART. UART TX - PA0 UART RX - PA1

Please note, that you will void the product warranty and we will not accept returns/make refunds if you flash a custom firmware. Messaged you on Discord.

[Adminiuga]

Yep, I understand and accept the warranty/refund ramifications.

Just to double check, in your message the uart is NCP uart, not the host UART, is it? NCP UART TX - PA0 --> host UART rx NCP UART RX - PA1 <-- host UART tx Is this correct?