Native serial UART protocol support for RaspBee and ConBee in Home Assistant without deCONZ software

[Gamester17]

Dresden-Elektronik should consider adding native serial protocol (UART) support for ConBee and RaspBee hardware to open source home automation software such as Home Assistant and its Hass.io (Home Assistant appliance OS for RAspberry Pi).

That is, make open source home automation software such as Home Assistant support using ConBee and RaspBee adapters directly via serial potocol over UART and CLI using existing open source Zigbee libraries without the need to the deCONZ gateway software.

Commercial motivation for Dresden-Elektronik; open source home automation software such as Home Assistant and Hass.io (Home Assistant embedded Linux OS for Raspberry Pi) are the very popular today and as they are already available and used by perhaps millions of people right now, Dresden-Elektronik would most likely sell many more devices if its RaspBee and ConBee hardware had native support them inside open source home automation software such as Home Assistant and Hass.io

Please checkout: https://home-assistant.io

as well as

https://home-assistant.io/hassio/

Home Assistant recently gained native serial protocol (UART) support for ZigBee Home Automation via the ZigPy (Zigbee for Python) open source library that currently can only communicate with XBee and EmberZNet based devices (competing USB adapter by Silicon Labs) via other Python modules which in turn communicates with different radios native serial protocol (UART) for zigpy, for more information see:

https://github.com/zigpy/zigpy

https://github.com/zigpy/bellows https://github.com/zigpy/zigpy-xbee https://github.com/doudz/zigpy-zigate

Update! @damarco and @Equidamoid have both independently started working on zigpy radio libraries a native serial UART protocol for Conbee/Raspbee here:

https://github.com/zigpy/zigpy-deconz

https://github.com/Equidamoid/pyconz/

See also:

https://home-assistant.io/components/zha/

https://github.com/home-assistant/home-assistant/tree/dev/homeassistant/components/zha

https://github.com/home-assistant/home-assistant/pull/6263

https://github.com/home-assistant/home-assistant/pull/12187

https://github.com/home-assistant/home-assistant/pull/19664

https://github.com/home-assistant/home-assistant-polymer/pull/2389

https://github.com/home-assistant/home-assistant-polymer/pull/2421

[mrjerryjohns]

This is the raw serial data I received from the Raspbee - seems like the ASDU payload has an extra starting byte?

c0 - Start of Frame

17 - APS_DATA_INDICATION 
06 - SeqNumber
00
3f
00
38
00
2e - Device State
02 - Destination Address Mode (NWK)

00 - Destination Address
00

00 - Dest Endpoint
03 - Source Address Mode (IEEE)

7b - Source Address
79
02
ff
ff
2e
21
00

00 - Source Endpoint

00 - Profile Id
00

31 - Cluster ID
80

1b - ASDU Length
00

00 - ASDU
00
01
00
01
7b
79
02
ff
ff
2e
21
00
1d
6d
4c
03
01
88
17
00
32
57
35
00
00
f7

00 - Reserved
af

00 - LQI

a2 - Reserved
00
01
02

96 - RSSI

cd - CRC
f2

c0 - End of Frame

[manup]

Just checked the spec, looks good I forgot one field :)

First 0x00 is the ZDP sequence number followed by ZDP status.

[mrjerryjohns]

No worries!

What does this sequence number map to? Is this similar to the Request ID in the request/confirm?

[manup]

It should be a copy from the sequence number you specified in the request (here 0x00).

[mrjerryjohns]

It doesn't match the sequence number I sent in the Read Received Data Request (CommandId = 0x17. SequenceNum = 0x06):

[SerialProtocol] TX: c0
[SerialProtocol] TX: 17
[SerialProtocol] TX: 06
[SerialProtocol] TX: 00
[SerialProtocol] TX: 07
[SerialProtocol] TX: 00
[SerialProtocol] TX: 00
[SerialProtocol] TX: 00
[SerialProtocol] TX: dc
[SerialProtocol] TX: ff
[SerialProtocol] TX: c0

It doesn't match the RequestId either that I sent in the Data Request.

[mrjerryjohns]

I tested it out with another request, and I noticed that the first byte in the ASDU payload in the indication always matches the first byte in the ASDU payload sent in the data request. I validated this by trying out various different values and it always tracked.

[manup]

Yes that's what I meant, the first byte in the ASDU (ZDP) frame is the ZDP sequence number.

The sequence number above (0x06) is on UART protocol level :)

[mrjerryjohns]

Got it, thanks @manup.

[mrjerryjohns]

While there is a 'requestId' that ties the confirm and enqueue messages, is there a similar ID I can use to to tie the indication to the other two? While the ZDP sequence number is usable, it is at a higher layer, and not usable across other profiles.

[mrjerryjohns]

@manup, any suggestions?

[manup]

Sorry missed the last message.

No that's not possible, since data indications on APS level are not bound to a request, you may receive APS indications without prior requests. That's the reason why higher layers like ZDP and also ZCL frames contain a sequence number to handle such relationships.

[apper13]

@manup Could you please invite me to the serial protocol repository? Just received my conbee :)

[davidlzs]

@manup, if possible, would you please add me to the serial protocol repository as well? Got to play with my Rasbee and Conbee via serial port.

[mrjerryjohns]

No that's not possible, since data indications on APS level are not bound to a request, you may receive APS indications without prior requests.

Would have been great if Zigbee had went the approach of other standard networking protocols and had sequence numbers at the various layers to prevent bleeding across layers. The APS indications could have had a flag in them that indicated whether they were in response to a prior request, or initiating the request from the sender. That would have trivially distinguished unsolicited messages from responses.

[mrjerryjohns]

I have a Phillips Hue bulb (A19) that I'm able to successfully query its descriptor for. However, when I try to do a ZCL discover attributes request against the ZLL profile id (0xc05e), I notice that the indication comes back from it from profile id HA (0x0104). Is this legal?

[manup]

Yes, albeit a bit strange it makes sense. This is specified in the ZLL standard, all commands on a ZLL endpoint should be transferred with the HA profile id for compatibility to HA profile. The only exception is the ZLL commissioning cluster (0x1000).

[PatrickE94]

@manup Another developer with high amibitions is requesting an invitation to the UART protocol repo as well! Bought the ConBee stick today!

[manup]

Done :)

[PatrickE94]

@manup thanks a bunch!

[Equidamoid]

Hi @manup,

I've got another question going slightly beyond the serial protocol. I have ZCL requests working (after enforcing source endpoint to 1 as you suggested), but just hit another wall when tried to get status without changing it:

• reading properties gives a response that zigpy decodes as error
• "enable reporting" call does not seem to have any effect

could you take a look? I put serial logs to a gist to avoid polluting this thread: https://gist.github.com/Equidamoid/6922ff52056cb0d344f051581e956884 .

[manup]

Glad it works so far.

By dissecting the command the read attributes response looks fine.

Reporting attributes isn't supported by Philips Hue lights so that must fail in this case.

17 ind
07 seq
00 status
2c00 frame length
2500 payload length

26 dev state
02 dst. addr nwk addr
6376 dst nwk addr
01 dst endpoint

03 src addr mode (ieee)
f3a03b0301881700 src ieee address
0b src endpoint

0401 profile id
0600 cluster id

0800 asdu length

18 zcl.frame control
   direction server -> client
   disabled default response
02 zcl.seq
01 command read atributes response
0000 attribute on/off
00 status success
10 type bool
00 off

00af reserved

af LQI

9d000104 reserved

bb RSSI

17f9 CRC

c0

[Equidamoid]

@manup, Thanks! Probably, the Osram socket switch also does not support it. But how about reporting request, is it supposed to work?

[rapture216]

@manup I didn't want to bother you, but I'm having some trouble. I'm trying to set the Network Key on the Raspbee, which should work be sending something like: 0b05000800010018c0dec0d3c0d3c0dec0dec0dec0dec0de

But when I do that, I keep getting: 0b05070800010018 Which has the status "Invalid Value." So I tried reading the parameter, and I get Invalid Value back again. So I tried just requesting a random parameter, like 0xde and it gave me back Unsupported.

So, I guess (1) when I get Invalid Value back trying to read a parameter, what does that mean versus Unsupported, and (2) is there something I'm doing wrong trying to write/set the Network Key?

[manup]

Can you provide the full frame which you are sending to write the parameter? The above looks wrong somehow.

Uncomplete dissect:

0b write param

0500 buf length (5) should be 18?

--> parameter id, key number (0), 16 byte key

08 parameter id (ZM_DID_NWK_EXTENDED_PANID) should be 0x18?!

00010018c0dec0d3c0d3c0dec0dec0dec0dec0de

edit: sorry I seem to got it wrong, need to recheck ....

[manup]

Another attempt, the request should look like:

0b WRITE_PARAMETER
05 seq
00 reserved

1900 frame length: 7 + payload length
1200 payload length: 1 + 17

18   parameter id (network key)

00   key index, THIS IS MISSING IN THE DOCS

c0dec0d3c0d3c0dec0dec0dec0dec0de key

I've found two bugs in the protocol documentation:

1) Frame length for WRITE_PARAMETER should be 7 + payload length 2) key index 0x00 must be placed before the key, this is missing in the docs

I'll check it in more detail and will update the documentation.

[rapture216]

@manup ah, yes, sorry, you're right - I was referencing my notes when I posted, and it look like I accidentally mashed together a read command and a write command when I jotted it down. Actual write was

0b WRITE_PARAMETER
07 seq
00 reserved

2800 frame length
1100 payload length
18 parameter id

c0dec0dec0dec0dec0dec0dec0dec0de key

I didn't notice before, but it looks like my code is definitely calculating frame length wrong; I need to track that down.

[rapture216]

@manup Trying to figure out what I was doing wrong with the Network Key, I found some parameter codes that were responsive to reads but not documented, and I was wondering if you knew what they were:

0x02
0x0D
0x13
0x15
0x16
0x19 (responds with INVALID_VALUE)
0x1E
0x1F (responds with INVALID_VALUE)
0x21
0x22
0x23

[manup]

Most of these are legacy parameters not used anymore and just there for backward compatibility reasons, to keep things stable when newer firmware is used with an older deCONZ version.

[rapture216]

@manup thanks so much!

[Gamester17]

Looks as if @Equidamoid made some progress on a "pyconz" UART module for the ZigPy project here:

https://github.com/Equidamoid/pyconz/

@rcloran Any chance of you inviting him to join the ZigPy project on GitHub and include it by default?

[kljord]

I'm sure you're getting sick of this, but can I get an invitation too? :)

[manup]

I'm sure you're getting sick of this, but can I get an invitation too? :)

done :)

[markusschweitzer]

@manup would it also be possible to add me to the repository? Thanks in advance :)

[manup]

@manup would it also be possible to add me to the repository? Thanks in advance :)

done :)

[basvdlei]

@manup I also just got a Raspbee and would love to have access to the UART protocol repository :)

[manup]

done again ;)

[eiaro]

@manup Mind adding me as well? Working on a Zigbee CL for NET library.

[manup]

sure, done :)

[eiaro]

I have the command VERSION (0x0d), but I lack the request and response format. Is this not used?

[manup]

Upps, indeed description is missing, added to the todo list.

The response contains the device firmware version as unsigned 32-bit integer.

For example recent version:

0x261f0500

[orktes]

@manup could you add me as well?

[manup]

sure :)

[dhylands]

@manup and a request for myself as well. I've already gotten a copy of the UART Serial Protocol document, but I've run into several discrepancies and undocumented things. I have a working Zigbee library for the Digi and I'm adding support for deCONZ. Currently, everything is coded in node, and I'm following a flavor similar to the xbee-api for the deconz support.

[manup]

Done. Thanks very much for the report I've received it from our support team and put the issues on the todo list for the next refresh of the documentation. If you have more questions about the serial protocol api or you need hardware just contact me here.

[dhylands]

I found another undocumented parameter. After sniffing the serial comms from deConz to the ConBee it seems that in order to control the permit join time on the dongle you need to write to parameter 0x21.

ID: 0x21 (33) Name: PermitDuration Type: U8 Description: 0-255 0 means disable, 0xff means allow joining anytime, other values are the number of seconds to allow a device to join the network. Mode: R/W

Does that seem correct?

[manup]

That's correct albeit more important is the related ZDP command Mgmt_Permit_Join_req since this will open the network for all routers not only the gateway (unless this is desired).

[dhylands]

Right. So my understanding is that I need to Write Parameter 0x21 to get the dongle to allow devices to connect, and I also need to send the ZDP Permit_Join command (presumably broadcast to address 0xfffc). I was already doing the ZDP broadcast, but that didn't seem to be sufficient to get the first device onto the network.

[manup]

I would expect the broadcast to be sufficient since all commands are loopbacked in the mcu, but never testet this explicitly, so the Parameter write is a safe bet.

[dhylands]

For anybody that's interested, I've created a github repository that has node.js code similar to xbee-api for talking directly to the ConBee. You can find it here: https://github.com/mozilla-iot/deconz-api I've been integrating that into the zigbee adapter that I've written for the Mozilla IoT gateway and have been able to control an outlet, It also includes some code (see dump/dump.js) for parsing/dumping the USB serial streams between a host program and the ConBee (captured via wireshark ala https://blog.davehylands.com/capturing-usb-serial-using-wireshark/)

I need to write some documentation and it needs more testing.

[damarco]

@manup Would you mind adding me to the serial protocol repo as well?