Native serial UART protocol support for RaspBee and ConBee in Home Assistant without deCONZ software

[Gamester17]

Dresden-Elektronik should consider adding native serial protocol (UART) support for ConBee and RaspBee hardware to open source home automation software such as Home Assistant and its Hass.io (Home Assistant appliance OS for RAspberry Pi).

That is, make open source home automation software such as Home Assistant support using ConBee and RaspBee adapters directly via serial potocol over UART and CLI using existing open source Zigbee libraries without the need to the deCONZ gateway software.

Commercial motivation for Dresden-Elektronik; open source home automation software such as Home Assistant and Hass.io (Home Assistant embedded Linux OS for Raspberry Pi) are the very popular today and as they are already available and used by perhaps millions of people right now, Dresden-Elektronik would most likely sell many more devices if its RaspBee and ConBee hardware had native support them inside open source home automation software such as Home Assistant and Hass.io

Please checkout: https://home-assistant.io

as well as

https://home-assistant.io/hassio/

Home Assistant recently gained native serial protocol (UART) support for ZigBee Home Automation via the ZigPy (Zigbee for Python) open source library that currently can only communicate with XBee and EmberZNet based devices (competing USB adapter by Silicon Labs) via other Python modules which in turn communicates with different radios native serial protocol (UART) for zigpy, for more information see:

https://github.com/zigpy/zigpy

https://github.com/zigpy/bellows https://github.com/zigpy/zigpy-xbee https://github.com/doudz/zigpy-zigate

Update! @damarco and @Equidamoid have both independently started working on zigpy radio libraries a native serial UART protocol for Conbee/Raspbee here:

https://github.com/zigpy/zigpy-deconz

https://github.com/Equidamoid/pyconz/

See also:

https://home-assistant.io/components/zha/

https://github.com/home-assistant/home-assistant/tree/dev/homeassistant/components/zha

https://github.com/home-assistant/home-assistant/pull/6263

https://github.com/home-assistant/home-assistant/pull/12187

https://github.com/home-assistant/home-assistant/pull/19664

https://github.com/home-assistant/home-assistant-polymer/pull/2389

https://github.com/home-assistant/home-assistant-polymer/pull/2421

[damarco]

@manup I noticed that in the response to a change_network_state request (command id 0x08) the network state is always 0 (=NET_OFFLINE), irrespective of the actual state. Is that expected?

[damarco]

@manup Is there a way to learn both short and long addresses of a device directly via the serial protocol without going to any of the upper layer protocols (ZDP, etc.)? The aps_data_indication response always only includes either the short or long address.

[dhylands]

My understanding is that the answer to your question is no, but somebody else may know better.

Many devices send out an End Device Announcement message which includes both the 16 and 64 bit addresses.

Otherwise, I think you need to use something like the Management LQI request/response, which may not include sleeping devices (depends on when things got power cycled and what routers you have on your network etc).

[damarco]

Thanks Dave! There seems to be a command that works in the other direction (command 0x1D), which updates the firmware about a certain mapping. So I was wondering if there is maybe another command to query the firmware about a mapping.

[damarco]

@manup One more question, I hope you don't mind.. I still see messages from the bootloader every hour. In another thread you mentioned that this is due to a watchdog trigger even though I do have devices connected. Anything else that can be done to prevent those resets?

[Hedda]

@Equidamoid note that @damarco is also working on library for deconz serial protocol for zigpy here:

https://github.com/damarco/zigpy-deconz

https://github.com/Equidamoid/pyconz/

[damarco]

@manup I still see the watchdog trigger every hour, any hint would be appreciated.

[manup]

Sorry, there is a undocumented reset watchdog command. I'll add it to the documentation during the next days.

[damarco]

Great, thanks!

[Hedda]

FYI, Home Assistant will soon be getting nice config panel GUI for configuring ZHA (via the zigpy library):

This will make configuring Zigbee devices in Home Assistant much more accessible (similar to Z-Wave):

https://github.com/home-assistant/home-assistant/pull/19664

https://github.com/home-assistant/home-assistant-polymer/pull/2389

https://github.com/home-assistant/home-assistant-polymer/pull/2421

[jehrichs]

@manup I start to like your USB Stick on my beaglebone. But it seems that the linked serial protocoll pdf in this thread is rather outdated.

Could you please add me to the https://github.com/dresden-elektronik/deconz-serial-protocol repo as well? The APS_DATA_REQUEST seems to have been expanded. Thanks :)

[manup]

The APS_DATA_REQUEST seems to have been expanded. Thanks :)

Done :)

[rapture216]

@manup I didn't realize the document had been updated - could I be added to that other repo as well?

[manup]

Done, note there is only one repo. The docs will also be made public somewhere later this year.

[rapture216]

Thanks.

[Gamester17]

@manup Thanks for mentioning that docs been updated, (PDF-file name has no version so did not notice).

Edit: Docs, by the way, needs to be updated to say "Zigbee" instead of "ZigBee" as the official Zigbee Alliance changed the Zigbee name stylization from “ZigBee” to “Zigbee”, as in they are no longer writing Zigbee with a capital “B” in the middle of the name.

[damarco]

@manup Any additional information about the reset watchdog command? It is the only thing holding me back from releasing the zigpy library. A short example would be enough.

[manup]

@damarco sorry for the delay, looking forward to update the description this week.

Maybe helpful, the watchdog is controlled by a unsigned 32-bit parameter (TTL in seconds):

    ZM_DID_DEV_WATCHDOG_TTL = 0x26

You can read and write this parameter with the usual functions. deCONZ for example sets the parameter from time to time to 3600 seconds if everything seems to work.

[damarco]

Thanks, @manup ! That did the trick. The library is now available here: zigpy-deconz

[damarco]

@manup Now that native support is added to home assistant I see several instances with older firmware versions which apparently do not support the watchdog_ttl parameter (like 261E0500 for example). Is it a different parameter id for older firmware versions?

On another note, as I only have a RaspBee at the moment, would you be willing to provide a ConBee stick for development purposes?

[manup]

@manup Now that native support is added to home assistant I see several instances with older firmware versions which apparently do not support the watchdog_ttl parameter (like 261E0500 for example). Is it a different parameter id for older firmware versions?

The watchdog parameter was introduced in firmware version 0x261f0500.

It is not supported on older versions, so that a request to write this parameter will return a response of 0x04 (unsupported) but otherwise shouldn't cause any harm.

On another note, as I only have a RaspBee at the moment, would you be willing to provide a ConBee stick for development purposes?

Sure just send me your shipping address to mpi@dresden-elektronik.de with a reference of this issue here. I'll forward it.

[damarco]

Thanks, manup! Is there a way to query the device for its firmware version? For older firmware versions, is there another way to prevent watchdog resets?

[manup]

Thanks, manup! Is there a way to query the device for its firmware version?

Yes with the VERSION (0x0D) command, it returnes the fimrware version as 32-bit unsigned int.

For older firmware versions, is there another way to prevent watchdog resets?

Older versions either don't reset at all or will reset the watchdog as long as activity is detected. Like sending/receiving commands due polling.

[Adminiuga]

The watchdog parameter was introduced in firmware version 0x261f0500.

Hrm, I was running on 0x261F0500, but was getting UNSUPPORTED status code when writing reset watchdog config parameter.

[manup]

Ah sorry my bad the actual test is firmwareVersion > 0x261f0500

[damarco]

Yes with the VERSION (0x0D) command, it returnes the fimrware version as 32-bit unsigned int.

What is the format of the request? it seems to be missing from the documentation..

[Adminiuga]

@manup is there a way to read the configured endpoint ID in case user changed the defaults?

[manup]

@damarco the following code describes the command pretty well (will be added in the next docs release)

https://github.com/mozilla-iot/serial-prober-node/blob/16c300d584b7e72cc59975d161c3828da8d0b857/serial-test.js

[manup]

@Adminiuga yes there is an undocumented API command.

However I'd suggest just use a normal ZDP Active_Endpoints_req followed by respective ZDP Simple_Descriptor_req with the coordinator as destination address.

This is quite handy since the same code can be used to read endpoints and simple descriptors of other nodes in the network.

[Adminiuga]

@manup what is the command 0x19 we're seeing when trying to pair a device? This is on 0x262f0500 firmware? The full frame looks like:

Feb 21 01:04:00 hass hass[2786]: 2019-02-21 01:04:00 DEBUG (MainThread) [zigpy_deconz.uart] Frame received: 0x191b0016000f008c3094354300c70a000022bd60d392
Feb 21 01:04:00 hass hass[2786]: 2019-02-21 01:04:00 DEBUG (MainThread) [zigpy_deconz.api] Unknown command received: 25
Feb 21 01:04:00 hass hass[2786]: 2019-02-21 01:04:00 DEBUG (MainThread) [zigpy_deconz.uart] Frame received: 0x191c0016000f008c3094354300c70a000022bd60d392

[manup]

0x19 – Zigbee Green Power Data Indication

The payload is just a raw pass through of a received ZGP message. First byte is the NWK frame control and the rest as described in the Zigbee Green Power specification.

These frames are typically send by ZGP devices like the Philipis Hue Tap and new Philips Friend of Hue switches.

[Gamester17]

Is it by the way possible to upgrade the firmware of Zigbee end-point devices through this serial API?

Would be nice if Home Assistant could upgrade the firmware of both adapters as well as endpoint devices.

[ebaauw]

Is it by the way possible to upgrade the firmware of Zigbee end-point devices through this serial API?

Technically: yes. Practically: you would have to implement an OTAU server (cluster 0x0019) in Home Assistant, effectively rebuilding the (closed source) OTAU plugin. Imho that's an awful lot of work for something that "would be nice".

Also note that not all manufacturers make available the firmware of their devices to other Zigbee gateways, e.g. Hue devices can only be firmware upgraded through the Hue bridge.

[RandomHumanoid]

@manup Hi, Please could you add me to the https://github.com/dresden-elektronik/deconz-serial-protocol repo.

I bought a ConBee with the intent of implementing my own Raspberry Pi-based controller software using it and it sounds like this document will fill in the missing bits between what I see in the protocol document shared publicly on your company website and the functionality I see available in ZigBee from reading the specs. I understand and respect that you don't want your support people to be bothered by lots of amateur developers with questions.

Thanks in advance

[Hedda]

@manup Will the new ConBee 2 (ConBee II) hardware be backwards compatibility with old serial protocol?

More than one reseller already leaked that the new ConBee hardware version will be released very soon.

[manup]

Yes the serial protocol is identical and will be maintained for all devices equally :)

[strokkur]

@manup Could you add me to the https://github.com/dresden-elektronik/deconz-serial-protocol repo? It would be nice to have a low-level interface between ConBee and Home Assistant.

[Kane610]

@strokkur

@manup Could you add me to the https://github.com/dresden-elektronik/deconz-serial-protocol repo? It would be nice to have a low-level interface between ConBee and Home Assistant.

There already is, ZHA integration already have Conbee/Raspbee support

[Adminiuga]

zigpy-deconz being the low-level interface for HA integration.

[Kane610]

@strokkur if you want to contribute talk to @Adminiuga or @dmulcahey on hass discord 👍

[Hedda]

FYI; the new "ConBee II" (a.k.a. "ConBee 2") hardware has officially been released now.

Apparently the new "ConBee II" ZigBee USB dongle/stick features improved signal amplifier with longer radio range and it is based on a 32-bit ARM-Cortex-M0 SoC microcontroller (Microchip ATSAMR21E18A) instead of an 8-bit AVR based CMOS microcontroller (Atmel ATmega256RFR2) like the previous version of ConBee and the current RaspBee versions.

• https://shop.dresden-elektronik.de/conbee-2.html
• https://www.dresden-elektronik.de/fileadmin/Downloads/Dokumente/Produkte/11_Phoscon_gateway/Phoscon_ConBee-II_10-2018_eng.pdf
• http://www.amazon.de/dp/B01FDWOIHK

[dhylands]

For some messages that I'm sending I'm getting back an INVALID_PARAMETER status code. I'm talking to an IKEA motion sensor. It sends a genOnOff - onWithTimedOff and I then respond with a defaultRsp. The defaultRsp gets an INVALID_PARAMETER confirmStatus.

2 questions:

1 - Any idea why I'm getting an INVALID_PARAMETER? 2 - What does device state 0x80 indicate (it isn't mentioned in the documentation)

While preparing this, I noticed that the command identifier in the defaultRsp was set to 0x00 rather than 0x42. I updated my code to send the 0x42, but I'm still getting the INVALID_PARAMETER.

Here's my decoded packets starting with the reception of the onWithTimedOff up to the INVALID_PARAMETER:

Rcvd: 0e 42 00 07 00 aa 00 ff fe

0e    Device State Changed
42    Sequence Number
00    Status
07 00 Frame Length
aa    Device State 0x80-unknown 0x20-free slots 0x08-Data indication 0x02-Connected
00    Reserved
ff fe Check

Sent: 17 42 00 07 00 00 00 a0 ff

17    Data Indication - Read Received Data request
42    Sequence Number
00    Reserved
07 00 Frame Length
00 00 Payload Length
a0 ff Check

Rcvd: 17 42 00 2c 00 25 00 22 01 62 92 a3 03 5f dd 8e fe ff 57 0b 00 01 04 01 06 00 08 00 01 0d 42 00 58 02 00 00 00 af ff a2 00 01 02 db 84 f5

17    Data Indication - Read Received Data Response
42    Sequence Number
00    Status
2c 00 Frame Length (44)
25 00 Payload Length (37)
22    Device State 0x20-free slots 0x02-Connected
01    Destination Address mode - Group
62 92 NWK address - I'm assuming this is a group created by the dongle for the IKEA support
a3    Destination endpoint
03    Source Address
5f dd 8e fe ff 57 0b 00   - MAC address of IKEA motion sensor
01    Source endpoint
04 01 Profile ID
06 00 Cluster ID (genOnOff)
08 00 ADSU Length
01 0d 42 00 58 02 00 00 ADSU payload
  01    - frame Cntl - frameType: 1 (cluster specific), manufSpec: 0, direction: 0, disDefaultRsp: 0
  0d    - ZCL sequence
  42    - command - On with timed off
  00    - on/off control
  58 02 - on time (600)
  00 00 - off wait time
00    Reserved
af    Reserved
ff    LQI
a2 00 Reserved
01 02 Reserved
db    RSSI
84 f5 Check

Sent: 12 44 00 1b 00 14 00 12 00 02 25 f1 01 04 01 06 00 a3 05 00 18 0d 0b 00 00 04 00 69 fd

12    Data Request (Enqueue Send Data Request)
44    Sequence
00    Reserved
1b 00 Frame len (27)
14 00 Payload len (20)
12    Request ID
00    Flags
02    Dest Addr Mode - 02 = NWK address
25 f1 NWK address
01    Destination endpoint
04 01 Proifile ID
06 00 Cluster ID
a3    Source Endpoint
05 00 ADSU len
18 0d 0b 00 00  ADSU data - defaultRsp
04    Use APS Acks
00    Radius
69 fd Check

Rcvd: 12 44 00 09 00 02 00 22 12 6b ff

12    Data Request (Enqueue Send Date Response)
44    Sequence
00    Status
09 00 Frame len
02 00 Payload len
22    Device State  0x20-Free slots 0x02-connected
12    Request ID
6b ff Check

Rcvd: 0e 45 00 07 00 a6 00 00 ff

0e    Device State Changed
45    Sequence Number
00    Status
07 00 Frame Length
a6    Device State 0x80-unknown 0x20-Free slots 0x04-confirm 0x02-connected
00    Reserved
00 ff Check

Sent: 04 45 00 07 00 00 00 b0 ff

04    Data Confirm (Query Send Data State Request)
45    Sequence Number
00    Reserved
07 00 Frame len
00 00 Payload len
b0 ff Check

Rcvd: 04 45 00 13 00 0c 00 22 12 02 25 f1 01 a3 a6 00 00 00 00 02 fd

04    Data Confirm (Query Send Date Response)
45    Sequence Number
00    Reserved
13 00 Frame Length (19)
0c 00 Payload Length (12)
22    Device State  0x20-Free slots 0x02-connected
12    Request ID
02    Dest addr mode (NWK)
25 f1 NWK Address
01    Destination endpoint
a3    Source Endpoint
a6    Confirm Status - A6 = INVALID_PARAMETER
00 00 Reserved
00 00 Reserved
02 fd Check

[manup]

Two things that I see:

1) a3 Source Endpoint

Is your gateway configured to have this endpoint? Unless otherwise configured, the default HA endpoint is 0x01 for RaspBee/ConBee firmware.

2) I don't think you need/should to send a ZCL default response back since the former command was a group-cast. I guess IKEA sets the disable default response: 0 wrongly here.

Default response is usually needed for some (unicast) ZCL attribute reports, like Philips Hue motion sensor and Dimmer requires them to stay connected.

[adetante]

@manup can I access the serial protocol documentation? I'm writing a wrapper in rust to communicate with the Conbee, based on this document but it doesn't seem complete?

[manup]

That is basically the latest document, it covers all main parts needed to implement gateway on top of it. The important missing part is the watchdog description, it will be added in the next revision. But is also described here in this issue above.

You may also take a look into the other open implementations of Mozilla Things Gateway https://github.com/mozilla-iot/deconz-api and Zigpy https://github.com/zigpy/zigpy-deconz

[adetante]

OK, thanks for your feedback. I thought it wasn't the latest version, because the "Permit Join" parameter is not described, for example. I will look at the implementation of Mozilla.

[dhylands]

Two things that I see:

1. `a3    Source Endpoint`
   Is your gateway configured to have this endpoint? Unless otherwise configured, the default HA endpoint is 0x01 for RaspBee/ConBee firmware.

2. I don't think you need/should to send a ZCL default response back since the former command was a group-cast. I guess IKEA sets the disable default response: 0 wrongly here.
   Default response is usually needed for some (unicast) ZCL attribute reports, like Philips Hue motion sensor and Dimmer requires them to stay connected.

Thanks for the insights.

I changed the a3 to 01 and the invalid_param went away. The reason I was using a3 in the first place was because the "on with timed off" message from the motion sensor arrived with a destination endpoint of a3. I assumed that had something to do with special support for the IKEA stuff added to the deConz dongle (because I don't get any notification messages from the IKEA motion sensor when using a Digi dongle). I was taking the destination endpoint and making it the source endpoint when I sent the reply.

I'll also fix my code to not respond to group casts.

[adetante]

For information, I started to implement a Rust library for the serial protocol. Under construction!

If there are Rust developers around here, I'm interested in a code review.

[dagsunde]

@manup Could you please add me to the private repository for the serial protocol documentation for the RaspBee?

[manup]

Done :)