Native serial UART protocol support for RaspBee and ConBee in Home Assistant without deCONZ software

Gamester17 commented 7 years ago

Dresden-Elektronik should consider adding native serial protocol (UART) support for ConBee and RaspBee hardware to open source home automation software such as Home Assistant and its Hass.io (Home Assistant appliance OS for RAspberry Pi).

That is, make open source home automation software such as Home Assistant support using ConBee and RaspBee adapters directly via serial potocol over UART and CLI using existing open source Zigbee libraries without the need to the deCONZ gateway software.

Commercial motivation for Dresden-Elektronik; open source home automation software such as Home Assistant and Hass.io (Home Assistant embedded Linux OS for Raspberry Pi) are the very popular today and as they are already available and used by perhaps millions of people right now, Dresden-Elektronik would most likely sell many more devices if its RaspBee and ConBee hardware had native support them inside open source home automation software such as Home Assistant and Hass.io

Please checkout: https://home-assistant.io

as well as

https://home-assistant.io/hassio/

Home Assistant recently gained native serial protocol (UART) support for ZigBee Home Automation via the ZigPy (Zigbee for Python) open source library that currently can only communicate with XBee and EmberZNet based devices (competing USB adapter by Silicon Labs) via other Python modules which in turn communicates with different radios native serial protocol (UART) for zigpy, for more information see:

https://github.com/zigpy/zigpy

https://github.com/zigpy/bellows https://github.com/zigpy/zigpy-xbee https://github.com/doudz/zigpy-zigate

Update! @damarco and @Equidamoid have both independently started working on zigpy radio libraries a native serial UART protocol for Conbee/Raspbee here:

https://github.com/zigpy/zigpy-deconz

https://github.com/Equidamoid/pyconz/

https://github.com/home-assistant/home-assistant/tree/dev/homeassistant/components/zha

https://github.com/home-assistant/home-assistant/pull/6263

https://github.com/home-assistant/home-assistant/pull/12187

https://github.com/home-assistant/home-assistant/pull/19664

https://github.com/home-assistant/home-assistant-polymer/pull/2389

https://github.com/home-assistant/home-assistant-polymer/pull/2421

Framspott commented 6 years ago

Hi Chris,

I am finally able to connect to the RaspBee! I had the wrong Port in the arguments.... He recognizes the RaspBee module and my Hue Bulb, I think.

Unfortunately I am not able to give any input to the console. He is asynchronically outputting "Node Updating" messages...

for example:

Node Updated ZigBeeNode [IEEE=00212EFFFF0171C1, NWK=0000, Type=UNKNOWN] [pool-1-thread-3] INFO com.zsmartsystems.zigbee.console.ZigBeeNetworkStateSerializerImpl - ZigBee saving network state done. Node Added ZigBeeNode [IEEE=0017880103FB6929, NWK=4EAA, Type=ROUTER] [pool-1-thread-1] INFO com.zsmartsystems.zigbee.console.ZigBeeNetworkStateSerializerImpl - ZigBee saving network state done. Node Added ZigBeeNode [IEEE=0017880103A4E06D, NWK=00E5, Type=END_DEVICE] [pool-1-thread-5] INFO com.zsmartsystems.zigbee.console.ZigBeeNetworkStateSerializerImpl - ZigBee saving network state done. Node Updated ZigBeeNode [IEEE=00212EFFFF0171C1, NWK=0000, Type=UNKNOWN] [pool-1-thread-4] INFO com.zsmartsystems.zigbee.console.ZigBeeNetworkStateSerializerImpl - ZigBee saving network state done. Node Updated ZigBeeNode [IEEE=00212EFFFF0171C1, NWK=0000, Type=UNKNOWN] [pool-1-thread-2] INFO com.zsmartsystems.zigbee.console.ZigBeeNetworkStateSerializerImpl - ZigBee saving network state done.

Slowly we are getting somewhere... ;)

All the best

Florian

Von: "Gamester17" notifications@github.com An: "dresden-elektronik/deconz-rest-plugin" deconz-rest-plugin@noreply.github.com CC: "Framspott" framspott.its-m2016@fh-salzburg.ac.at, "Mention" mention@noreply.github.com Gesendet: Mittwoch, 14. Februar 2018 16:15:40 Betreff: Re: [dresden-elektronik/deconz-rest-plugin] Native serial UART protocol support for RaspBee and ConBee in Home Assistant without deCONZ software (#158)

@Framspott not sure if your question is on-topic however pretty sure that basic information is all available in the RaspBee and ConBee user manuals respectivly

https://www.dresden-elektronik.de/fileadmin/Downloads/Dokumente/Produkte/ZLL/RaspBee-BHB-en.pdf

https://www.dresden-elektronik.de/fileadmin/Downloads/Produkte/12-ConBee/ConBee_User_Manual.pdf

You of course also need to enable and configure UART specifically for Raspberry Pi (3)

https://elinux.org/RPi_Serial_Connection

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

Framspott commented 6 years ago

Sorry, wrong email address... ;(

Framspott commented 6 years ago

Thanks for the help! Now it works! It was all about the wrong port! The website https://elinux.org/RPi_Serial_Connection helped a lot!

Thanks again and all the best!

Framspott

hard2hack commented 6 years ago

Can I get an invite to the serial protocol repo as well please? I'd like to write my own implementation as a didactic activity to then be able to contribute to other projects

manup commented 6 years ago

Done :)

aptitude commented 6 years ago

@manup Thank you for the invite!

Gamester17 commented 6 years ago

Anything we can do to encourage zigpy/bellows to prioritize support for ConBee and RaspBee hardware?

@rcloran any chance that you looked at the code that @topfs2 linked above to zigpy/bellows?

https://github.com/zsmartsystems/com.zsmartsystems.zigbee

Just suggestion to read and learn from their mistakes and success, and look at their tweaks, etc.

FYI; this library was developed by @cdjackson who also developed the OpenHAB binding for ZigBee:

https://github.com/openhab/org.openhab.binding.zigbee/

https://github.com/cdjackson

http://www.cd-jackson.com

Noticed that Eclipse Public License used is not compatible with GPL but maybe can still learn something?

topfs2 commented 6 years ago

@manup Could you add me to the serial protocol repo too?

Adminiuga commented 6 years ago

@manup Could you please invite me to the serial protocol repo too? Just bought the conbee and interested in bellows/conbee support.

manup commented 6 years ago

Heya, done :)

brianpeiris commented 6 years ago

@manup I would also like access to the protocol repo please. I'm trying to add compatibility to the Mozilla IoT gateway.

manup commented 6 years ago

@brianpeiris done :) the Mozilla IoT gateway is very interesting I've read the blog post about it a few weeks ago.

gpaglia commented 6 years ago

@manup may I also get access to the low leve serial protocol repo? Won't be bothering for support, I promise ;-) Thanks a lot

manup commented 6 years ago

@manup may I also get access to the low leve serial protocol repo? Won't be bothering for support, I promise ;-) Thanks a lot

Sorry for the delay, and feel free to ask if there are protocol questions.

Equidamoid commented 6 years ago

@manup so, I also started looking into it, aiming at having zigpy support raspbee. Is there any way of debugging the serial protocol implementation? So far I'm able to receive some reasonable data (DEVICE_STATE_CHANGED and some mysterious "0x1c" messages not mentioned in the doc), but requests I send (like '070100000800000000f0ffc0' for explicitly getting device state) seem to be ignored.

manup commented 6 years ago

You can enable debug output --dbg-prot=2 to get more insight on the protocol level. Other than that you might use a USB sniffer to see what is send between deCONZ and firmware (requires a ConBee).

The 0x1C refers to a new command ZM_CMD_MAC_POLL which is used to access mac data requests from end-devices when they poll the gateway. It's not documented yet but will be in a future version of the doc.

Equidamoid commented 6 years ago

Hi @manup,

Is it a Deconz commandline flag? I'll try it in the evening. Unfortunately I have raspbee, so no USB sniffing for me. I tried to intercept write()s with strace, but wasn't able to find a simple open-read-write sequence for the device. There is also a trick with socat and virtual tty, but I'm not sure if Deconz will agree to work with it.

Can I also get access to that serial repository mentioned above?

Thanks!

manup commented 6 years ago

Yes it's a commandline flag, you can open deCONZ like:

$ deCONZ --http-port=80 --dbg-prot=2

There is also a trick with socat and virtual tty, but I'm not sure if Deconz will agree to work with it.

Might work, in the end deCONZ just opens the serial device /dev/ttyS0.

Can I also get access to that serial repository mentioned above?

Done :)

Equidamoid commented 6 years ago

@manup thanks!

Apparently I still had linux console attached to ttyS0 and deconz web ui was too shy to complain that it also can't communicate to the device :) So in the end after fixing that and couple of typos I have it working including getting actual data packages from the device. The next step will be to understand zigpy api.

Equidamoid commented 6 years ago

Getting back to active hacking will take some time, so let's share the current status so far: https://github.com/Equidamoid/pyconz (I hope you guys don't mind abusing name of deCONZ, in other case I'm open for suggestions of better name ;))

It receives the data and logs the packets received. I haven't tried sending commands since I have not clue so far how to generate a proper command.

The next big step is to connect my code to zigpy or some other zigbee implementation.

PS: Shouldn't this discussion be moved somewhere else, it seems to be a big fat offtopic for "deconz-rest-plugin" repo.

samvik commented 6 years ago

@manup Any chanse I could get access to the serial protocol repository? Thanks.

saurabh984 commented 6 years ago

@manup Hi Manuel, I'm working on the ZigBee binding for OH2 and would like access to the documentation please.

Thanks

manup commented 6 years ago

done :)

jlmwlv commented 6 years ago

@manup Hi - could I also have access to the serial protocol documentation? I'm trying to diagnose registration problems with an unsupported device

flavio20002 commented 6 years ago

My RaspBee just arrived from Amazon...I'm in!

DominikGebhart commented 6 years ago

@manup Could you add me to the serial protocol repo too? Thanks in advance!

metint commented 6 years ago

@manup

We have ordered and just received a ConBee dongle. We are planning to use it as part of our gateway implementation. But we have realized that it is not possible with the current public interface. Can you also add me to the Serial Protocol repo so we can make a better use of it? Thank you.

manup commented 6 years ago

done :)

MadMouse commented 6 years ago

@manup Would it be possible to get access to the serial Repo, I plan on integrating on Android things.

manup commented 6 years ago

Done, .. cool looking forward to it :)

user-ks commented 6 years ago

@manup and all other contributors to this tread: Thank you VERY MUCH for all the postings on this topic. It was very informative reading. As a mere mortal user wanting to implement Home Assistant + zigbee , I have read the entire thread in full.

But I have a gentle/kind question... Can someone who is actively involved post a summary on the current state of development and achievements on this matter? (this thread is now running for 10 months, which seems quite a long time in the fast-moving IoT world...)

So, what is the summary status today on:

direct (UART) support for zigbee transceivers
RESTful API support (eventually via via Docker image of deCONZ by dresden elektronik)

My apologies for asking this rather dumb question here to the active developers. But I guess that many users trying to implement HA + zigbee would like to know how to practically proceed. (If the question might be off-topic, apologies and thanks for pointing me in the right direction and/or to the right resource.)

Kane610 commented 6 years ago

@user-ks Home assistant is fully integrated to using conbee/raspbee through the deconz component which implements deconz web api, so the implementation is not stand alone but uses Dresden elektroniks (DE) software to bridge to the hardware and zigbee devices. There is also an community add-on/container that you can with minimal effort use deconz in a standalone container or in an hass.io environment. All information is linked to from the deconz components website.

Gamester17 commented 6 years ago

@user-ks sorry there is no news on any native UART API support for Home Assistant as far as I know.

10-months might seem long to you but its really not in FOSS (free and the open source) communities. FOSS projects and sub-projects usually means that people involved do the work without getting paid and as such mostley only volunteer to code specific things that personally interest them and seem fun as they will be working on it in their spare time for free. You can not set a dead-line or time-estimate when you do not have a boss priotitizing what you should spend your time on and when. This is also unfortunately why many FOSS projects and sub-projects o unfinished, gets abandoned or never gets released even though someone might have started coding it. The benefit with GitHub is that you can code an unfinshed project and sub-project in public and others have the option to help you or fork and take over the lead should you not have time to finish be forced to abandon it due to other prioritizes.

PS: Any questions regarding non-native UART API are off-topic for this so no discussing those here.

Equidamoid commented 6 years ago

Hi all,

I'm still in a struggle of making raspbee to work with zigpy (which can be then used for HomeAssistant). At this point I managed to make ZDO requests work, and after hacking zigpy to support ZDO request 0x0031 (LQI, effectively querying for neighbours of a device) I get some meaningful responses (like IEEE addresses matching ones visible in deCONZ api). That kind of proves that the low-level things work.

Also I get steady flow of messages from one of my hue dimmers by some reason, probably when I was playing with deCONZ it managed to subscribe somehow.

However, if I try to, for example, toggle a light (cmd 0x02, cluster 0x06) or read it's state (property 0x0 of the same cluster) nothing happens at all. It looks like any attempts to communicate to endpoints other than 0 (zdo) are completely ignored.

So far I have couple of ideas, but can't find a reasonable way to check them:

my raspbee is configured to join as a router (the coordinator is another raspbee, controlled by deCONZ), are routers capable of interacting with end devices? I could try to use the coordinator one, but I don't really want to run around my home reassembling the network from scratch if something goes wrong, so I haven't tried yet %)
there is a secret "make it work" command to make endpoint recognize me? I tried "ZDO bind" (0x21), which AFAIU should be something like subscription to changes in given cluster, get 0 in payload in response ("success", right?), but nothing happens after that. Can't find any evidence of it in the docs.
something is broken in zigpy (possibly by the terrible way I attached it to my code) -- can't easily check, comparing sniffed deCONZ communication with my logs will take quite some time.

Any ideas?

manup commented 6 years ago

However, if I try to, for example, toggle a light (cmd 0x02, cluster 0x06) or read it's state (property 0x0 of the same cluster) nothing happens at all. It looks like any attempts to communicate to endpoints other than 0 (zdo) are completely ignored.

Since your setup already works with ZDO commands I figure there might be something missing with the ZCL payload/header. Can you post the full frame you are sending — for example to toggle a lights on/off cluster?

mrjerryjohns commented 6 years ago

@manup, would you be able to add me to the serial protocol repo as well? Thank you immensely!

Equidamoid commented 6 years ago

@manup Here is what I write to the serial port for ZCL (hex-encoded): [2018-07-16 17:04:39,381 - cli->dev: c0c0120600190012000200022a200304010600030300010202000551ffc0

The device is a "smart plug" from Osram, deCONZ calls it ('OSRAM', 'Plug 01').

manup commented 6 years ago

From dissecting the frame the src endpoint looks wrong, it should be 1 if you haven't changed deCONZ settings for coordinator endpoints.

c0 // c0 twice?
c0
12   APS_DATA_REQUEST
06   sequence number
00   reserved 0
1900 framelength (7 + payload length)
1200 payload length
02   request ID
00   flags
02   destination address mode (0x02 NWK address)
2a20 NWK dst address
03   dst endpoint
0401 profile id
0600 cluster id
03   src endpoint (should be 1 with deCONZ default settings)
0300   ASDU length
010202 ASDU
00  tx options
05  radius
51ff CRC16
c0

Equidamoid commented 6 years ago

@manup thanks a lot! It worked! Could you explain a bit or point to some docs/keywords for googling where (and how) does this explicit source endpoint 1 come from?

manup commented 6 years ago

Endpoint 1 is the default Home Automation endpoint of the deCONZ coordinator configuration for ConBee/RaspBee.

mrjerryjohns commented 6 years ago

@manup, would you be able to add me to the serial protocol repo as well? Thank you immensely!

Not sure if you saw this, but any way I could check out your serial protocol repo? Thanks!

manup commented 6 years ago

Check :)

mrjerryjohns commented 6 years ago

Thank you!

DvStephen commented 6 years ago

@manup appreciate you are getting a lot of requests for the serial protocol repo access but when you get chance if you could add me, I've been working with digi based xbee devices over UART in node.js so I'm interested in working on similar for the raspbee.

manup commented 6 years ago

Check :)

mrjerryjohns commented 6 years ago

@manup, I've been trying to interface directly with my Conbee dongle using the serial protocol you shared. I issued a very simple DEVICE_STATE query, confirming that it matched byte-for-byte with the query the main deCONZ app is doing (by sniffing the UART bytes). Despite that, I get no response from the dongle.

My application opens up a handle to the UART device, sets it up using the following code-snippet:

spd = B115200;
cfsetospeed(&tty, (speed_t)spd);
cfsetispeed(&tty, (speed_t)spd);

cfmakeraw(&tty);

tty.c_cc[VMIN] = 1;
tty.c_cc[VTIME] = 0;

tty.c_cflag &= ~CSTOPB;
tty.c_cflag &= ~CRTSCTS;    /* no HW flow control? */
tty.c_cflag |= CLOCAL | CREAD;

tcsetattr(mFd, TCSANOW, &tty);

Any ideas why?

manup commented 6 years ago

Upps just checked the protocol documentation, seems the baud rate is missing, the MCU only supports 38.400 baud reliably. B115200 is not supported, please use B38400.

Here is n snipped from older unix connection code (meanwhile we only use Qt serial port interface), not sure if it still works:

    struct termios2 attr;
    speed_t speed = 38400;

    bzero(&attr, sizeof(attr));

    if (ioctl(comFd, TCGETS2, &attr) == -1)
    {
        DBG_Printf(DBG_ERROR, "%s error ioctl(TCSETS2): %s\n", Q_FUNC_INFO, strerror(errno));
        return -1;
    }

    attr.c_ispeed = speed;
    attr.c_ospeed = speed;
    attr.c_cflag &= ~CBAUD;
    attr.c_cflag |= BOTHER;

    //cfmakeraw(&attr);

    // manual make raw (parameters from man page)
    attr.c_iflag &= ~(IGNBRK | BRKINT | PARMRK | ISTRIP | INLCR | IGNCR | ICRNL | IXON);
    attr.c_oflag &= OPOST;
    attr.c_lflag &= ~(ECHO | ECHONL | ICANON | ISIG | IEXTEN);
    attr.c_cflag &= ~(CSIZE | PARENB);
    attr.c_cflag |= CS8;

    if (ioctl(comFd, TCSETS2, &attr) == -1)
    {
        DBG_Printf(DBG_ERROR, "%s error ioctl(TCSETS2): %s\n", Q_FUNC_INFO, strerror(errno));
        ...
    }

mrjerryjohns commented 6 years ago

Yup, changing the baud-rate did the trick, thanks @manup!

mrjerryjohns commented 6 years ago

Have gotten quite far, been able to enqueue APSDE requests as well as get Indications back.

Have issued a Mgmt_Lqi_Req (ProfileId: 0, ClusterId: 0x0031) against the NWK Address of 0x0000, Endpoint 0, and I got the following ASDU payload:

[APSME] 00
[APSME] 00
[APSME] 01
[APSME] 00
[APSME] 01
[APSME] 7b
[APSME] 79
[APSME] 02
[APSME] ff
[APSME] ff
[APSME] 2e
[APSME] 21
[APSME] 00
[APSME] 1d
[APSME] 6d
[APSME] 4c
[APSME] 03
[APSME] 01
[APSME] 88
[APSME] 17
[APSME] 00
[APSME] 32
[APSME] 57
[APSME] 35
[APSME] 00
[APSME] 00
[APSME] f7

Parsing using the Mgmt_Lqi_Rsp frame spec, it looks like the status = 0, NeighborTableEntries = 0, StartIndex = 1 (why 1?) and NeighborTableListCount = 0.

However, there's clearly an item in the NeighborTableList, because there's data after that.

I've compared with the response that DeCONZ gets as well and it gets the same exactly response.

Any ideas why it doesn't seem to match the Zigbee spec?

manup commented 6 years ago

Looks good to me, but the first byte is too much?

[APSME] 00 ?
[APSME] 00 ZDP status
[APSME] 01 neighbor table entries
[APSME] 00 start index
[APSME] 01 neighbor table list count

[APSME] 7b Ext. PanId
[APSME] 79
[APSME] 02
[APSME] ff
[APSME] ff
[APSME] 2e
[APSME] 21
[APSME] 00

[APSME] 1d Ext. Address 
[APSME] 6d
[APSME] 4c
[APSME] 03
[APSME] 01
[APSME] 88
[APSME] 17
[APSME] 00

[APSME] 32 NWK address
[APSME] 57

[APSME] 35 Flags ...

[APSME] 00 Permit join

[APSME] 00 Depth

[APSME] f7 LQI

dresden-elektronik / deconz-serial-protocol

Native serial UART protocol support for RaspBee and ConBee in Home Assistant without deCONZ software #2