Native serial UART protocol support for RaspBee and ConBee in Home Assistant without deCONZ software

Gamester17 commented 6 years ago

Dresden-Elektronik should consider adding native serial protocol (UART) support for ConBee and RaspBee hardware to open source home automation software such as Home Assistant and its Hass.io (Home Assistant appliance OS for RAspberry Pi).

That is, make open source home automation software such as Home Assistant support using ConBee and RaspBee adapters directly via serial potocol over UART and CLI using existing open source Zigbee libraries without the need to the deCONZ gateway software.

Commercial motivation for Dresden-Elektronik; open source home automation software such as Home Assistant and Hass.io (Home Assistant embedded Linux OS for Raspberry Pi) are the very popular today and as they are already available and used by perhaps millions of people right now, Dresden-Elektronik would most likely sell many more devices if its RaspBee and ConBee hardware had native support them inside open source home automation software such as Home Assistant and Hass.io

Please checkout: https://home-assistant.io

as well as

https://home-assistant.io/hassio/

Home Assistant recently gained native serial protocol (UART) support for ZigBee Home Automation via the ZigPy (Zigbee for Python) open source library that currently can only communicate with XBee and EmberZNet based devices (competing USB adapter by Silicon Labs) via other Python modules which in turn communicates with different radios native serial protocol (UART) for zigpy, for more information see:

https://github.com/zigpy/zigpy

https://github.com/zigpy/bellows https://github.com/zigpy/zigpy-xbee https://github.com/doudz/zigpy-zigate

Update! @damarco and @Equidamoid have both independently started working on zigpy radio libraries a native serial UART protocol for Conbee/Raspbee here:

https://github.com/zigpy/zigpy-deconz

https://github.com/Equidamoid/pyconz/

https://github.com/home-assistant/home-assistant/tree/dev/homeassistant/components/zha

https://github.com/home-assistant/home-assistant/pull/6263

https://github.com/home-assistant/home-assistant/pull/12187

https://github.com/home-assistant/home-assistant/pull/19664

https://github.com/home-assistant/home-assistant-polymer/pull/2389

https://github.com/home-assistant/home-assistant-polymer/pull/2421

rtenklooster commented 6 years ago

I agree native support would be great. However in the meantime you could use node-red to publish all events to HA by using mqtt. https://github.com/dresden-elektronik/deconz-rest-plugin/issues/159

I explained how it can be done here

manup commented 6 years ago

Dresden-Elektronik should consider adding native support for RaspBee and ConBee hardware to Home Assistant open source home automation software and its Hass.io (Home Assistant appliance OS for RAspberry Pi).

Hi @Gamester17 , Home Assistant is indeed a thriving system which we'd like to support RaspBee/ConBee. However our resources are currently concentrated on the deCONZ core, REST API and the new app.

Therefore we can only offer to provide software support for REST API and low level documentation for the UART protocol between deCONZ and RaspBee/ConBee (which would be required to use the driver of the EZSP module). Beside that we can provide Hardware for the devs which like to integrate it into Home Assistant, they should be visible somehow in the HA developer community though :)

Gamester17 commented 6 years ago

FYI, rcloran who made bellows ( https://github.com/rcloran/bellows/ ) which is used by Home Assistant and Hass.io said here that he is planning on splitting out the ZigBee parts of bellows (soon -- work is in progress), which will allow you to use it with libraries that interface with others. Which means that you will possible be able to use the existing bellows implementation in Home Assistant to interface with deCONZ

https://github.com/rcloran/bellows/issues/50

abmantis commented 6 years ago

Hey @manup . If you can provide the hardware (or a cheaper price), I would gladly work on integrating it with HA, using the REST API (I would probably make a python module for DeConz, and integrate the module in HA). I've done a few contributions to HA already. ATM I have some Xiaomi sensors and switches and will buy a IKEA lamp.

donnib commented 6 years ago

I have done exactly that albeit not finished.

abmantis commented 6 years ago

Any updates?

manup commented 6 years ago

Hey @manup . If you can provide the hardware (or a cheaper price), I would gladly work on integrating it with HA, using the REST API (I would probably make a python module for DeConz, and integrate the module in HA). I've done a few contributions to HA already. ATM I have some Xiaomi sensors and switches and will buy a IKEA lamp.

Hi @abmantis can you please send your shipping address /contact to support@dresden-elektronik.de and reference me (Manu) we'll then send you a RaspBee and ConBee which you can use for your HA module development. Thanks and regards :)

abmantis commented 6 years ago

Thank you @manup ! I just sent the e-mail! I'm looking forward to work with it! :D

Gamester17 commented 6 years ago

@abmantis do you think that you will work on getting it to work with bellows by @rcloran ?

https://github.com/rcloran/bellows/

@rcloran said wrote here on issue # 50 that he will splitt bellows to make it modular:

https://github.com/rcloran/bellows/issues/50

"I'm planning on splitting out the zigbee parts of bellows (soon -- work is in progress), which will allow using that with libraries that interface with other ."

Anyway, just mentioning that as Home Assistant implementation uses bellows.

Point being that it would be nice with a plugin for Hass.io that would be easy to configure.

donnib commented 6 years ago

So here is my preliminary NOT FINISHED work : https://bitbucket.org/mihaicph/deconz-custom-module

@manup i have invested in the raspBee myself, could i kindly ask for a Conbee to investigate further issues with IKEA that we discussed in other threads ?

manup commented 6 years ago

@manup i have invested in the raspBee myself, could i kindly ask for a Conbee to investigate further issues with IKEA that we discussed in other threads ?

Absolutely, please also send a mail to support@dresden-elektronik.de with your shipping address and reference me (Manu).

abmantis commented 6 years ago

@Gamester17 it depends on when @rcloran does it I guess. I still have to look at how bellows works. Another option is helping @donnib to finish his component, if he is ok with that.

corneyl commented 6 years ago

It would be great to have native support for RaspBee/ConBee! Will this also mean that we do not have to run the Deconz GUI software? Or is it still needed to run the REST API?

manup commented 6 years ago

Will this also mean that we do not have to run the Deconz GUI software? Or is it still needed to run the REST API?

This depends on which level the integration is implemented, both is possible. REST API or serial UART protocol (much more complex, documentation available on request).

mariusmotea commented 6 years ago

can i receive UART protocol documentation? I also want to see if i can improve zigbee integration because currently deconz get overloaded very easy with multiple i/o requests

buzatu commented 6 years ago

Hi I'm also interested in the UART protocol documentation.

Hector47 commented 6 years ago

Hello I'm interested in the serial protocol documentation for the ConBee

Gamester17 commented 6 years ago

@manup any reason for not just publicizing the Serial UART protocol for RaspBee and ConBee?

I'm sure that many home automation projects would be interested, not just Home Assistant.

I would think that it would be in Dresden-Elektronik interest as it should sell more devices.

manup commented 6 years ago

I've have created a private repository which provides the download link you should receive a invitation via GitHub.

@manup any reason for not just publicizing the Serial UART protocol for RaspBee and ConBee?

It's not public since we'd like to keep the support for it low, because ZigBee at this low level can be very tricky.

Further the spec is fairly new and not ready nor simple enough for prime time. However we are absolutely ok to provide the spec for projects like Home Assistant and similar.

djwlindenaar commented 6 years ago

I've just gotten my raspbee module and am planning to help integrate it in Home Assistant. What's the status up to now? is there a repository where this is being developed?

@manup I'd like to contribute to the integration, so could you send the UART protocol specification to me?

manup commented 6 years ago

@manup I'd like to contribute to the integration, so could you send the UART protocol specification to me?

Sure, you should receive a invitation via GitHub soon.

buzatu commented 6 years ago

@manup Hello, would it be possible to have a private repository with the firmware as well?

Gamester17 commented 6 years ago

@manup Hello, would it be possible to have a private repository with the firmware as well?

Or a public repository with the firmware?

donnib commented 6 years ago

What is the benefit of having a HASS integration thru UART instead of the REST API ? Doesn't that require to replicate all the work done here in the REST API ?

manup commented 6 years ago

I'm afraid no, the firmware is provided as binary in /usr/share/deCONZ/firmware but sources won't be published any time soon, since they contain years of IP and security sensitive material.

However the RaspBee/ConBee bootloader and firmware flash tool GCFFlasher accepts custom firmware. For example the open source 802.15.4 Stack Uraculi can be flashed:

http://uracoli.blogspot.de/2013/07/raspbee-ieee-802154-module-for.html

In the future I expect firmware supporting the OpenThread stack, which might be developed and supported as open source.

manup commented 6 years ago

What is the benefit of having a HASS integration thru UART instead of the REST API ? Doesn't that require to replicate all the work done here in the REST API ?

Usually there are two points involved 1) get rid of deCONZ (which kinda makes sense) 2) not knowing the complexity and tricky parts behind device support, especially switches and sensors. In my experience new devs completely underestimate the time it takes to develop wide device support.

corneyl commented 6 years ago

I think the main disadvantage of deCONZ is that it has a GUI and therefore needs X. If it could run as a daemon without gui, with only the webinterface enabled by default, than it would make sense to use the API. Are there any plans to make deCONZ headless?

Gamester17 commented 6 years ago

What is the benefit of having a HASS integration thru UART instead of the REST API ? Doesn't that require to replicate all the work done here in the REST API ?

@donnib using UART instead of the deCONZ REST API removes the dependency of deCONZ in HASS. Yes it may means replicating work from deCONZ, but it also means that HASS could be made to only use single library and a single API to talk to multiple Zigbee adapters from different manufacturers, without having to support multiple APIs.

Using deCONZ REST API means that you first need to setup and configure deCONZ, and then always have it running. Using a serial UART protocol to talk to RasBee/ConBee adapter the existing Zigbee implementation in Home Assistant can talk directly to the hardware.

Think about it like this instead; there are today tens of Zigbee adapters available from different manufacturers, so what if HASS wanted to support all of them all they each only offered a third-party software with a REST API. Then each had to be configured seperatly, instead of having only library that talks to them all. Open source home automation software projects normally do not have the resources (volenteer developers) to maintain support for multiple Zigbee adapters/controllers from different manufacturers, and then the end-users will get less choice in which adapters/controllers are supported.

For Z-Wave this problem is solved via OpenZWave, which is one single library that supports multiple Z-Wave USB-adapters / PC-controllers from different manufacturers. For ZigBee to reach the same kind of adoption in the open source home automation scene you really also need something similar; one single library that can talk to multiple Zigbee USB-adapters / PC-controllers from different manufacturers.

Think big picture; Biggest benefit from this approach with a single library supporting multiple adapters from different manufacturers is normally that more than one open source home automation software project will re-use that one library and collaborate upstream on improving that single library.

manup commented 6 years ago

Are there any plans to make deCONZ headless?

Yes there was some work done in that regard. You can run deCONZ without a running X as systemd service, X libraries must be installed as dependencies though, so the only drawback are a few mega bytes for these.

/etc/systemd/system/deconz.service

Gamester17 commented 6 years ago

@manup are there any plans for Dresden-Elektronik to provide a C library for UART?

donnib commented 6 years ago

@Gamester17 so still to understand :

talk to multiple Zigbee adapters from different manufacturers, without having to support multiple APIs.

If you connect Philips, Xiaomi, IKEA for example to deConz then REST API is the same for all of them, no need to implement anything different, of course the manufacturers behave and have implemented the Zigbee standard so you need to take that into account no matter what e.g also in deConz REST API. My point being is that manufacterers should concentrate in implementing the Zigbee standard in a standard way as close as possible then deConz will work and give the developers a well known RESTful API which is standard. In the end one gateway and many different devices. I might be missing something here so please elaborate if i am wrong.

Using deCONZ REST API means that you first need to setup and configure deCONZ, and then always have it running.

Valid point however i don't see this as a big hassle and no matter what you would need a UI to the Zigbee devices unless you want to replicate something like deConz in HASS which i am not sure is a good idea. Who will do OTA updates ? HASS ?

Gamester17 commented 6 years ago

If you connect Philips, Xiaomi, IKEA for example to deConz then REST API is the same for all of them, no need to implement anything different

@donnib You have misunderstod. I'm not talking about different end-point devices (such as lightbulbs and sensors), but instead I mean supporting multiple Zigbee transceiver-adapters (the RF-transmitter and receiver USB/serial controller) from different manufacturers, that is the adapter that you plug into your PC or Raspberry Pi. That is not the same thing as supporting several end-point devices from different manufacturers.

RaspBee/ConBee are a transceiver hardware from Dresden-Elektronik, but Dresden-Elektronik are not the only manufactuer making Zigbee transceiver-adapters; and the goal from HASS would be to not only support RaspBee/ConBee transceiver hardware (USB/serial RF-transmitter and receiver) but also support Zigbee transceiver-adapters from other manufacturs as well, and the best thing would then be if all such adapters could be supported the same way, and even better though the same library.

Today HASS currently only support the one single Zigbee transceiver; that is the Linear HUSBZB-1 Quickstick Combo USB Controller (also sold branded as GoControl QuickStick Combo ) made by Nortek Security & Control LLC.

Maintaining support for multiple Zigbee transceiver-adapters (USB/serial RF-transmitter and receiver) from different manufacturers would be a lot of work for an open source home automation software project if all manufactuers uses completly different methods for talking to the transceiver hardware. This is why you will see open source home automation software projects today only supporting a a single Zigbee transceiver hardware from only one manufacturer.

It would make everything easier for all open source home automation software projects if all Zigbee transceiver hardware manufacturers used the same protocol. Then all open source home automation software projects could easily support all Zigbee transceiver hardware from all manufacturers.

no matter what you would need a UI to the Zigbee devices unless you want to replicate something like deConz in HASS which i am not sure is a good idea.

HASS (Home Assistant) already have a GUI for it, and it uses the same GUI for Zigbee and Z-Wave devices. One GUI to rule them all! From an end-users point-of-view usability would be horrible otherwise. Please understand that here is HASS (Home Assistant) that is the gateway, not deCONZ.

Think if you as an end-user have not only Zigbee devices, but also Z-wave devices and even 315/433 MHz RF-devices as well. Don't you see how not user-friendly it would be from an end-users point-of-view if they would have to setup and configure different gateway software for each of those. The goal for HASS would instead be to someway get to a plug-and-play state where end-users could just buy a USB-transciever-adapter for Zigbee, a USB-transciever-adapter for Z-Wave, and a USB-transciever-adapter for 315/433 MHz, and they could be from different manfacturers, so while one end-user buys a Zigbee transciever-adapter from Dresden-Elektronik a other end-user buys a Zigbee transciever-adapter Nortek Security & Control LLC and both should just work plug-and-play without the end-user having to configure anything, and the only thing left for the end-user to do is to scan/pair end-point devices from the same GUI. Again, one GUI to rule them all!

Using deCONZ is fine if all you have is Zigbee end-point devices in your ecosystem at home, but it is not good enough if you want to be user-friendly to those people who also have Z-Wave and 315/433 MHz end-point devices as well. And the deCONZ gateway is no good design at all if you are an open source home automation software project aiming to support multiple Zigbee transciever-adapters (USB/serial RF-transmitter and receiver) from different hardware manufacturers (and not just a specific adapter from one manufacturer).

Who will do OTA updates ? HASS ?

Yes the idea is that HASS would handle OTA updates of end-point devices. Other open source home automation software like Domoticz already does this for Z-Wave end-point devices. ZigBee should be same.

donnib commented 6 years ago

@Gamester17

You have misunderstood. I'm not talking about different end-point devices.....

Yes i was kinda thinking that, that might be the case. It makes sense now however that would require that each manufacturer of these implement EZSP protocol (your request here). One could have started there then implement a REST API based on the EZSP protocol i guess. I can't seem stop thinking about that even if all HW manufacturers did implement EZSP they would probably not implement it in the exact way so you would need device specific code in HASS or whatever system you have.

Thx for the explanation.

Gamester17 commented 6 years ago

It makes sense now however that would require that each manufacturer of these implement EZSP protocol

My request is not to specifically support the EZSP protocol. That is just once option. My request is for "Native serial UART protocol support for RaspBee and ConBee in Home Assistant and HASS", so specifically just not use REST API but instead use serial UART protocol to nativly talk to the hardware directly and not using a third-party gateway. The goal here is to remove the need to use a third-party gateway software.

Another option would to directly implement native support for multiple serial UART protocols in the bellows library ( https://github.com/rcloran/bellows/ ) by @rcloran as that is what HASS uses today.

It is possible to make that bellows library to talk multiple serial UART protocols with several USB-adapters from different hardware manufacturers, but it would require @rcloran to splitting out the Zigbee parts of bellows as mentioned here => https://github.com/rcloran/bellows/issues/50

manup commented 6 years ago

and the best thing would then be if all such adapters could be supported the same way, and even better though the same library.

From the viewpoint of Home Assistant and similar projects this totally makes sense and I agree, but when you look at it from the companies point of view it becomes very different.

If every dongle has the same interface maybe the same firmware as well there would be china clones within few months. And this is the exact problem in this specific field companies can't compete with Hardware alone, its all about competition on software level.

This is sad but that's how it is, people always just wan't cheapest option, why would you buy a ConBee if a XYZ dongle from china costs a fraction? It is only about the software but nobody wants to pay for software these days. Imagine every dongle uses the same UART API, ... only the cheapest dongle would be sold; you could then use a $ 5 dongle with deCONZ, but how would dresden elektronik earn money then? Charge for deCONZ won't work.

I'd like to think of ConBee/RaspBee as license for deCONZ/WebApp. Opening documentation for UART protocol to allow integration of further systems on top, at the expense of implementing the features which deCONZ does. In that regard Home Assistent would compete with deCONZ/WebApp on software level (well the lighting part) but still dresden elektronik can benefit by selling the hardware.

By the way similar to Z-Wave there is a ZigBee Gateway specification, not sure if anybody implemented it though.

I'm very open to ideas and suggestions but never forget in the end of the day companies need to pay their employees.

Another example, look at Philips they really pushed this whole smart lighting forward, they build very high quality products puts heaps of money in development, now everybody jumps to cheaper lights, be it IKEA or OSRAM which don't even provide the advanced polished firmware, there is no loyalty for hardware and sadly it seems quality too.

Gamester17 commented 6 years ago

@manup Perfectly understandable. However I and many others don't want to use deCONZ gateway software at all, as instead we want to have native support for the hardware in open source software applications such Home Assistant / HASS, OpenHAB, and Domoticz. That is, we only want to use the ConBee/RaspBee USB and serial hardware. Please don't forget about us, you can still make money on us as long as you make good hardware and provide excellent UART documentation for such open source projects ;)

manup commented 6 years ago

@Gamester17 that's totally fine and documentation is in fact one of the next important things which will be improved :) For the UART protocol there will likely be no C library but at least some code snippets to quickly get started.

Decision on where to use the REST/Websocket API or low level UART API is up to the implementor, we only provide the tools and documentation. :)

I do like that systems try to pickup on UART level, hence various manufacturers can be used, which is fine and makes sense, but I also notice that there is very limited spare time from volunteer ZigBee developers to implement all the various device support and tricky parts.

From time to time I check various open source HA systems on how far native ZigBee support has proceeded, but due the lack of volunteers, time or full time developers things progress slowly. In that regard I think at least until low level support is backed by many developers, it would be more efficient to use REST/Websocket API at the beginning, to get wide device support, and implement low level API later/separately — with the drawback of being manufacturer depended for the former one.

donnib commented 6 years ago

From my personal experience it really didn't take much time to add the support for deConz REST API in the HASS (we are talking about few days) when you take away time spent in learning to write python / how to write HASS custom module so i am not 100% convinced it's worth the time to go for low level API. I personally like to let people who know how to handle the low level make that implementation and let others leverage that by using well defined API's.

Gamester17 commented 6 years ago

@donnib You made your point; you think deCONZ and REST API is good enough, but please respect that dicussion about that is off-topic for this request so please refrain from discussing it more here.

FYI; HASS already have native serial UART protocol support for the competing Zigbee USB-adapter called "Linear Quickstick Combo" USB Controller, most often refered for its product article number "HUSBZB-1", (also sold branded as GoControl QuickStick Combo ) made by Nortek Security & Control LLC. So adding native support for a Zigbee USB-adapter using serial UART protocol in HASS is not impossible, not is it a bad idea from an end-users point-of-view, in fact it is again a more user-friendly implementation.

Personally I would just have gone out and bought that one myself too but it is not sold in Europe (and its Z-Wave chip supported European frequency for Z-Wave in Europe, which it does not).

And again, from an end-users point-of-view it is implemented in a more user-friendly mather than than having to use an external gateway software and REST API (which again I ask you not to dicuss further here).

rcloran commented 6 years ago

Without having seen the Conbee serial protocol, and so making a number of assumptions about it, my guess is that the best route for hass would be to use the zigbee layer of bellows, and drop in a conbee serial implementation. Basically, write a class which implements the ControllerApplication interface in bellows.

I have some initial support for XBee S2C working like this, but have not found the time to finish it and publish it.

On currently supported devices by bellows - I know of a handful of other people using devices other than the HuSBZB-1. Any device which uses an EM35x chipset (or any other Ember chipset which implements EZSP), And has the default UART firmware will have a good chance of working.

manup commented 6 years ago

Basically, write a class which implements the ControllerApplication interface in bellows.

Yes that should work and be fairly easy to implement for basic ZDO,APS and ZCL stuff. Not sure about EM35x and XBee capabilities but with ConBee/RaspBee UART API you have further access to:

ZigBee Green Power (ZGP, Rx) needed for Philips Hue Tap
ZCL Interpan communication used for example by ZLL Touchlink commissioning

Gamester17 commented 6 years ago

@rcloran perhaps you accept donation of ConBee and RaspBee hardware from @manup just in case? ;)

lcorsini commented 6 years ago

I'm interested in HASS direct UART support, mainly because I plan to use the raspbee on the same hardware where hass is running, and on a raspberry resources are always limited 😃 @manup is right about loyalty, I bought a raspbee in first place to flash ikea bulbs to use them with hue (only to see how still unpolished are in comparison with hue bulbs, at least the color ones seems good) but also because philips is somewhat limited with their range of products (only one type of remote, until now no E14) for example right now different manufacturers bulbs works, but no third party remotes support, so I've got a pile of useless ikea remotes since they are bundled with color bulbs. So in my case I want to install the raspbee inside my main PI (still an original first PI 1 model B) and use it to drive the remotes, and I could do that with websockets, but in a better world I would like to get rid of all bridges (xiaomi and hue) and use only my hass machine to drive all zigbee hardware and integrate it with other hass automations, in this sense a native library could be better than websockets.