search

esphome / feature-requests

ESPHome Feature Request Tracker
https://esphome.io/
404 stars 26 forks source link

[REQUEST] ESPHome support on Sonoff ZBBridge as a network-attached remote Zigbee Coordinator adapter (or Zigbee Router) for Home Assistant's ZHA integration and Zigbee2MQTT #688

Open Hedda opened 4 years ago

Hedda commented 4 years ago

Describe the problem you have/What new integration you would like

Please consider looking at using Sonoff ZBBridge as a WiFi-connected networked Zigbee serial adapter for projects like ZHA integration component in Home Assistant (via zigpy) and other projects.

Would be cool if we could use ESPHome firmware on this new Sonoff ZBBridge with some kind of serial server software so that could easily be used remotely in your home as dumb Zigbee to WiFi bridge/gateway for zigpy and bellows as those are the libraries which should be required for Home Assistant's ZHA integration component to communicate over serial/UART with the Zigbee radio module inside Sonoff ZBBridge.

Normally you would use a USB-adapter or serial-dongle with a compatible Zigbee radio module together with Home Assistant's ZHA integration component but as this new Sonoff ZBBridge already looks to be a prepackaged ESP8266 and a very powerful Zigbee 3.0 compatible radio it would be nice if could just flash ESPHome firmware onto Sonoff ZBBridge and have ESPHome somehow forwarding the serial port (UART Bus) via TCP/IP to zigpy and bellows radio library just via some easy configuration settings from ZHA UI.

Perhaps as an initial proof-of-concept just using something simple like ser2net (remote serial server) and socat combination to allow ZHA to remote connect to it via serial/UART over the network, like what Zigbee2mqtt does.

Better might be if could use a more secure encrypted connection with username and password or similar solutions. Perhaps you could use the serial bridge code from esp-link for this? It is supposed to have advanced serial bridge code

More backstory and information bellow:

Itead has just launched a new product called "Sonoff ZBBridge" which at less than $17 US-dollar I think is very inexpensive for a complete Zigbee to Wi-Fi Bridge/Gateway packaged box that contains a very powerful Zigbee 3.0 radio module.

According to the teardown on notenoughtech.com it contains a ESP8266 ("ESP8266EX") and a Silicon Labs EFR32MG21 Zigbee 3.0 radio module. EFR32MG21 is one of the latest chip in EFR32 Gecko Series 2 MCU.

Early indications based on that specifications imply that its Zigbee radio might be EmberZNet / Ember / EZSP compatible so could be that if only manage to hack/replace the ESP8266 firmware then could perhaps just forward the serial/UART communication to bellows to control its Zigbee radio.

zigpy devs already do something somewhat similar to connect remotely with ESP8266 based "ZiGate Pack WiFi" which works via the zigpy-zigate library

A similar to Sonoff ZBBridge but non-Zigbee RF to WiFi device called "Sonoff RF Bridge" already existing and that device has been hacked to run third-party open-source ESP8266 firmware from ex. ESPHome, Tasmota, ESPurna, and OpenMQTTGateway. So to that extend hacking instructions for different methods to flash the ESP8266 firmware on a such similar "Sonoff RF Bridge" device already exist.

Please describe your use case for this integration and alternatives you've tried:

ESPEasy (ESP Easy) firmware can be used as a generic Ser2Net gateway (serial port wrapper) so that could be one way, but again it would be nicer with a plug-and-play solution that is made for ZHA

As you might already know, Tasmota also has a somewhat similar concept Zigbee to WiFi bridge concept with its Zigbee2Tasmota (Z2T) subproject but by connecting an ESP8266 to a Texas Instrument CC2530 however they have taken it one step further by acting as an MQTT gateway, but that also means that using the devices in Home Assistant is more limited than it would be if it could instead only forward the serial communication to the matching zigpy radio library for use in the ZHA integration component. Texas Instrument CC2530 is also much less powerful Zigbee radio.

Additional context

Hedda commented 4 years ago

I guess that since the main hurdle will probably be to get a good serial network link working between the Zigbee radio module on serial port/ UART bus to a zigpy radio library then as a development environment today you could instead of waiting for this new Sonoff ZBBridge to be available you could use a popular ESP8266 development board with matching capabilities to the ESP8266EX in Sonoff ZBBridge, like NodeMCU ESP8266 (or Wemos D1 Mini which also has a ESP8266EX).

Then to it connect any already zigpy supported serial Zigbee module (like example CC2530) to its matching zigpy radio library by using ser2net or any other open-source remote serial server / serial port forwarding software that is compatible with ESP8266 / Arduino.

Hedda commented 4 years ago

Please note that #681 and #688 are two sperate requests with different goals.

Hedda commented 4 years ago

This request might be related to #660 = Request for "Stream server component (serial-over-wifi)".

As I understand that ##688 would depend on some kind of stream server for Serial or UART over Wi-Fi.

Though while related I would not say that #660 replaced #688 and their ultimate goals are different.

Hedda commented 4 years ago

To get the equivalent SMD module from Silicon Labs then might want to look at EFR32MG1B Series 1.

Almost any ESP8266 development board (like a NodeMCU or a Wemos D1 Mini) with a Silicon Labs EFR32MG based Zigbee module like the Ebyte E180-ZG120B would just about be about the equivalent to having a hacked Sonoff ZBBridge, so such a setup could be used as a development environment.

The downside to using such setup instead of Sonoff ZBBridge is that the E180-ZG120B module will likely not come preloaded with firmware so have to build EmberZNet PRO Zigbee Stack firmware with coordinator device type config and flash firmware to the E180-ZG120B module first.

See a longer discussion about building Silabs EmberZNet firmware here -> https://github.com/xoseperez/espurna/issues/2224

Ebyte's E180-ZG120A and E180-ZG120B are Zigbee 3.0 module based on EFR32MG1B SoC:

EFR32MG1B Series 1 is a 20dbm powerful Zigbee 3.0 radio capable SoC / chip in Silicon Labs EFR32 family which I understand also used the Ember / EZSP interface so could be made compatible with zipy's bellows radio library if flashed with the right firmware? So I guess main question is which exact firmware to use?

E180-ZG120B (and the previous E180-ZG120A) modules are sold on eBay and Aliexpress at low prices.

Example:

Ebyte also makes an inexpensive development board called "E180-ZG120B-TB" made for testing it.

This development is sold for less than $9 US-dollar on Aliexpress or about twice that on eBay UK:

Note! Ebyte is now making two EFR32MG12 based modules called E180-ZG120A and E180-ZG120B

The downside to using such setup instead of Sonoff ZBBridge is that the E180-ZG120B module will likely not come preloaded with firmware so have to build EmberZNet Zigbee Stack firmware with coordinator device type config and flash firmware to the E180-ZG120B module first.

Sonoff ZBBridge could be the perfect hardware if it was not for the case that it needs to be hacked.

Hedda commented 4 years ago

FYI @SillyDay don't own a E180-ZG120B but has tried to build a firmware for it as per discussion here:

https://github.com/Koenkk/zigbee-herdsman/issues/168

SillyDay GitHub repo for EFR32 firmware:

https://github.com/SillyDay/EFR32

osos commented 4 years ago

I support the thought of adding a generic serial interface for the Sonoff ZBBridge, that would have EspHome enable the Sonoff bridge for not only ZHA, but for all the Zigbee projects.

I suggest the request renamed in accordance hereto, eg.

[REQUEST] ESPHome support on Sonoff ZBBridge to act as a remote Zigbee adapter

Following the implemetation of ser2net should make it highly generic.

Hedda commented 4 years ago

Following the implemetation of ser2net should make it highly generic.

I believe what socat and ser2net as a client-server model is missing to make it much more user-friendly which would be very nice if could be added to an ESPHome implementation is some kind of features/functions for automatic-discovery and Zero-configuration networking (zeroconf) that would allow projects like Home Assistant's ZHA to discover Sonoff ZBBridge with ESPHome Zigbee bridge adapter on the network to make installation and configurations on the client-side as simple as possible.

I support the thought of adding a generic serial interface for the Sonoff ZBBridge, that would have EspHome enable the Sonoff bridge for not only ZHA, but for all the Zigbee projects.

I suggest the request renamed in accordance hereto, eg.

[REQUEST] ESPHome support on Sonoff ZBBridge to act as a remote Zigbee adapter

@osos Good point! Renamed topic.

Hedda commented 4 years ago

FYI, as per related request https://github.com/zigpy/zigpy/issues/405 I just learned that there is an existing ESP8266-based networked-attached Zigbee-adapter called "ZiGate Pack WiFi adapter" which has a new v2.0 firmware that archives this requested function of UART-to-TCP/IP (for Serial-port to WiFi-bridge function) using "ESP-LINK from Jeelab" software, in addition, using ESP-LINK also adds mDNS zeroconf to allow automatic network discovery and configuration:

Description of functions that using ESP-LINK will add to v2.0 firmware for ZiGate Pack WiFi adapter:

https://translate.google.com/translate?sl=fr&tl=en&u=https%3A%2F%2Fzigate.fr%2Fdocumentation%2Fdescription-du-firmware-v2-xx%2F

As I understand it, all ZiGate hardware look to be modular in design and the "ZiGate Pack WiFi adapter" is really just an optional ESP8266 based "dumb" UART-to-TCP/IP (for Serial-port to WiFi-bridge function) for the standard "ZiGate TTL adapter" that allows users to connect to it remotely using TCP/IP over your home LAN (Local Area Network) instead of plugging it directly to your computer via the optional USB adapter.

Specifically, please see the picture of "ZiGate Pack WiFi adapter" https://zigate.fr/produit/zigate-pack-wifi-v1-3/ compared to the picture of "ZiGate TTL USB adapter" https://zigate.fr/produit/zigate-ttl/

Thus "ZiGate Pack WiFi adapter" allows ZHA users to have a networked Zigbee adapter setup like this:

ZHA <–> zigpy/zigpy-zigate <–> TCP/IP over LAN <–> ZiGate-WiFi <–> UART <–> ZiGate Radio

Suggesting this now as I just learned from @doudz there that the new v2.0 version of the ZiGate Pack WiFi adapter firmware contains "ESP-LINK from Jeelab" software which among other things adds mDNS and UART WiFi Bridge support over TCP. As I understand, version v1.x of the firmware for the ZiGate Pack WiFi adapter basically only contained a simple UART/serial-port server forwarding service ( serial server software that just acts as a dumb Zigbee to WiFi bridge for zigpy-zigate), while the new version v2.x also has more advanced features (which does not need to used) it still also contain a simple UART/serial-port server forwarding service, but now mDNS also makes it easier to discover the adapter on your local network.

Now it would be awesome if the ZHA integration component for Home Assistant from an end-user perspective supported just as an easy detection and configuration of network networked-attached Zigbee coordinator adapters, like the Sonoff ZBBridge and the ZiGate Pack WiFi adapter.

I would therefore also suggest using some kind of Zero-configuration networking (zeroconf) method, like for example mDNS, (as mDNS is already in use in Home Assistant Core), to make the ZHA integration component for Home Assistant automatically detect, connect, and configure compatible networked-attached Zigbee coordinator adapters like the "ZiGate Pack WiFi adapter" as that is otherwise already supported by the zigpy-zigate radio library for zigpy.

sedatesoftware commented 4 years ago

Not to take away from this request, however for simpler use cases, this product works directly with Home Assistant. https://www.itead.cc/cc2531-usb-dongle.html

Hedda commented 4 years ago

@sedatesoftware USB adapters are really off-topic here, so only ESP8266 based Zigbee is relevant here. The point here is that is Sonoff ZBBridge is no1; based on ESP8266, no2; contains a very powerful high-performance Zigbee 3.0 capable module, and no3; is sold in a nice package for a low price.

Yes we know Home Assistant's ZHA integration component supports many USB and Serial-based Zigbee adapters from different manufacturers, but as ESPHome is specifically a firmware for ESP8266 (an Arduino compatible MCU with integrated WiFi capabilities) posting the request here implies that only adapters that can be connected to an ESP8266 are really relevant to this discussion.

Technically you could connect CC2530 module/adapter to an ESP8266 (as the Zigbee2Tasmota project does) because it uses a serial interface, but you can not connect a CC2531 module to ESP8266 because it only has a USB interface and lacks a serial interface which you need for a ESP8266 connection.

Regardless, CC253x from Texas Instruments are older low-performance Zigbee modules/adapters that only support the old Zigbee Home Automation 1.2 specification, while the Sonoff ZBBridge contains a very powerful high-performance EFR32MG21 Zigbee module/adapter with a serial interface and it also supports the newer Zigbee 3.0 specifications.

Fact remains that even if you buy the parts in bulk yourself, for the very low price that the Sonoff ZBBridge is sold for you can not put together your own DIY ESP8266 based Zigbee to WiFi bridge with such a high-performance Zigbee 3.0 capable radio module as the EFR32MG21 offers.

Hedda commented 4 years ago

FYI, @s-hadinger has now got one and started review https://github.com/arendst/Tasmota/issues/8583

First signs it is the EFR32 Zigbee module it has inside it has a EmberZNet / Ember based firmware on it.

Hedda commented 4 years ago

There is now also a deep dive follow-up discussion here in parallel about hacking or flashing its EFR32:

Hedda commented 4 years ago

FYI; @s-hadinger is making great progress hacking the Zigbee module in Sonoff ZBBridge for Tasmota:

@mtx512 has now built a custom EZSP NCP firmware for the EFR32 module inside Sonoff ZBBridge:

Hedda commented 4 years ago

I support the thought of adding a generic serial interface for the Sonoff ZBBridge, that would have EspHome enable the Sonoff bridge for not only ZHA, but for all the Zigbee projects.

I suggest the request renamed in accordance hereto, eg.

[REQUEST] ESPHome support on Sonoff ZBBridge to act as a remote Zigbee adapter

Following the implemetation of ser2net should make it highly generic.

Interestingly @s-hadinger now also added a TCP serial bridge to Tasmota firmware for remote access:

Description:

Add a new feature: Serial to TCP bridge (similar to esp-link). This allows to remotely communicate to a MCU through ESP8266. Needs #define USE_TCP_BRIDGE

It adds 2 GPIO types: TCP Tx (208) and TCP Rx (209) and can work with hardware or software serial.

Commands:

  • TCPBaudRate <x>: sets the baudrate for serial (only 8N1 mode), min 1200, max 115200 by 1200 increments.
  • TCPStart <port>: listens to port <port>. This features supports 2 parallel TCP connexions, which can be useful if you need a terminal + a specific protocol (like XMODEM). The 3rd connection will disconnect an previous connection. The number of parallel connections is a compile-time option.
  • TCPStart 0 or TCPStart: shuts down the TCP server and disconnects any existing connection.

For security reasons, the TCP bridge is not started at boot, and requires an explicit TCPStart command (can be automated with Rules).

Hedda commented 4 years ago

FYI, Silicon Labs recently just released a less expensive "EFR32xG22 Wireless Gecko Starter Kit" (SLWSTK6021A) development kit for $99 which only contain one mainboard/development board and a couple of EFR32MG22 modules which are low-power Zigbee Green Power compatible Mightly Gecko Series 2 radios.

https://www.silabs.com/products/development-tools/wireless/efr32xg22-wireless-starter-kit

While the EFR32MG22 low-power radios it comes with are not really usable as Zigbee coordinators, for $99 that kit can be very useful to developers since it jas been confirmed that buying that kit also gives full official access to Silicon Labs Zigbee Stack, Zigbee SDK, and libraries (current, future, and old versions).

Hedda commented 4 years ago

FYI, Silicon Labs recently just released a less expensive "EFR32xG22 Wireless Gecko Starter Kit" (SLWSTK6021A) development kit for $99 which only contain one mainboard/development board and a couple of EFR32MG22 modules which are low-power Zigbee Green Power compatible Mightly Gecko Series 2 radios.

https://www.silabs.com/products/development-tools/wireless/efr32xg22-wireless-starter-kit

While the EFR32MG22 low-power radios it comes with are not really usable as Zigbee coordinators, for $99 that kit can be very useful to developers since it has been confirmed that buying that kit also gives full official access to Silicon Labs Zigbee Stack, Zigbee SDK, and libraries (current, future, and old versions).

Those who are interested in that official $99 SLWSTK6021A started kit might also be interested in buying article SLWRB4180A for around $49 at the same time as that is the matching EFR32MG21 based high-power (+20 dBm) radio board for that dev board kit.

https://www.silabs.com/products/development-tools/wireless/slwrb4180a-efr32-wireless-gecko-radio-board

(Google search shows that SLWRB4180A can be bought from many of resellers globally who also stock the SLWSTK6021A kit).

Hedda commented 4 years ago

FYI, @s-hadinger has after the last pull request posted saying that he now feels Tasmota ready for testing of EZSP v8 interface:

https://github.com/arendst/Tasmota/issues/8583

https://github.com/arendst/Tasmota/pull/8839

Not for everyone yet since still need to flash its EFR32 chip with some type of J-Link Debug Probe (might always be required?).

girlpunk commented 3 years ago

I'd be rather interested in seeing this working. Did manage to find this guide for setting things up with Tasmota, which shows it's possible to flash the Zigbee chipset without an external programmer

https://www.digiblur.com/2020/07/how-to-use-sonoff-zigbee-bridge-with.html

Hedda commented 3 years ago

Yes it is now supported by "tasmota-zbbridge" firmware and there several different ways of flashing the EFR32 firmware

For alternative methods see https://notenoughtech.com/home-automation/flashing-tasmota-on-sonoff-zigbee-bridge/

Hedda commented 3 years ago

Any further thoughts on Sonoff ZBBridge support now that this is fully supported and already very stable feature in Tasmota?

https://github.com/arendst/Tasmota/issues/8583

That Sonoff ZBBridge is inexpensive but contain great Zigbee hardware with potential for different types of Zigbee controller uses, as inside it has an ESP8266 WiFi SoC/MCU connected via UART to a very powerful Silicon Labs EFR32MG2 (Mighty Gecko Series 2) Zigbee SoC/MCU on the same board.

https://sonoff.tech/product/smart-home-security/zbbridge

Tasmota allows using its ser2net like TCP serial-server inside ESP8266 firmware to take direct control of the Zigbee chip that it is connected to via UART/serial and then connect to Zigbee devices without any further involvement on Tasmota (which via its TCP serial-server only will only act as a dumb pass-through WiFi-to-serial bridge without any knowledge of what the data traffic that is going back and forth between the remote Zigbee chip AND different home automation software like example Home Assistant).

You can even make your own DIY WiFi bridge with an ESP8266 board + an ICC-A-1 module torn out of an IKEA Tradfri device:

https://github.com/MattWestb/IKEA-TRADFRI-ICC-A-1-Module

Example with ZHA integration component in Home Assistant which add native support for Zigbee adapters, including the ZBBridge:

https://www.digiblur.com/2020/07/how-to-use-sonoff-zigbee-bridge-with.html

Sure, it could have been even greater with ESP32 but still geat ESP8266 hardware with Silabs EFR32 MG2 Zigbee coordinator!

Hedda commented 3 years ago

FYI, tip now is to instead use Tube's Zigbee Gateways open-source hardware by @tube0013 which runs ESPHome FW by default.

Based on WT32-ETH01 (Wireless-Tag) ESP32 board which has wired Ethernet that makes for a much more stable serial connection.

He designed two variants; one has a Silicon Labs EFR32 Series 2 module and one that has a Texas Instruments CC2652P module.

https://github.com/tube0013/tube_gateways

https://github.com/zigpy/zigpy/discussions/584

https://www.tubeszb.com/shop/coordinators/2

image

image

thegroove commented 3 years ago

I made a thing:

https://github.com/thegroove/esphome-zbbridge

Note that I don't necessarily recommend using this, even though I made it. @Hedda makes valid comments, and in fact, my gateway from @tube0013 is in the mail, but while waiting for it to arrive, I figured I'd give my Sonoff ZB Bridge a bit of loving while it "bridges" the gap.

This is quite literally @Oxan 's code refactored to a "real" custom component, making it a bit easier to use. Like the original code, it's not necessarily bound to the Sonoff ZB Bridge, use it with any setup you like. I plan to use it on my future gateway mostly as-is, with only the .yaml needing adjustment.

Hedda commented 3 years ago

I made a thing: https://github.com/thegroove/esphome-zbbridge

@thegroove Do you advise to first flash the ESP8266 on it with Tasmota firmware using Tasmotizer as per Digiblur blog or? See:

https://www.digiblur.com/2020/07/how-to-use-sonoff-zigbee-bridge-with.html

Noticed that README.md did not include any instruction on flashing ESPHome to Sonoff ZBBridge or mention any prerequisites:

https://github.com/thegroove/esphome-zbbridge/blob/main/README.md

Suggest that README.md could also need a guide on "how-to" hack/flash ESPHome to Sonoff ZBBridge ESP8266 chip as well.

PS: Did not know if Zeroconf auto-discovery was supported so create a request under issues -> https://github.com/thegroove/esphome-zbbridge/issues/1

genebean commented 3 years ago

Here’s to hoping this comes to fruition. I just ordered the Sonoff since it’s actually in stock and plant to flash it as talked about here but would much prefer ESPHome on it.

Hedda commented 3 years ago

Tip! Check out these discussions for DIY Sonoff ZBBridge (ESP8266/ESP8285 or ESP32 board with a flashed Zigbee SMD module):

https://github.com/zigpy/zigpy/discussions/743

and

https://github.com/zigpy/zigpy/discussions/584

Which in turn lead both in intersects with Tube's Zigbee Gateways project:

https://github.com/tube0013/tube_gateways

PS: Also, see this discussion regarding the flow control parameter in ESPHome-> https://github.com/esphome/feature-requests/issues/1126

Hedda commented 3 years ago

By the way, check out related Zeroconf in https://github.com/thegroove/esphome-zeroconf and thegroove/esphome-zbbridge#1

Hedda commented 3 years ago

FYI, @oxan has now made his UART/Serial stream server available as a new external component for ESPHome v1.18+ on GitHub:

https://github.com/oxan/esphome-stream-server

external_components:
  - source: github://oxan/esphome-stream-server

stream_server:

ESPHome version 1.18 and later has added a new feature to support such external component which can now easily be imported:

https://esphome.io/components/external_components.html

Hedda commented 2 years ago

FYI, ZB-GW03 eWeLink Ethernet Zigbee Gateway is based on an ESP32 and Silabs EFR32 Zigbee and has been hacked with Tasmota.

https://github.com/arendst/Tasmota/discussions/12764

ZB-GW03 has been hacked with Tasmota32 and look to support the same Tasmota and Silabs firmware as ITead Sonoff ZBBridge, however since the ZB-GW03 has wired Ethernet port instead of WiFi it should be a better option compared to ITead Sonoff ZBBridge.

https://templates.blakadder.com/ewelink_ZB-GW03

image

image

image

ZB-GW03 v1.0 and ZB-GW03-V1.2 is apparently sold rebranded under many names, including SmartWise and EACHEN brands:

https://ewelink.eachen.cc/product/eachen-ewelink-zigbee-bridge-pro-zbbridge-pro/

https://ewelinkcommunity.net/device-lists/zigbee/eachen-zb-gw03/

https://www.okosabbotthon.hu/smartwise-zigbee-bridge-pro

https://www.amazon.de/APP-Fernbedienung-Sprachsteuerung-Funktioniert-Verbindung-Smart-Produkten/dp/B094JKVLNR/

image

image

Looks like is it is based on the same "SM-011 V1.0" module by CoolKit as ITead uses in their Sonoff ZBBridge Zigbee Bridge:

https://github.com/zigpy/zigpy/discussions/586

https://github.com/xsp1989/zigbeeFirmware/

https://www.coolkit.cn/product/sm-011/

https://github.com/CoolKit-Technologies/DevDocs/tree/master/Zigbee

https://github.com/CoolKit-Technologies/DevDocs/blob/master/Zigbee/SM-011%E5%BA%94%E7%94%A8%E6%8C%87%E5%AF%BC%E4%B9%A6.md

https://translate.google.com/translate?hl=&sl=auto&tl=en&u=https%3A%2F%2Fgithub.com%2FCoolKit-Technologies%2FDevDocs%2Fblob%2Fmaster%2FZigbee%2FSM-011%E5%BA%94%E7%94%A8%E6%8C%87%E5%AF%BC%E4%B9%A6.md

Hedda commented 2 years ago

FYI, there's now a detailed step-by-step instruction about flashing Tasmota on ZB-GW03 eWeLink Ethernet Zigbee Gateway (wired network bridge) for Zigbee2Tasmota (also using with Home Assistant's ZHA):

https://thehelpfulidiot.com/a-wired-sonoff-zigbee-alternative

Again, basic info for experienced Tasmota and Z2T users here :

https://templates.blakadder.com/ewelink_ZB-GW03

and again there is also a discussion about it in Home Assistant community forum here:

https://community.home-assistant.io/t/zb-gw03-ewelink-ethernet-zigbee-gateway-now-hacked-with-tasmota-zbbridge-so-can-be-used-via-mqtt-or-as-a-remote-zigbee-adapter-with-home-assistant-zha/341223

UPDATE! The ESP32 inside ZB-GW03 has now also been hacked with ESPHome for use as a remote Zigbee Coordinator adapter:

https://github.com/syssi/esphome-zb-gw03

https://github.com/syssi/esphome-zb-gw03/blob/main/docs/flashing.md

urechi commented 2 years ago

Hi all,

Now that we have a dumb but working ESPHome ZBBridge from @thegroove and also WireGuard Support for ESP32 from @CarlosGS , @mrkeuz and @davey , could we combine the two and get a secure remote Zigbee hub such as sonoff zigbee bridge or ZB-GW03? As a use case, I could have a secondary location, such as a holiday property (I wish I had one) and I should be able to securely connect from ZHA or Z2M to the ZBBridge with the socket://{wireguard_ip}:{port} over the internet through the VPN tunnel. A single HA instance (with the Wireguard addon as the VPN server) could now control multiple zigbee network locations and all this possible without a site-to-site VPN implementation.

A ZBBridge with Serial-Over-Wireguard, to manage an entire zigbee network remotely. This sounds interesting!

Hedda commented 2 years ago

As a use case, I could have a secondary location, such as a holiday property (I wish I had one) and I should be able to securely connect from ZHA or Z2M to the ZBBridge with the socket://{wireguard_ip}:{port} over the internet through the VPN tunnel. A single HA instance (with the Wireguard addon as the VPN server) could now control multiple zigbee network locations and all this possible without a site-to-site VPN implementation.

FYI, ZHA (or Zigbee2MQTT) serial connection/communication with a remote Zigbee serial adapter/module is only a good idea if used locally over a stable LAN (Local Area Network) like wired Ethernet at the same location. It is not even recommended over a Wi-Fi connection since dropped connections and high latency cause issues with remote serial communication in the serial protocols they use.

Note that there are warning in both ZHA and Zigbee2MQTT documentation about using a serial connection to Zigbee modules over Wi-Fi, including information that the reason for not doing so is poor robustness in the serial protocol that the Zigbee stacks use.

https://www.home-assistant.io/integrations/zha#warning-about-wi-fi-based-zigbee-to-serial-bridgesgateways

https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_adapter.html#_1-install-ser2net

ZHA (or Zigbee2MQTT/Z2M) serial connection/communication with a remote ESPHome (or Tasmota) based Zigbee network adapter over the internet via a VPN tunnel is as such not a good example use case scenario or a great idea in the real world.

A better real-world use case scenario for a remote Zigbee adapter via ESPHome (or Tasmota) is when you have Home Assistant installed in a virtual machine on a server or NAS in your basement or corner and want to have the Zigbee coordinator installed in the centre of your house on the main floor.

ZHA (and Zigbee2MQTT/Z2M) really needs a stable/robust connection to the Zigbee serial module because the serial protocols that Zigbee stacks use for communication are by themselves are not made to handle loads of communications errors due to bad connections.

If you do have a secondary location, such as a holiday property (also I wish I had one), and want to use ZHA then you really need to set up an additional host with a Home Assistant instance on that location so can connect the Zigbee serial module locally directly or over the local LAN.

PS: Also note that both ZHA and Zigbee2MQTT can currently only connect to one Zigbee Coordinator for each instance and unless you implement hack you can currently also not install more than one instance of ZHA and Zigbee2MQTT in Home Assistant. Thus if you want to use two Zigbee Coordinators with ZHA you need to use more than one instance of Home Assistant. Zigbee2MQTT is a little easier since stand-alone from Home Assistant so can just install several instances of Zigbee2MQTT and then connect them all via MQTT instead.

Hedda commented 2 years ago

FYI, ZB-GW03 eWeLink Ethernet Zigbee Gateway is based on an ESP32 and Silabs EFR32 Zigbee and has been hacked with Tasmota.

FYI; @syssi posted in the Home Assistant forum saying he made a ESPHome configuration for the ZB-GW03 and tested with ZHA:

https://community.home-assistant.io/t/zb-gw03-ewelink-ethernet-zigbee-gateway-now-hacked-with-tasmota-zbbridge-so-can-be-used-via-mqtt-or-as-a-remote-zigbee-adapter-with-home-assistant-zha/341223/26

https://github.com/syssi/esphome-zb-gw03

https://github.com/syssi/esphome-zb-gw03/blob/main/docs/flashing.md

https://github.com/syssi/esphome-zb-gw03/blob/main/docs/router.md

urechi commented 2 years ago

Thanks @Hedda, your detailed reply is very much appreciated. So then in order to have a solid remote ZB-GW03 zigbee bridge outside of the home automation hub LAN, what would be the best option at the moment? An internet exposed MQTT broker and Zigbee2Tasmota(2MQTT) flashed to the bridge? The downside to this would be that MQTT auto-discovery with HA doesn't work for zigbee routers and end devices, only the zigbee coordinator is auto-discovered by HA. Having to add those devices manually might be exhausting if you have a large zigbee network.

Is there an equivalent in ESPHome, something like Zigbee2ESPHome2MQTT?

Hedda commented 2 years ago

Is there an equivalent in ESPHome, something like Zigbee2ESPHome2MQTT?

No, there is not at this time. Closest is to setup serial stream proxy server in ESPHome for a Zigbee module to act as a "ser2net" serial tunnel over TCP the so that either the ZHA integration or Zigbee2MQTT can use it as a remote Zigbee Coordinator adapter via "socat", but again that is not recommended between sites as a stable synchronous connection is then needed.

Also, Zigbee2MQTT is not as user-friendly or well-integrated, and generally ESPHome strives for a more user-friendly experience.

So then in order to have a solid remote ZB-GW03 zigbee bridge outside of the home automation hub LAN, what would be the best option at the moment? An internet exposed MQTT broker and Zigbee2Tasmota(2MQTT) flashed to the bridge? The downside to this would be that MQTT auto-discovery with HA doesn't work for zigbee routers and end devices, only the zigbee coordinator is auto-discovered by HA. Having to add those devices manually might be exhausting if you have a large zigbee network.

Not tested it but I think that the best solution if stability is the main requirement when having two completely separate locations/sites and specifically want to use ZB-GW03 with ESPHome and ZHA, it would probably be to try to setup a second Home Assistant instance on the secondary location/site and run ZHA in it too, so have separate instances of Home Assistant with ZHA on both locations/sites. Configure VPN for Home Assistant between the two locations/sites, then aim to setup some kind if entity forwarder on the secondary site (probably using MQTT) so that that whole secondary Home Assistant can act as a remote adapter.

Two other off-topic solutions that do not involve ESPHome:

VPN to either Zigbee2Tasmota (Tasmota Z2T firmware on ZB-GW03) if using ZB-GW03 on the second site.

If ZB-GW03 is not a requirement then just use Zigbee2MQTT with Texas Instrument based Zigbee adapter on the second site and setup VPN to the primary site.

Disclaimer: Personally I do not have two sites but if I did then I would want to run a full instance of Home Assistant on both sites. Again I have not tested it however assume that there got to be some way to connect the second instance to the first instance.

Hedda commented 2 years ago

Not tested it but I think that the best solution if stability is the main requirement when having two completely separate locations/sites and specifically want to use ZB-GW03 with ESPHome and ZHA, it would probably be to try to setup a second Home Assistant instance on the secondary location/site and run ZHA in it too, so have separate instances of Home Assistant with ZHA on both locations/sites. Configure VPN for Home Assistant between the two locations/sites, then aim to setup some kind if entity forwarder on the secondary site (probably using MQTT) so that that whole secondary Home Assistant can act as a remote adapter.

...

Disclaimer: Personally I do not have two sites but if I did then I would want to run a full instance of Home Assistant on both sites. Again I have not tested it however assume that there got to be some way to connect the second instance to the first instance.

FYI, see these architecture RFCs/ideas regarding linking two or more Home Assistant instances -> https://github.com/home-assistant/architecture/issues/246

Hedda commented 2 years ago

FYI; @syssi posted in the Home Assistant forum saying he made a ESPHome configuration for the ZB-GW03 and tested with ZHA:

https://community.home-assistant.io/t/zb-gw03-ewelink-ethernet-zigbee-gateway-now-hacked-with-tasmota-zbbridge-so-can-be-used-via-mqtt-or-as-a-remote-zigbee-adapter-with-home-assistant-zha/341223/26

https://github.com/syssi/esphome-zb-gw03

FYI, syssi now added step-by-step instructions for how-to flash ZB-GW03 with ESPHome firmware (including backup and restore):

https://github.com/syssi/esphome-zb-gw03/blob/main/docs/flashing.md

Hedda commented 2 years ago

FYI, you can now also flash "Zigbee Router" firmware for the ITead Sonoff ZBBridge to convert it to a Zigbee signal-repeater/range-extender instead of as a remote Zigbee Coordinator/Controller if it runs Tasmota, check out this blog post article by digiblur:

https://www.digiblur.com/2022/02/convert-sonoff-zigbee-bridge-to-router-repeater.html

xsp1989 has uploaded a signed "Zigbee Router" firmware for ITead's ESP8266/ESP8285 and Silicon Labs EFR32MG21 based Sonoff branded Zigbee bridge/gateway/proxy-server appliance that could maybe make it possible to make it into a network-attached ESPHome/Tasmota controlled Zigbee Router device?

https://drive.google.com/file/d/1H-M5CTh_XGGVl2te4vHLQ5SfGPHL6yCS/view?usp=sharing

This is the signed routing firmware used by ZBB, the usage method is the same as the unsigned firmware.

https://github.com/xsp1989/zigbeeFirmware/blob/master/firmware/Zigbee3.0_Dongle/RouterForDongle/README.md

Do you think ESPHome can send basic serial commands to this Zigbee Router firmware to initiate paring/joining and restart/reset?

The benefits of a Zigbee Router device is that it can be remotely reset/restarted or un-paired and re-paired if it hangs or you want to join a different Zigbee network when switching to a different Zigbee implementation. So that ESPHome is basically only used to initiate the virtual join/pair button and restart/reset the device similar to a remote power-cycle it hangs.

The idea around this concept is also discussed for ITead's Sonoff ZBBridge hacked with Tasmota/ESPHome here:

https://github.com/arendst/Tasmota/discussions/11536

https://github.com/thegroove/esphome-zbbridge/issues/3

https://github.com/syssi/esphome-zb-gw03/issues/12

https://github.com/xsp1989/zigbeeFirmware/issues/2

https://github.com/xsp1989/zigbeeFirmware/issues/16

PS: I have not tried this myself as I got rid of my ITead Sonoff ZBBridge because being WiFi-based it did not work any good as a remote Zigbee Coordinator adapter but might reconsider buying one (or a ZB-GW03 with Ethernet port) if it can work as a Zigbee Router device that can be paired/joined and restarted/reset remotely via Tasmota and/or ESPHome.

Hedda commented 2 years ago

FYI, @xsp1989 is now sharing a signed Zigbee Router firmware for ITead Sonoff ZBBridge and ZB-GW03 to his google drive here:

https://drive.google.com/file/d/1H-M5CTh_XGGVl2te4vHLQ5SfGPHL6yCS/view?usp=sharing

This is the signed routing firmware used by ZBB, the usage method is the same as the unsigned firmware.

https://github.com/xsp1989/zigbeeFirmware/blob/master/firmware/Zigbee3.0_Dongle/RouterForDongle/README.md

Firmware for their "SM-011 V1.0" EFR32MG21 Zigbee module by CoolKit has now been tested on Sonoff ZBBridge and ZB-GW03:

https://github.com/syssi/esphome-zb-gw03/blob/main/docs/router.md

https://www.digiblur.com/2022/02/convert-sonoff-zigbee-bridge-to-router-repeater.html

This inofficial Zigbee Router firmware allow you to re-purpose these as Zigbee routers (a.k.a. signal-repeaters/range-extenders).

The ideas around also re-using these cheap Zigbee gateway/bridges as Zigbee Router devices is being discussed more here:

https://github.com/arendst/Tasmota/discussions/11536

https://github.com/xsp1989/zigbeeFirmware/issues/16

https://github.com/xsp1989/zigbeeFirmware/issues/2

PS: ITead Sonoff ZBBridge is based ESP8266/ESP8285 with only WiFi, while the ZB-GW03 eWeLink Ethernet Zigbee Gateway ZB-GW03 v1.0 / V1.2 is based on is based on ESP32 and have an RJ-45 Ethernet port (it is sold rebranded under many names, including; EACHEN and SmartWise brands):

https://community.home-assistant.io/t/zb-gw03-ewelink-ethernet-zigbee-gateway-now-hacked-with-tasmota-zbbridge-so-can-be-used-via-mqtt-or-as-a-remote-zigbee-adapter-with-home-assistant-zha-and-zigbee2mqtt/341223/

Hedda commented 2 years ago

FYI, ITead has apparently now started selling "SONOFF Zigbee Bridge Pro" (model "ZBBridge-P" or "ZB Bridge-P") so I assume are the same as the previously "Sonoff ZBBridge-P" leaked on FCC thus should be based on CC2652P and ESP32-D0WD-V3:

https://itead.cc/product/sonoff-zigbee-bridge-pro/

https://fccid.io/2APN5ZBBRIDGEP

image

Its packaging is the same as and their previous Zigbee gateway/bridge and the only listed difference mentioned on its website is that it can handle 128 instead of 32 devices, contains a “buzzer” which webpage mention that the buzzer can be used as an alarm (siren?) or as a doorbell chime (if use Zigbee wireless buttons as doorbells), and a “built-in high-precision local RTC” (Real-Time Clock) for gateway time synchronization of local scene schedules.

image

ESP32 has replaced the ESP8266/ESP8285, and it also looks like added an onboard clock-battery too, presumably for the RTC:

https://fccid.io/2APN5ZBBRIDGEP/Internal-Photos/Internal-Photos-5613131

image

image

image

image

image

image

image

image

Sadly it looks like even this new variant of Sonoff ZBBridge still uses Wi-Fi instead of a wired Ethernet network like the newer ZB-GW03:

https://community.home-assistant.io/t/zb-gw03-ewelink-ethernet-zigbee-gateway-now-hacked-with-tasmota-zbbridge-so-can-be-used-via-mqtt-or-as-a-remote-zigbee-adapter-with-home-assistant-zha-and-zigbee2mqtt/341223

Ethernet is preferred by DIY users who flash Tasmota or ESPHome in order to use it as a remote Zigbee adapter with ZHA or Zigbee2MQTT because tunnelling serial communication over a serial stream server connection and the serial APIs of the Zigbee stacks are not robust/stable over WiFi as assume a direct connection, which is why both ZHA and Zigbee warns against using WiFi-based networked-attached Serial-to-IP Zigbee bridges/gateways/adapters:

https://www.home-assistant.io/integrations/zha#warning-about-wi-fi-based-zigbee-to-serial-bridgesgateways

https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_adapter.html

Hedda commented 2 years ago

UPDATE! ZBBridge-P (a.k.a. "ZB Bridge-P") has been released as "SONOFF Zigbee Bridge Pro" as is now available for purchase:

https://itead.cc/product/sonoff-zigbee-bridge-pro/

Both NotEnoughTech and digiblur have received theirs and while neither of them has tried to flash/hack it yet they posted some early thoughts on the original eWeLink hub firmware and hardware board, confirming that the first retail version ITead's SONOFF Zigbee Bridge Pro with PCB board labelled "ZBBridge-P V1.1 2021.11.12" is indeed based on Espressiff ESP32-DOWD-V3 (that ESP32 variant by the way also has integrated support for Bluetooth 4.2 including BLE), Texas Instruments CC2652P Zigbee module labelled "SM-031 V1.1 2021.07.28" (so probably manufactured by CoolKit Technologies like the Zigbee module for previous Sonoff bridge), with a cell-battery for the RTC (Real-Time-Clock), a built-in buzzer for chime or alarm-siren, and also an Espressiff branded PSRAM (ESP-PSRAM16H) with 2MB/16Mbit capacity.

digiblurDIY also mention that he managed to backup original firmware from ESP32 without issue so suspect that it is not locked:

https://www.youtube.com/watch?v=P5op57y5i6k?t=920&ab_channel=digiblurDIY

https://notenoughtech.com/home-automation/sonoff-zigbee-bridge-pro/

https://www.youtube.com/watch?v=fsIjFxCPJLA&ab_channel=NotEnoughTECH

Internal pictures posted from FCC show using an unannounced Texas Intruments CC2652P (CC2652P1) MCU chip based “SM-031 v1.1” radio module which must be made by CoolKit (company that also makes their EFR32MG21 based “SM-011 V1.0” module).

The new module based on Texas Instruments CC2652 probably come preloaded with Zigbee Coordinator firmware from Koenkk:

https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_3.x.0/bin

https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator

https://github.com/Koenkk/Z-Stack-firmware/tree/master/coordinator/Z-Stack_3.x.0

Again the main downside the this new and the old Sonoff ZBBridge is that they only have WiFi and not a wired Ethernet LAN port.

Hedda commented 2 years ago

UPDATE! ZBBridge-P (a.k.a. "ZB Bridge-P") has been released as "SONOFF Zigbee Bridge Pro" as is now available for purchase:

https://itead.cc/product/sonoff-zigbee-bridge-pro/

Both @NotEnoughTech and digiblur @have received theirs and while neither of them has tried to flash/hack it yet they posted some early thoughts on the original eWeLink hub firmware and hardware board, confirming that the first retail version ITead's SONOFF Zigbee Bridge Pro with PCB board labelled "ZBBridge-P V1.1 2021.11.12" is indeed based on Espressiff ESP32-DOWD-V3 (that ESP32 variant by the way also has integrated support for Bluetooth 4.2 including BLE), Texas Instruments CC2652P Zigbee module labelled "SM-031 V1.1 2021.07.28" (so probably manufactured by CoolKit Technologies like the Zigbee module for previous Sonoff bridge), with a cell-battery for the RTC (Real-Time-Clock), a built-in buzzer for chime or alarm-siren, and also an Espressiff branded PSRAM (ESP-PSRAM16H) with 2MB/16Mbit capacity.

FYI @mercenaruss first posted in https://community.home-assistant.io/t/itead-sonoff-zigbee-bridge-pro-zbbridge-p-new-cc2652p-and-esp32-based-zigbee-gateway-to-wifi-bridge/378924/ that he plans to look at a firmware for this device and then posted in https://github.com/arendst/Tasmota/discussions/14419 saying that he is making progress on an initial ESPHome proof-of-concept build for this new Sonoff ZBBridge-P bridge from ITead.

Before attemting this yourself however note that needs to be made clear that users should for now not flash unmodified CC2652P images from Koenkk which the community normally uses as the CC2652P Zigbee module in this ZBBridge-P does not use the standard pins for BSL so flashing an unmodified CC2652 firmware to the Zigbee module will mean that will not be able to flash it again via BSL and will instead need to use a cJTAG probe adapter.

Originally posted by @mercenaruss in https://github.com/arendst/Tasmota/discussions/14419#discussioncomment-2856425

I build a firmware on ESPHome,to be used with ZBBridge-P and Z2M. I managed to flash using cc2538-bsl & ZigStar MultiTool, unchecked AutoBSL. Trigered from webinterface Firmware Update and hit START in ZigStar MultiTool.

ZigStar-ZBBridge Screenshot 2022-05-31 145835

  1. Flash firmware using ESPHOME Flasher
  2. Flash Koenk firmware using ZigStar Multitool
  3. Connect to captive portal and setup your Wifi
  4. Identify ZBBridge IP and setup in Z2M, port 6638 ( port: tcp://ip:6638 )
  5. Enjoy zs-zbridge-p.zip

Working like charm in Z2M. P.S. Will add Reset button capability,buzzer and led indication of status. Also will added to ZigStar GW firmware

Community can use modified firmware what is available on tasmota repository. At the moment firmware is working properly without any issues.

I managed to identify all GPIO’s used in the board.

https://www.youtube.com/shorts/s920pAZsC7E?&ab_channel=RaduG.

Hedda commented 2 years ago

FYI @mercenaruss first posted in https://community.home-assistant.io/t/itead-sonoff-zigbee-bridge-pro-zbbridge-p-new-cc2652p-and-esp32-based-zigbee-gateway-to-wifi-bridge/378924/ that he plan to look at a firmware for this device and then posted in arendst/Tasmota#14419 saying that he is making progress on an initial ESPHome proof-of-concept build for this new Sonoff ZBBridge-P bridge from ITead.

Before attempting this yourself however note that needs to be made clear that users should for now not flash unmodified CC2652P images from Koenkk which the community normally uses as the CC2652P Zigbee module in this ZBBridge-P does not use the standard pins for BSL so flashing an unmodified CC2652 firmware to the Zigbee module will mean that will not be able to flash it again via BSL and will instead need to use a cJTAG probe adapter. Community can use modified firmware what is available on tasmota repository. At the moment firmware is working properly without any issues.

...

I managed to identify all GPIO’s used in the board.

https://www.youtube.com/shorts/s920pAZsC7E?&ab_channel=RaduG.

For reference, on Sonoff ZBBridge-P the CC2652 firmware from arendst’s Tasmota repository on GitHub here should be flashed:

https://github.com/arendst/Tasmota/tree/development/tools/fw_SonoffZigbeeBridgePro_cc2652

Hedda commented 10 months ago

FYI, noticed that grobasoz added some Zigbee RCP and CLI firmware images for ESP32-C6:

No idea how or if they work, (grobasoz is known in Home Assisistant's ZHA community for providing unofficial Zigbee NCP firmware images for a wide range of Silicon Labs based Zigbee Coordinator adapters)

PS: DamKast recently announced "zigpy-zboss" as an (unofficial and experimental) ZBOSS radio library for zigpy as that could in theory make zigpy compatible with ESP32-C6/ESP32-H6 if their firmware use the standard ZBOSS API/CLI which if it is the case should make it usable as a remote network-attatched Zigbee Coordinator for Home Assistant's ZHA integration if configure via serial server pass-through as proxy using socat (similar to ser2net) -> https://github.com/zigpy/zigpy/issues/1052 (More zigpy-zboss discussion related to zigpy here -> https://github.com/zigpy/zigpy/issues/394)