itead / Sonoff_Zigbee_Dongle_Firmware

204 stars 22 forks source link

[REQUEST] Utilize new common "Silicon Labs firmware builder" automated build system from Home Assistant / Nabu Casa developers to create reproducible builds of EmberZNet Zigbee NCP and OpenThread (Thread) firmware images #17

Open Hedda opened 1 year ago

Hedda commented 1 year ago

Please consider making use of the "Silicon Labs firmware builder" which is a new open-source firmware build system that Home Assistant founders have now among other things announced as a common script tool set for building firmware images for Silabs based hardware.

https://www.home-assistant.io/blog/2023/02/08/state-of-matter-and-thread/#building-firmware-for-silicon-labs-chips

(Note that this is only just one section from https://www.home-assistant.io/blog/2023/02/08/state-of-matter-and-thread/ )

In summary, they have open-sourced and released the ""Silicon Labs firmware builder" project which is a collection of automated build scripts to help make it easier to create reproducible firmware builds for Zigbee and Thread based SoCs from Silicon Labs.

Please consider making use of this firmware build system and if possible also contribute any improvements to upstream project:

https://github.com/NabuCasa/silabs-firmware-builder

https://github.com/NabuCasa/silabs-firmware-builder/blob/main/README.md

As I understand it, right now it allows you to build either Zigbee EmberZNet NCP (Zigbee Coordinator on Silicon Labs Network Co-Processor) firmware images, OpenThread RCP (Radio Co-Processor), or RCP Multi-PAN firmware images for OTBR (multi-protocol Zigbee EmberZNet and OpenThead Border Router using Radio Co-Processor) profiled for their own Home Assistant SkyConnect USB stick/dongle and their EFR32MG21-based radio module integrated into the Home Assistant Yellow appliance, however, the idea is that you can build profiles for other Silicon Labs hardware adapters/dongles/sticks/modules as well. It could of course also be extended to not only support different Silicon Labs based hardware adapters/dongles/sticks/modules but also different type of firmware images, (like maybe OpenThead Border Router on NCP).

(Examples of firmware image builds that it can produce can be found here -> https://github.com/NabuCasa/silabs-firmware ).

Note that was just one of their announcements in that blog post; in related news, they also released "SL Web Tools" which enable flashing firmware to Silabs-based adapters directly from a web page as well as a "SiLabs multi-flasher" as a Python script flashing tool:

https://www.home-assistant.io/blog/2023/02/08/state-of-matter-and-thread/#silabs-multi-flasher--sl-web-tools

(Again note that this is only just one section from https://www.home-assistant.io/blog/2023/02/08/state-of-matter-and-thread/ )

https://github.com/NabuCasa/universal-silabs-flasher

https://github.com/NabuCasa/sl-web-tools

https://www.youtube.com/watch?v=-88K23e8XYw&t=2s&ab_channel=HomeAssistant

The "SL Web Tools" project is available as open source. It allows users to manage the firmware on Zigbee/Thread sticks using SiLabs chips directly from their browser, with no terminals or compilation necessary. This allows any creator to offer easy firmware updates to their users and make experimental updates available to test bug fixes quickly."

To see it in action, SL Web Tools for flashing firmware has been added to the Home Assistant SkyConnect documentation.

https://skyconnect.home-assistant.io

SL Web Tools is powered by another new open-source Python package “silabs-universal-flasher” → GitHub - NabuCasa/universal-silabs-flasher: Flashes Silicon Labs radios running EmberZNet or CPC multi-pan firmware

For reference; the firmware build system for Silicon Labs was made by Home Assistant developer/engineer and Nabu Casa employee Stefan Agner (a.k.a. agners, who also developed the EFR32MG21-based Home Assistant SkyConnect USB stick/dongle as well as the EFR32MG21-based radio module integrated into the Home Assistant Yellow appliance for Rasberry Pi 4 Compute Module). And the "universal-silabs-flasher" firmware tool was made by puddly who is also one of the lead developers of zigpy and the radio libraries for zigpy that is used in Home Assistant's ZHA integration (among other projects).

PS: They made updates to their documentation as relevant to Silicon Labs based adapters with correct firmware for Zigbee or Thread:

@Daniel-zhan-itead

Hedda commented 1 year ago

FYI, ksjh has begun some work on such automated firmware building using Github Actions and started a related discussion on it:

https://github.com/ksjh/silabs-firmware-builder

https://github.com/ksjh/silabs-firmware-builder/discussions/2

Note that most of the automation magic with silabs-firmware-builder looks to happen in Github Actions workflows:

https://github.com/features/actions

https://docs.github.com/en/actions

Hedda commented 1 year ago

FYI, ksjh has begun some work on such automated firmware building using Github Actions and started a related discussion on it:

https://github.com/ksjh/silabs-firmware-builder

ksjh/silabs-firmware-builder#2

Note that most of the automation magic with silabs-firmware-builder looks to happen in Github Actions workflows:

https://github.com/features/actions

https://docs.github.com/en/actions

FYI, ksjh (and darkxst) now made EmberZNet NCP (EZSP), OpenThread RCP, and RCP MultiPAN patches for the Sonoff ZBDongle-E:

https://github.com/ksjh/silabs-firmware-builder

PS: In addition, if you need more proof-of-concept examples and references, two additional projects have now also forked https://github.com/NabuCasa/silabs-firmware-builder and patched their forks for other specific hardware. Check out:

https://github.com/tube0013/silabs-firmware-builder

and

https://github.com/giannello/silabs-firmware-builder

Hedda commented 5 months ago

FYI, it looks like puddly has in the past few weeks rewritten and reorganized much of Nabu Casa’s upstream Silabs firmware builder, and even added Z-Wave Controller too, plus archived their repo for binaries:

https://github.com/NabuCasa/silabs-firmware-builder/pulls?q=is%3Apr+is%3Aclosed

https://github.com/NabuCasa/silabs-firmware-builder/commits/main/

[!IMPORTANT]
New firmwares are released straight from the SiLabs Firmware Builder repository. This repository contains all past firmwares and will not receive future updates.

https://github.com/NabuCasa/silabs-firmware

So, new official Zigbee and Thread firmware releases for SkyConnect + Yellow and future Silicon Labs firmware will from now be released straight from the SiLabs Firmware Builder repository

As such the "old" silabs-firmware repository is now listed as archived and will not receive more updates but it will remain available as it contains all past firmware versions.

PS: Off-topic however interesting is that the silabs-firmware-builder now also contain Z-Wave Controller firmware for the upcoming official Home Assistant Z-Wave Controller USB adapter hardware that Nabu Casa is working on -> https://www.home-assistant.io/blog/2024/05/08/zwave-is-not-dead

Hedda commented 5 months ago

@Daniel-zhan-itead @liangjia2019 @guozi7788 I would still like to strongly recommend that ITead look into adopting an official fork of Nabu Casa's "silabs-firmware-builder" that allows you to easier reproduce and maintain firmware builds for Silabs adapters.

Again, Nabu Casa seems to currently be in the process of rewritting tooling and reorganized manifests (configuration files for different adapter models and firmware builds) in their upstream Silabs firmware builder to both update it and make it easier for others to reuse it as well as enable support for other type of firmware builds for Silicon Labs based adapters. At least it looks like puddly has in the past few month or so rewritten and reorganized much of Nabu Casa’s upstream Silabs firmware builder, (and even added Z-Wave Controller too in addition to EmberZNet NCP, Multi-PAN, and OpenThread RCP profiles via manifest files):

https://github.com/NabuCasa/silabs-firmware-builder

It now contains a dedicated folder for manufacturer that are suppose to contain manifest files with config for each build type:

https://github.com/NabuCasa/silabs-firmware-builder/tree/main/manifests/nabucasa

Some additional changes also looks planned:

https://github.com/NabuCasa/silabs-firmware-builder/pulls?q=

Otherwise if you want to fork and use something that already work for ZBDongle-E today then look at forking darkxst's fork of it:

https://github.com/darkxst/silabs-firmware-builder

That already have manifests for third-party adapters including a manifest for ZBDongle-E that include custom configuration:

https://github.com/darkxst/silabs-firmware-builder/tree/main/manifests

(with zbdonglee.json config -> https://github.com/darkxst/silabs-firmware-builder/blob/main/manifests/zbdonglee.json used in combination with defaults.json -> https://github.com/darkxst/silabs-firmware-builder/blob/main/manifests/defaults.json )

PS: Off-topic but another FYI; Nabu Casa working on Bluetooth RCP firmware for their EFR32MG21 based adapter/dongle too.

Hedda commented 3 weeks ago

@Daniel-zhan-itead @liangjia2019 @guozi7788 I would still like to strongly recommend that ITead look into adopting an official fork of Nabu Casa's "silabs-firmware-builder" that allows you to easier reproduce and maintain firmware builds for Silabs adapters.

@Daniel-zhan-itead @liangjia2019 @guozi7788 FYI, many other downstream devs are now collaborating on improving upstream:

PS: Upstream will soon also migrate from Gecko SDK Simplicity SDK which will bring larger changes to dependecies and builds.