Need insight and suggestions about the framework.

[mark-klein1]

I am exploring options for a minimalist automation system to integrate multiple Zigbee Aqara cubes into a web application.

The framework here might fit the bill given it is designed mainly for a narrow range of industrial applications, and importantly it cares about how responsive and reliable the automation is. If we use several dozen Aqara cubes the responsiveness will become a bottleneck on home automation platforms.

I am looking into and contrasting OpenHAB and Home Assistant as both are open source. OpenHAB uses Java which I have way more experience than Python, but other than that I like the Home Assistant Yellow box that includes a Zigbee coordinator and its small plastic enclosure is perfect for the intended audience.

However it is doubtful the HA YellowBox can process 20,30,40,50 users each moving around an Aqara cube. So the solution in this repo might be the way to guarantee performance and reliability. Still using the Yellowbox as a Zigbee gateway but offloading all the processing to a nimble separate framework

The framework has to keep track of all the cubes and keep real time communication with other servers over the internet.

If anyone can add insight, give options, compare system, etc it will be appreciated. You can also send me email at Mark.Klein1@gmx.de as I seldom log into github but read the emails more often. We can also use Whatsapp if you wish.

Also I have a second unrelated project where the framework here might be usable as it is to automate a propane gas valve to build a smart oven. This is a very simple system consisting of a temperature sensor and the propane valve, essentially process control of the oven. Any suggestions about what could be used as a very simple controller for this ?

There are other much bigger projects in mind too, like building a private data center for AI, in case anyone would like to get involved. I am located in Germany, we have a free building that can be used for any interesting project without any cost.

Thanks in advance.

[thwerks]

Hi Mark, In my testing, Home Assistant isnt very reliable and it seriously lags, so i personally only use it for a HUD or for its graphical interface or android app, hence the purpose of this lightweight framework. So far my systems using this HA/NodeJS framework in combination with HA Core work very well and the delay is extremely low. I have developed some basic valve control and pump modules and its very reliable.

Its good timing you reached out to me because im just about to get into my lab again and have an evaluation of this system, do some testing and add some new features. Namely incorporating setState via the twoColors Websocket API and hopefully direct communication with ESP home devices using either an existing api or just make my own; incorporating these two things further reduces the need for using Home Assistant which is just pathetic in terms of responsiveness and reliability.

As far as Zigbee goes, i didnt plan on incorporating it into my framework directly but im open to it. At the moment Zigbee devices would have to proxy throught Home Assistant Core but then be handed directly to this Framework, the delay is very small in this case.

I dont know anything about OpenHAB but as far as i know, the Home Assistant GUI is the best around but none of these systems can even come close to NodeJS in terms of raw performance and reliability. NodeJS can concatenate at nearly the speed of C++ and simple modules are easy to thoroughly work out flaws and unruly behavior; something that seems rather difficult to achieve with HA and other large solution incorporating so many elements.

But thats really the point of this project, to remove all logic and as much communication as possible away from these large solutions and move it over to a pure NodeJS framework that is very light weight, with a few basic modules and input/output mechanizes but leave the front end to these large heavyweight solutions. At that point, i dont really think it matters much if you use OpenHAB or HA Core, though i dont have any api for OpenHAB and im not familiar with it.

[thwerks]

I also wanted to mention that i have some Zigbee devices and i want to make a switch control module in this framework. I might just proceed with that directly using setstate in the TwoColors Websocket API. So now really is a good time. We could even use your project as an example for creating this module or a variant of it anyway.

[mark-klein1]

Thanks for your prompt reply , I found your project yesterday but quickly had the impression that you were building what I was looking for. I do not even care much about their GUI , only because HA includes a Zigbee chip I was looking into it, as the Aqara cubes are the core element for what we want to build. So if you can add support to interface with Zignee devices without HA I am all for it , what would replace the HA Zigbee coordinator in that case ? .

I need to have many people playing with the cubes all at the same time.. and use their actions to interface to servers off premises that actually run the web applications. The HA integrations could be used for other capabilities in the future, like beacons but this is probably doable without HA as well.

I want to use a small factor computer like the Yellowbox, it would be a standalone dedicated solution and nothing else would run in it.

[thwerks]

Creating a zigbee coordinator would probably be outside of the scope of this NodeJS framework because that alone is more complex than the entire system itself and since this framework is just for raw logical processing and communication it wouldnt ever have a GUI for which you could easily manage and add Zigbee devices.

That being the case I'd say there's two realistic options.:

Firstly you could find an independent Zigbee (or better Zigbee2MQTT) Coordinator system that has a nice API and or find an already made NodeJS Zigbee /Zigbee2MQTT API to integrate into a standalone module that may or may not connect to my framework. If you went this route it think its much better to find a Zigbee2MQTT controller (or NodeJS API) rather than Zigbee. I have very little experience with the inter workings of Zigbee so i wouldnt be inclined to create this by myself.

Second option is easier still, just use HA Core as the zigbee coordinator only and pass all the IO directly to my NodeJS framework.

Since HA has a coordinator already and has a nice GUI by which you can add and manage all your zigbee devices, and also since HA Core has very low latency (like 1-5ms); my gut feeling is that using HA Core would probably work just fine. So i suggest you try it that way first before looking for into a potentially much more complicated solution which at best might only shave off a couple milliseconds.

I measured 1-5ms with very low end hardware (Rock64 with 1GB of ram) running HA core; thats quite low really. I'd expect even lower latency with proper x86 hardware so i think thats acceptable for your use case and would most importantly be very easy to implement. I will just say the 1-5ms latency is for HA and the websocket API itself, i have no idea how efficient HA's Zigbee stack is or how it compares to others.

I'd be willing to help you get my framework up and running for your application. I would just ask that you pay attention to my documentation and say where it falls short so that i may modify my documentation to be more complete and helpful to others.

You can see here how to install HA Core: https://github.com/thwerks/NodeJS-Home-Assistant-Framework/blob/main/install-ha-esphome.md

[thwerks]

This looks pretty interesting. This might be exactly what you want, this probably would be a lot more efficient than using HA Core as Zigbee coordinator. If you can get this working with your hardware, i will write an MQTT plugin for my framework so you can use connect to it.

https://www.npmjs.com/package/zigbee2mqtt https://www.npmjs.com/package/zigbee2mqtt-frontend

[mark-klein1]

Ok that sounds good. I more or less had reached the same conclusion about using HA for the Zigbee coordinator.

I am not really familiar with any of these home automation solutions but understand what they do , The zigbee chip in the Yellowbox was the reason I looked into that. I will have to look if there is another hardware option but the yellowbox from HA are fine if we do not find any limitations to run your system . They also announced using a Pi 5 with double the power for the future.

I refer to the small factor for the size convenience, the cost is not a major factor, the important thing is reliability as it will be used by people with zero tech knowledge. It has to be a box with a power button and nothing else more or less..

I will look into your suggestions, the zigbee2mqtt on nodejs is a good route .the hardware that handles the zigbee protocol is the key to integrate the cubes. HA Yellowbox has an integrated chip on board for that, no usb ConBee II Universal Zigbee USB Gateway and the like, that is elegant and convenient

Please send any hints about using the yellow box with your framework or any other hardware solution you recommend.

[mark-klein1]

What machine, OS version and development environment are you using ? I am getting a new workstation soon and can duplicate what you have to minimize issues. When the projects become important I dedicate a machine to each project, a bit paranoid but it is easier to find and fix problems as nothing can possibly be interfering with anything.

[thwerks]

I use only Debian 12 Live or LMDE6 on my PCs, they are basically the same but nowdays im moving more to LMDE6 as it has gone though our testing well and is a little more polished. I only have VS Code for my DE.

As far as SBCs, i usually use like Armbian minimal or Diet Pi sometimes; i dont think there is really any difference between Debian and Armbian as fare as HA Core and my software are concerned. Any ubuntu based ones are gonna be fine also and Rasbian of course; I thtink Armbian Minimal is the most lean and reliable in my opinion.

My framework was written for NodeJS 16 but will now move to v20. So all that is needed is to install NodeJS 20, NPM and telegram api using npm (because it has every dependency the framework uses) and follow the instructions for installing docker and HA core. https://github.com/thwerks/NodeJS-Home-Assistant-Framework/blob/main/install-ha-esphome.md

After that, just need to construct you own "automation" as outlined in the ha.js example file that provides the logic you require.

I have no idea what your situation is but it sounds like both your input and outputs are both Zigbee so if thats the case you'll need to setup and run the ha.js app and have a look at what devices are available to you by visiting http://127.0.0.1:200/ha

[thwerks]

i just added a new file:

https://github.com/thwerks/NodeJS-Home-Assistant-Framework/blob/main/ha-plane.js

The pump control automation has been removed, i just tested in nodejs 20.

you must install this module npm install @2colors/esphome-native-apis . it may require the telegram module later or others as your configuration progresses.

if you put only the IP address and token of your HA Core in the config there as specified, it will run, spit out all available HA entries and then terminate.

Then you can put the entities you want to control in the cfg

I realize im lacking a lot of information about reading and setting states and how to use the event listener for input and outputs in HA.Ill make a compete guide for that tomorrow. its late here.

[mark-klein1]

Thanks, that was helpful. No rush at this time at least, take your time. We are on holidays for the rest of the year and it will take longer than usual to receive the workstation and other hardware pieces. So initially it will be slow on our side, but I can read documentation and acquaint myself with your framework.

Please explain more about the Telegram dependencies. It is always a little unnerving depending on external libraries that we do not control. We usually develop inhouse everything that is really important, I found a very good library for Whatsapp for example but do not use it because they change things that break the code or might pull the plug when you least need it.

Ubuntu / Debian is not problem, Never used Armbian Minimal but like their note about being open and no bloatware or spyware. Using a black box for critical code is out of question here. I realise you are targeting small platforms for industrial uses while I do not care much about that level of efficiency , we prefer small size computers but as powerful as possible, plenty of memory, fast communication, and not having to worry about resources constrains. Reliability and happy users is what matters.

zigbee2mqtt supports a couple of Raspberry Pi hat adapters and their documentation explains how to add new devices, it is already integrated to HA. I will check into how they do that but it looks like HA + zigbee2mqtt is the easiest route to add your framework to use the cubes. It also decouples de zigbee network from HA so we might be able to toss HA in the future if needed.

[mark-klein1]

Aqara magic cube T1 Pro https://www.zigbee2mqtt.io/devices/CTP-R01.html We are almost there.

[thwerks]

Telegram package is now only required if enabled in the config, same with ESPhome API.

My framework does not support MQTT, it only communicates with ESPHome devices or Home Assistant directly. Home Assistant is the coordinator so it makes no difference to my framework if you use ZHA or Zigbee2MQTT.

If you see the NPM link i sent you, it takes no less than 4 different frameworks to make a complete Zigbee2MQTT stack in NodeJS, so its not a trivial matter. Whats the performance difference between using HA Core or a native NodeJS stack or other type of Zigbee2MQTT stack? Probably not much difference would be my guess.

Zigbee was never intended to be a performance or reliability oriented solution, it isnt very reliable and you cant even get RSSI information in ZHA which is pathetic. Bottom line, i would never use Zigbee for anything but home automation which is very trivial. ESPHome is technically a lot better than Zigbee. The APIs are much cleaner, the overhead is way less, the hardware is much more reliable and the advantages of using TCP/UDP over wifi or ehternet network as compared to a zigbee proprietary network are rather obvious.

So at this point i have no intention to ever include Zigbee functionality ever; MQTT relaying might be worth the effort but only if there was a significant performance advantage of some other coordinator over that of Home Assistant Core.

The usage of regular Home Assistant or sometimes refereed to as home assistant supervised is totally out of the question, that is just junk. You must use the Core version.

Im currently in the middle of making significant changes to the framework model so many things will change. Currently struggling to decide whether to implement a modular approach or keep it monolithic. i should have a new version in a week.

[mark-klein1]

I do not care which protocol is used as long as it works reliably for our intended applications. The reason we are forced to use zigbee is because we want to use the Aqara cubes. If we can find a similar cube controller using other protocols then we will also consider it.

We are pragmatic about solutions, the times for long debates about which programming language is best and the likes are long gone. I favor C/C++ and Java because that is what I grew up with but we now use javascript to build mobile and web apps with a single language. React and React native to also be consistent using similar environments without multiple frameworks and tool chains. NodeJS and your framework fit nicely here.

HA core is fine as long as it does not become a drag or reliability problem, already read some people complaining about having to maintain their HA system often to prevent it from dying or whatever. I think it is a usable tool as a monitoring console if we can use it remotely, but keep in mind we are not building anything related to home automation per se.

We need a framework to keep track of all the cubes states, as nimble and reliably as possible. We will have to experiment and test to see as to what it is capable of, as yourself mentioned the protocol is not speed demon or something you should bet much on it but also read that it is more robust than wifi.

In general I guess eventually your framework should be agnostic as to what protocol we use, I am most interested in its architecture and what algorithms are used so we can adapt it to whatever we need as things evolve over time. Initially anything that works is fine, so for me at this point if it is monolithic or not but it works then good.

[mark-klein1]

I already received a new workstation for development , a few Aqara Cubes, and I am trying to decide what Adapter to order for Zigbee2MQTT . Since the adapter will be external to the box I am also leaning towards a mini PC for the end users.

According to the Zigbee2MQTT documentation it supports automatic discovery of Zigbee network Adapters. In order to use this feature, the adapter must support discovery via mDNS Zeroconf. Only 2 adapters are tested and recomended with this capability, both support Wifi,USB and Ethernet and are expensive but look solid. My concern here is only for the end users, as everything will have to be monkey proof once they receive the mini PC. I have to minimize any interaction with them and maintenance issues.

I am also looking into the Node-Red integration for Zigbee2MQTT as this is obviously already running in NodeJS. Apparently it is easy to interact with the MQTT broker as Node-Red was originally created as a tool to manipulate mappings between MQTT topics. The Zigbee2MQTT integration has its own module so perhaps it is easy to take out only what we need.

It is also possible to integrated ESPHome directly to NodeRed without the Home Assistant. I do not know what significance this might or not have for your framework or if it makes sense for your particular situation. What is your opinion about Node-Red in general and in particular for your own framework ?. Would Node-Red help you out in any way ?

For our project we do not wish to become dependent of any other big project be HA , Node-Red or other home automation solutions as this would necessarily take time and maintenance from our side.

[thwerks]

I'm starting to test some of my projects now with mini PCs. I started using HP T630 and Dell Wyse 3040 or 3060 etc. They are very nice and reliable running Debian; been using the T630 for a couple years and very happy with them, the T620 also. Both models have a quad core version. I'm using many of the Sonoff Zigbee -P dongle and have no issues at all.

I'm now fully into a total transition of my core code. I decided to go with a modular design mainly because updating the code is much easier than with a monolithic design. Also I'm very disappointed with how many of the other APIs are written. They are so sloppy, filled with memory leaks and are so dirty i literally have to use fork to quarantine them into their own processes to ensure reliability and performance. This is a much more reliable approach; my software is used for machine control where there is a lot of money at risk if things malfunction at the wrong time.

I will say this. Home Assistant Core is very very reliable as an ESP subscriber and Zigbee Coordinator itself. Though It's automation is a complete joke and is totally worthless trash, and i mean that with all seriousness. I wouldn't trust it to fill a kiddie pool properly without something going wrong.

I've been performance testing the HA Core ZHA stack all week and consistently and reliably get it to respond with about 50-60ms of latency; that's round trip time, callback to the API, not bad. Though i do see about a 10ms difference between the APIs event statechange callback vs the official response callback which shows how ridiculously slow python is; the only difference between these two callbacks is that one is simply generated after the other and with a 10ms hit, ouch. Of the 60ms of lag, not even 1ms is incurred by my framework.

I personally think NodeRed is the most obnoxious piece of software ever created. Its like trying to wash the dishes with both hands tied behind you back. I can do things in nodejs in a matter of minutes that take an hour or more in NodeRed and without the gut wrenching overhead. I think the situation where NodeRed might actually be useful is in a very narrow set of circumstances; your situation might be one such situation. But NodeRed is not really a professional grade solution in my opinion; a one size fits all solution never fits anything well and NodeRed is pretty fat.

My framework is now fully modularized and now has full support of the ESP API and can function completely without HA. Only ZHA or Zigbesss2MQTT would require the use of Home Assistant or some other broker/coordinator. I've also added performance monitoring and other features that might apply to you.

I wouldn't ever use NodeRed for ESP stuff, i use the official ESP API as NodeRed uses but directly. Personally, i think using NodeRed for Z2MQTT is a pretty good idea to experiment and create a performance baseline for your project but i would get the data out of NodeRed immediately, like even at the MQTT singling level and get it into my framework because doing logic stuff in NodeRed is like making breakfast using endoscopy. Write your logic in plain JS in NodeJS. my frawork doesnt required HA or ESP APIs anymore.

would you need anything besides NodeRed to use Z2MQTT? Is it not possible to use NodeRed as the broker? If not so, i would dive right into that NodeJS Z2MQTT library i sent you earlier. If i even had any Z2MQTT stuff id be looking into it now but i have only ZHA stuff. I'm pretty sure that NodeJS Z2MQTT library would be significantly better than NodeRed or HA and probably anything else.

Ill be done with my new system in a few days, i tested most of it and its going smooth, just need a few days to implement some new design changes. Try to use UDP in NodeRed to send out JSON data, this is how you would connect to my framework.

[thwerks]

The whole [NodeJS-Home-Assistant-Framework] project will be closed and become ThingWerks Core. A general purpose IoT logic core with performance tracking, logging, telemetry and remote access, remote updated, config and NV data backup, plugins for HA, ESPHome and other things(like telegram etc) and eventually Z2MQTT hopefully; each can be enabled or left out depending on what the person wants. Then the individual IoT systems that i make (like Pumper pro, Power Werks, Water Werks, etc) and people like you make will be loaded as independent modules that leverage the TH-Core for standard plugins, IO and base features.

A lot of this sounds very similar to what you said your doing, we do different things but there is a lot of overlap here. This will be an extremely light weight offline system almost to the point where it could be considered glorified boilerplate code. But this serves a very specific purpose that i have repeatedly encountered and im wired of doing the same thing over and over.

the main purpose of doing this is a have a lot of IoT projects that share a lot of basic things, rather than recreating the wheel with every project, its better to have a basic core and load modules to the core. im open to ideas, if you have needs different than mine, we can try to include them. and il also make it so that APIs i use that you might not use can be excluded from the runtime environment, like selectable plugins or something.

Do you have any R&D budget? im willing to look into using the NodeJS Zigbee2MQTT API but i need hardware and time to dedicate for that. For sure that would be the ideal solution as compared to using HA, NodeRed or anything other crap like that. Going directly to a well made API to incorporate Zigbee2MQTT is of course the most flexible, fastest and most reliable way to do what you want.. I am very experienced in writing technical documentation and how-to guides.

Why dont you try using this and see if you like it. If so, i'd really appreciate your feedback and the things you need for your project would be a good test of this system anyway.

you mentioned you have servers. do you have a VPN concentrator that you can use to remotely manage your clients? if not i might consider adding a websocket service into the TW-Core to aid in remote connectivity for telemetry and administration etc.

[mark-klein1]

Sorry for the delay. I will read your comments tomorrow to continue working on this.

I had to travel from Germany to California and other places for some business issues and did not have a way to log in here. I left another programmer setting up the machines and the zigbee2mqtt install but he was confused with the installation and it is only partially working, so I requested some help and tomorrow we will try together.

[mark-klein1]

I quickly read your notes to follow up, as we progress I will reread everything to start picking up on the details.

Definitely we need the system to be nimble and very responsive, so an standalone lightweight system was always part of our requirements.

HA or NodeRed are only for prototyping, checking that things are working etc. we will never incorporate anything like that in commercial offerings, the same goes with any other piece of software we use and eventually we will have a system with no external dependencies. That is the only way to prevent headaches and building a robust reliable system.

We were able to make Z2MQTT work with the networked adapter by hard-coding the IP address on its configuration file, the automatic discovery routine using mDNS is malfunctioning: but everything is working now . Using MQTT explorer and the Z2MQTT web interface as realtime consoles, no need for HA or Rednode so far.

The system works very nicely except its responsiveness is a bit unpredictable, once in a while reacts almost realtime, but most often there is a lag of up to few seconds, and this we need to eliminate, so before we can use these cubes on zigbee we have to figure out how to make them reliably responsive. I do not know what is the cause, but this would be our main initial target as everything else might depend on that.

I assume this is also one of your main concerns if you are monitoring industrial equipment. I also assume you would need remote access to monitor your systems and fix something remotely, updates and so forth. For us using a commercial messaging system is not an option, only if we can have a custom inhouse built of a complete system we can use it, say Matrix or similar.

We can buy new hardware for these projects, our time is more of a problem , this is a sidekick project of low priority unless we come up with solutions so outstanding that we can dedicate ever increasing amounts of time and resources.

We develop things inhouse within internal networks, no outside connections, If someone needs an internet server we lease a dedicated server from outside vendors, that is how we can collaborate easily as only you and other programmers here will have access to that. The providers only load the OS you request and you own the machine from that point on, no restrictions.

Your idea of using a modular architecture with plugins is obviously advantageous, we would only need the minimum core functionality to guarantee reliability and responsiveness.

Let me know how we can test what you have now, to get familiar with your system. You mentioned something about pumps, I have a few garden pumps and several industrial electrical motors seating on shelves , they could be used for testing or building something interesting, I enjoy tinkering with mechanical and electrical things too.

[thwerks]

when you say you got Z2MQTT to work, does that mean some NodeJS Z2MQTT NPM package? specifically what API are you using for that?

i agree that HA or NodeRed arnt really suitable for a packaged solution. But when prototyping with HA as only a Z2MQTT broker i see latency in the 40-60ms range and its very consistent. You may want to try some testing using my Framework to establish a performance baseline. The >1000ms latency you speak of is ridiculous and ive never experienced anything like that. Whats the physical distance between the gateway and the devices?

My new IoT framework doesnt require any external dependencies anymore and is extremely lightweight, everything has been stripped out. The only dependency in your case would be the Z2MQTT API itself unless you planned on completely making that yourself which i dont suggest given the time considerations. I suggest using an already made NodeJS based solution if you weren't already; its not clear to me what exactly your using for that based on what you said.

my framework is completely standalone and doesnt require any remote access. for the most part its working very well i just have a few things i need to update and i should be dont in a couple days. then ill upload the latest files to github and you can check it out, a least prototyping with HA Core as a Z2MQTT broker is ready to go. As for integrating Z2MQTT directly into my framework, that might require some specific coding which you and i could work on together.

On Wednesday, January 24th, 2024 at 12:10 AM, mark-klein1 @.***> wrote:

I quickly read your notes to follow up, as we progress I will reread everything to start picking up on the details.

Definitely we need the system to be nimble and very responsive, so an standalone lightweight system was always part of our requirements.

HA or NodeRed are only for prototyping, checking that things are working etc. we will never incorporate anything like that in commercial offerings, the same goes with any other piece of software we use and eventually we will have a system with no external dependencies. That is the only way to prevent headaches and building a robust reliable system.

We were able to make Z2MQTT work with the networked adapter by hard-coding the IP address on its configuration file, the automatic discovery routine using mDNS is malfunctioning: but everything is working now . Using MQTT explorer and the Z2MQTT web interface as realtime consoles, no need for HA or Rednode so far.

The system works very nicely except its responsiveness is a bit unpredictable, once in a while reacts almost realtime, but most often there is a lag of up to few seconds, and this we need to eliminate, so before we can use these cubes on zigbee we have to figure out how to make them reliably responsive. I do not know what is the cause, but this would be our main initial target as everything else might depend on that.

I assume this is also one of your main concerns if you are monitoring industrial equipment. I also assume you would need remote access to monitor your systems and fix something remotely, updates and so forth. For us using a commercial messaging system is not an option, only if we can have a custom inhouse built of a complete system we can use it, say Matrix or similar.

We can buy new hardware for these projects, our time is more of a problem , this is a sidekick project of low priority unless we come up with solutions so outstanding that we can dedicate ever increasing amounts of time and resources.

We develop things inhouse within internal networks, no outside connections, If someone needs an internet server we lease a dedicated server from outside vendors, that is how we can collaborate easily as only you and other programmers here will have access to that. The providers only load the OS you request and you own the machine from that point on, no restrictions.

Your idea of using a modular architecture with plugins is obviously advantageous, we would only need the minimum core functionality to guarantee reliability and responsiveness.

Let me know how we can test what you have now, to get familiar with your system. You mentioned something about pumps, I have a few garden pumps and several industrial electrical motors seating on shelves , they could be used for testing or building something interesting, I enjoy tinkering with mechanical and electrical things too.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[mark-klein1]

To install and test the Z2MQTT solution we just followed the instructions in the Z2MQTT website.

We used a zigbee adapter with an Ethernet interface and attached it to a network ethernet switch separate from the computer. Most people use USB dongles for this and we also ordered USB dongles to test and compare, but the detached set up seemed easier to test from different machines running Linux , Windows, etc. so we started with that.

We are using a Windows machine and initially tried to clone the git files for a Windows only install to obtain a complete system with sources but ran into issues of permission to allow access and the like and did not have a lot of time so tried the Docker install instead. We followed the Docker installation instructions and eventually everything started working (Ignoring the auto detection feature which did not work with a fully networked adapter, perhaps it might work if we use the adapter in the USB configuration)

I was not too surprised about the delays because I have seen a few videos and people commenting about these cubes to behave a bit slow and describing similar behavior to what we encountered. All these comments from users using HA integrations. But I was hoping to have better performance since we did not have the extra overhead of HA on top of Z2MQTT

I am not familiar with HA let alone its internals, but I think they have other integrations with the zigbee networks in addition to Z2MQTT and the MQTT broker. Please comment about this and what different approaches we should try to compare performance and responsiveness. Your own framework probably uses a different approach, and you mentioned before zigbee was not part of your project , the performance of zigbee would definitely be much different than a wifi or other types of networks. It is by design a slow mesh network.

The challenge is to make these cubes be responsive in a predictable way , a bit slow is fine if at least it is predictable so we can design the whole system compromising for that . We will have to download the Z2MQTT code and dig into how the sniffer works, and what can be done to get faster and more reliable behavior if at all possible. The closer we integrate to the zigbee network the more control we will have, may be we have to ditch the MQTT broker too..