Any interest in providing the web UI/storage separately?

[joneshf]

Describe you new feature you'd like

This idea might not make sense, so feel free to reject. Or if you know of another project that already does what I'm thinking about, please redirect me. But, would it be possible to separate the transceiver interface from the web UI and the rest of the logic?

Backstory

We used to have Zooz ZST10 USB Stick for our Z-Wave setup. It worked well. But because its interface was USB, we had to have a computer located where we wanted the Z-Wave controller to be. That computer also ran Z-Wave.js UI (because exposing a USB Z-Wave controller outside of a computer is non-trivial) so we could interact with Z-Wave and also integrate with Home Assistant. Eventually we found a project about running a standalone Z-Wave controller over Wi-Fi.

We've now got a setup where a Zooz ZAC93 connects to an ESP32 over UART and that ESP32 runs a server on ESPHome. We run Z-Wave.js UI on a computer tucked away elsewhere, and connect to the Z-Wave controller via TCP. Not only are the parts cheaper ($23 for controller + ~$5 for ESP32 vs. $35 for USB stick) but we also can put Z-Wave.js UI on shared compute (Proxmox, k8s, etc.) instead of needing a dedicated computer in the middle of the house.

We have a similar setup for ZigBee. Instead of buying a dedicated ZigBee hub or ZigBee USB stick, we picked up a Smartlight SLZB-06 PoE ZigBee antenna. Similarly, ZigBee2MQTT sits on shared compute (Proxmox, k8s, etc.) instead of a dedicated computer in the middle of the house, and we connect to the ZigBee coordinator via TCP.

Actual Request

It feels like there could be a similar story for Somfy RTS. Currently, we're using ESPSomfy RTS on an ESP32 in the middle of the house. But as mentioned multiple times in the wiki, in issues here, and in that thread on the Home Assistant; the ESP32 doesn't have a whole lot of space on it, so some trade-offs are being made. What if instead of trying to use the ESP32 to be an all-in-one device that had a web UI backed by an HTTP server, an interface to the transceiver, and the information for each shade/device; it was only an interface to the transceiver and a TCP serial server? The rest of the project would be runnable elsewhere like Z-Wave.js UI, or ZigBee2MQTT.

The idea is that a VM/Pod/whatever can have considerably more compute than an ESP32. Enough compute that almost all of the restrictions and workarounds you've had to put in can go away: no need to restrict the number of shades, no need to limit how long names can be, no need limit groups, etc. But also, new ideas might be possible: multiple transceiver/ESP32 devices all interfacing with a single web UI server, having better observability because logs can be on full-time/metrics can be made available, or other things that are pipe dreams on an ESP32.

I'm not suggesting that you do away with the ability to have an all-in-one ESP32 that Just Works™. I think that's one of the draws of this project. More wondering if you're interested in the idea of supporting this additional use case. What do you think? Again, totally fine to say, "That's out of scope, kick rocks."

[rstrouse]

I did a similar thing with 15 ESP8266 devices that measure soil moisture around my landscapes. All told there are 75 sensors buried in the ground. The solution stands up a node server on a linux box and receives readings over TCP. It does this so it can calculate plant demand and target moisture levels at varying depths and manage a database of ~150 plants in my landscape. While that system is fine for me it wouldn't be something that I would be able to support beyond my own purpose built environment.

I also have my pool controller communicating with a Raspberry Pi out at the pad to control things like pH, ORP, pressure, relays and valves. This communicates with a node server on my linux homeserver to keep track of all the chemistry, control the equipment, and maintain sanitation levels. You can get a gander of this craziness at the link below.

https://github.com/tagyoureit/nodejs-poolController/wiki/DIY-REM-Chem-Controller

Unfortunately, not very many of the components that are required for ESPSomfy RTS could be removed in a multi-server situation like that either. While the ESP32 expands the available resources a lot over ESP8266, the joke was on me as the networking and server components consume most of it anyway. Heck I just tried to upgrade to ArduinoJson 7 and to my surprise it caused my C series chips to run out of code space. I had to roll it all back to v6 and then I read the disclaimer regarding why they don't care about code space anymore.

I am planning on doing a diet re-write at some point to see if I can reduce some of code and ditch a couple of external libraries.
I no longer use ArduinoJson for serialization anymore and I can't use it to parse the Github data since that is waaaay too big to digest on the ESP32 heap.

With ESP32 you have to very judicious at how much stuff you throw at the socket. As it stands right now the transceiver, command logs, and state changes can be received all the time. But redirecting the serial port to a TCP stream is a bit more code than I have space for at the moment. All that is needed for that is a TCP server that is ready to listen.

[joneshf]

Interesting. So if I'm understanding you, it sounds like Somfy RTS is in a whole different class of protocol than Z-Wave and ZigBee. I'm sort of surprised it matters at all what the protocol is. I was assuming that all of the logic/processing/serialization would happen on the server, that server would send raw bytes to the ESP32, the ESP32 would be running something like the ESPHome stream server where it forwards those raw bytes to the transceiver over UART, and the transceiver would send the bytes to the devices/shades (and the other way for receiving data).

But it sounds like there still needs to be a layer of translation between the ESP32 and the transceiver to make Somfy RTS work. Is that right, or am I misunderstanding what you're saying?

And sorry if my questions/ideas don't really make sense. I'm more of a software person than a hardware/firmware person.

---

I did a similar thing with 15 ESP8266 devices that measure soil moisture around my landscapes.

This sounds really neat! If you ever get the motivation to write about it, I'd definitely be interested in reading it.

[rstrouse]

Interesting. So if I'm understanding you, it sounds like Somfy RTS is in a whole different class of protocol than Z-Wave and ZigBee.

Oh yes. ESPSomfy RTS is decoding/encoding raw radio signals. The beat count is in the microseconds (not milliseconds) range so the decoding must occur on the ESP32 as the timing is critical when receiving. The preambles for this radio transmission are also not fixed so receiving requires decoding the incoming bits real time to ensure the entire frame is read.

A UART is always buffered and uses a dedicated preprocessor that reads the status RX/TX line to collect the bytes. While the CC1101 does have buffering capability for fixed ASK frames it proved not to work properly to read all RTS signals due to the weirdness in the way frames are constructed. This challenge starts with the preamble and ends with the silence between the frames.

ESPSomfy RTS uses bit-banging to read the OOK signal. OOK signals are not 0 or 1 inherently, they are determined by symbol timing of the previous deviation. Essentially, the ESP32 is performing what would normally be done in a software UART emulation. If you remove the UART from the equation on serial comms you will get an idea of what ESPSomfy RTS is doing. It is reading the interrupt from high/low low/high and translating the timing between those transitions within the tolerance of the symbol length.

Theoretically the notion of sending (transmitting) could use a preprocessor on a separate server but that code is not even close to being complex or burdensome. Essentially the code simply shift keys the radio high or low at very tight intervals to encode the data bits.

And sorry if my questions/ideas don't really make sense. I'm more of a software person than a hardware/firmware person.

I too am a software guy so my propeller spins in the proper direction. The hardware side was simply a curiosity and the radio tech was simply in the way. To be fair, being a software guy on platforms with many more resources, it's tough to not add extra defensive code and logging. It's like being back in the days of himem without overlay support or memory mapped files. Strings become evil on the stack and in the heap.

[joneshf]

Oh yes. ESPSomfy RTS is decoding/encoding raw radio signals. The beat count is in the microseconds (not milliseconds) range so the decoding must occur on the ESP32 as the timing is critical when receiving. The preambles for this radio transmission are also not fixed so receiving requires decoding the incoming bits real time to ensure the entire frame is read.

A UART is always buffered and uses a dedicated preprocessor that reads the status RX/TX line to collect the bytes. While the CC1101 does have buffering capability for fixed ASK frames it proved not to work properly to read all RTS signals due to the weirdness in the way frames are constructed. This challenge starts with the preamble and ends with the silence between the frames.

ESPSomfy RTS uses bit-banging to read the OOK signal. OOK signals are not 0 or 1 inherently, they are determined by symbol timing of the previous deviation. Essentially, the ESP32 is performing what would normally be done in a software UART emulation. If you remove the UART from the equation on serial comms you will get an idea of what ESPSomfy RTS is doing. It is reading the interrupt from high/low low/high and translating the timing between those transitions within the tolerance of the symbol length.

I only get some of what you said, but I think I understand what you're saying. Thanks for taking the time to explain how this stuff works. I guess this will have to stay a pipe dream.