Building a Bridge

[CSchoch]

Hi,

Which Hardware as Esp32 replacement would you use to build a proper Bridge to Wlan or Lan. Considered the high loss of esp32. Kr Christian

[pvvx]

ESP resources are insufficient to serve the full TCP / IP networks specification. For toys only. For normal operation on IP networks, they use Unix-like operating systems based on the Linux kernel ...

[ikke-t]

For the reference, I use Raspberry Pi with bluewalker sniffing the broadcast data, and pushing it to telegraf socket, which sends them to influxdb. It's often overkill, but I have the Raspis anyhow in my home automation setups.

[nagyrobi]

ESP32 is known for high loss mostly because it uses the same radio module for BLE and WiFi in a shared way. There are many Bluetooth implementations for ESP at various qualities of handling this.

You can still get good BLE performance on ESP32 if you're not using WiFi at all, but use an Ethernet module instead. Look for Olimex ESP-32-POE, WirelessTag WT32-ETH01, TTGO T-Internet-POE ESP32 or similars.

It will disable wifi completely and radio will only do BLE. It will not catch all the messages but will be perfectly usable for room temperature control in everyday life, multiple messages per minute will surely arrive into the system from each sensor advertising. Place it in a central location at relatively similar distance from most of them. Avoid placing the ESP node in racks, too close to switches or other network equipment as EMI interference may degrade signal reception.

Run ESPHome with Ethernet configuration and use esp32_ble_tracker with pvvx_mithermometer. Here's my full example config: https://github.com/nagyrobi/home-assistant-configuration-examples/blob/main/esphome/ble-sensors-ethernet.yaml

For ESP32 node with Ethernet LAN connection, use interval and window set to equal 5s. If you don't have LAN module, you can still try with WiFI, but use interval and window set to equal 300ms. Best is to scan passively. Ethernet will give better results.

Do not add web server or any other component which is not absultely necessary as esp32_ble_tracker is a bit memory intensive. It's recommended to dedicate this node only for BLE message reception. Using HA API with encryption leaves me with about 75k of free heap memory:

10 pieces of Xiaomi LYWSD03MMC, pvvx fw (all default settings) through one single WirelessTag WT32-ETH01 running ESPHome 2022.3, feeding HA server at a different location:

Before installed them in various places around the house:

Fixing method: 2 pushpins and self-sticking 20x10x2mm neodymium magnet :smiley: @Protoncek @CSchoch

[pvvx]

Tests have shown the ESP32 to be the worst of the few SoCs when running BLE only. When testing, the transfer of the counter of received advertising packets to the UART was used. The tests were carried out with additional battery backup. In addition to passes on the ESP, overloads and other failures very often occur. https://github.com/pvvx/ATC_MiThermometer/issues/59#issuecomment-789520091 PS: And with the new ESP32-C3, it was not possible to achieve the consumption described in the documentation. Chip production technology is too old, which gives excessive consumption not fit for use in stand-alone devices.

[pvvx]

ESP32 fails to create a "cascaded receiver-repeater" with battery power. On cheaper TLSR825x chips, the consumption when accompanied by receiving and transmitting from Xiaomi LYWSD03MMC was 70..75 μA.

An unfolding of some cycles of automatically synchronized transmission-reception of advertisements: Max Cycle: Advertising in BLE is transmitted with random delay insertion to reduce collisions. From this, the receive window has a different duration.

When testing such a cascaded system with automatic transmission-reception synchronization, not a single packet was lost per hour. On newer BLE SoCs, the consumption is even lower and it is possible to use the CR2032 to work for more than 8 months. And what is the purpose of using an ESP32 that requires a 1A power supply? And the consumption of most SoCs in continuous receive mode is only 6 mA.

PS: ESP32 is the last century of electronics... antique thing.

[nagyrobi]

The purpose is to have a lot of BLE sensors around the house to do the measurements. Those are the ones that need to be low power consumption, and battery powered to be able to put them in optimal measurement places.

A bridge's purpose is to gather data from BLE sensors and forward them to the home automation system. Often it has to be in a separate location than the server where the HA system is (server in basement for example, poor RF coverage, not to speak about virtualisation issues etc.). A bridge could be placed in the center of the house and does not need to be battery powered. Just choose a place with good RF sights and wall or PoE power. It will gather BLE data and forward it down to the server, preferably by LAN cable.

I really don't care about how much power eats an ESP32 as a bridge... it's still way less than a raspberry for example, which also can have a lot more other problems on long term like failing SD cards and boot issues after long operation times. Having a full blown linux running up just to gather data from BLE devices and forward them on to another system is an overkill and source for operational troubles on long term.

ESP32 agree it's not perfect, but it's a kind of set it and forget it solution which can be reliable enough for most people, with a much longer lifespan than a full-blown operating system solution will ever have.

Additionally combined with a PoE network switch one can run the entire system from a common managed power source protected with a UPS etc.

[nagyrobi]

ESP32 is the last century of electronics...

I consider this as an advantage because many people know it well, has great open source community support and knowhow is very wide.

Agree that newer chips are more modern and can do much more in a much better way but are still kind of exotic for the majority of the users. It will take more time to get widely supported by open source projects like ESPHome or Tasmota, and by that time they will be also considered antique. It's an endless loop :-)

[pvvx]

Меня действительно не волнует, сколько энергии потребляет ESP32 ... это все же намного меньше, чем, например, малина, у которой также может быть гораздо больше других проблем в долгосрочной перспективе, таких как сбои SD-карт и проблемы с загрузкой после долгого времени работы. Наличие полноценного Linux, работающего только для сбора данных с устройств BLE и пересылки их в другую систему, является излишним и источником операционных проблем в долгосрочной перспективе.

Даже древний RT5350 с активным Ethernet потребляет одинаково с ESP32. Even the ancient RT5350 with active ethernet consumes the same as the ESP32.

1. CPU без MMU не может работать стабильно с динамическим распределением памяти из-за дефрагментации.
2. TCP стек требует буфера в памяти для самого стека и памяти TIME_WAIT структур. Это не менее 0.5 Мегабайта.

BLE тем прорвало, что не требуется поддержка TCP и прочих сетевых протоколов.

1. A CPU without MMU cannot work stably with of heap memory due to defragmentation.
2. The TCP requires an in-memory buffer for the stack itself and the TIME_WAIT memory structure. This is at least 0.5 megabytes.

BLE has however made a breakthrough that does not require support for TCP and multiple network protocols.

ESP32 согласен, что это не идеально, но это своего рода решение «установил и забыл», которое может быть достаточно надежным для большинства людей, с гораздо более длительным сроком службы, чем когда-либо будет иметь полномасштабное решение операционной системы.

Такое понятие, как "Надежность", не рассматривается в Arduino и текущей IoT сфере. Such a concept as "Reliability" is not considered in Arduino and the current IoT field.

Кроме того, в сочетании с сетевым коммутатором PoE можно управлять всей системой от общего управляемого источника питания, защищенного ИБП и т. д.

Это требует перестройку жилых помещений - дополнительную проводку что и составляет большую стоимость такого решения. И часто просто не применимо в концепции «установил и забыл». This requires the restructuring of residential premises - additional wiring, which is the great cost of such a solution. And often it is simply not applicable in the concept of "set it and forget it."

Мой личный опыт показывает, что имея в системе порядка 20 устройств WiFi и 25 BLE, за время их работы более полу года нет ни одного отказа функционирования BLE устройств, но более половины WiFi оказывают каждую неделю. Основная причина – отключение энергии, т.к. проживаю загородом, далеко от поселков и линия электропередачи нестабильна (то падают деревья, то другие напасти). После восстановления электропитания происходят сбои соединения с AP из-за разного времени загрузки и подключения. Часть WiFi устройств так и висит без соединения с AP, пока их не перезапустить руками. Тамагочи, а не помощники... My personal experience shows that with about 20 WiFi devices and 25 BLE devices in the system, during their operation for more than half a year there is not a single failure of the BLE devices, but more than half of the WiFi is provided every week. The main reason is a power outage, because. I live in the countryside, far from the villages and the power line is unstable (either trees fall, or other misfortunes). After power is restored, connection to the AP fails due to different boot and connection times. Some WiFi devices hang without connecting to the AP until they are manually restarted. Tamagotchi, not helpers...

[pvvx]

I consider this as an advantage because many people know it well, has great open source community support and knowhow is very wide.

Agree that newer chips are more modern and can do much more in a much better way but are still kind of exotic for the majority of the users. It will take more time to get widely supported by open source projects like ESPHome or Tasmota, and by that time they will be also considered antique. It's an endless loop :-)

It does nothing for the main audience. Using ESP requires tools and specialized knowledge. ESP cannot be flashed from a smartphone.

[nagyrobi]

Well you probably didn't study deep enough the ESPHome project. Everybody who can deploy a Home Assistant instance by themselves, will be able to auto-generate preconfigured firmwares for any kind of custom ESP solution, one needs just a USB cable (or an USB-TTL adapter at most). Firmware can be flashed directly from the browser.

I'm not talking theoretics, I do have my WT32-ETH01 running and forwarding sensor data from several LYWSD03MMC to Home Assistant as native sensors.

Things I've mentioned do matter for this kind of audience, which is not small at all.

[pvvx]

Well you probably didn't study deep enough the ESPHome project. Everybody who can deploy a Home Assistant instance by themselves, will be able to auto-generate preconfigured firmwares for any kind of custom ESP solution, one needs just a USB cable (or an USB-TTL adapter at most). Firmware can be flashed directly from the browser.

Here I have a ESP32 module from the store. Plugged it into a USB charger. There is a smartphone. How to flash? The ESP community is playing their game and has nothing to do with regular users.

[nagyrobi]

This becomes offtopic ftom the original OP question. I presented a viable solution to @CSchoch and I'm terribly sorry you dont like it - but the fact is, it works and can be implemented relatively simply and cheap.

My post wasn't intended to offend you in any way, please excuse me if I made it feel that way. I'm not contradicting any of your statements.

[pvvx]

I also devoted a lot of time to ESP - more than 5 years. Based on this, I describe the problems that exist there. And the goals are not in the offtopic, but in finding solutions to these issues.

And look around - the main questions and problems occur among ESP users. People who use other means have no questions - you press a couple of buttons on the phone and everything works.

[Protoncek]

Here I have a ESP32 module from the store. Plugged it into a USB charger. There is a smartphone. How to flash?.

Well, if you mean first time - yes, you must connect it via UART and program it. But all later versions go via OTA and thus also via phone - connect to module via IP and upload new version - if that's what you mean. If you mean change only minor things - yes, ESPHome is built in a way that each and every thing requires upload whole bin file again and again... i hate this part, too. Tasmota is more flexible in this regard - it allows many changes via console. But, yes, ESPHome has problems with BLE. But, why tasmota on ESP32 doesn't? I mean - faaaar less... so, it's not entirely hardware's fault...

[nagyrobi]

you press a couple of buttons on the phone and everything works.

I'd be happy if you could post us a link to such a product which can forward BLE data to Home Assistant - nothing more.

[pvvx]

If you dig deeper, then a typical module is sold with "AT" firmware. Includes OTA. But you need to send the command via USB-UART. And in OTA there is a link to the URL, which can be redirected in the program on the smartphone. But this does not exclude the connection with a cable to a smartphone that has the ability to work with USB devices. It is easier for an ordinary programmer to write an analogue of esptool in HTML with javascript. But here, too, the "ESP community" did not show itself.

[nagyrobi]

But here, too, the "ESP community" did not show itself.

@Protoncek I think this shows us this conversation is not objective.

[pvvx]

As a result, the only remaining solution for BLE is the use of RPi and ready-made analogues like "Home Assistant Blue".

[nagyrobi]

Well that's not the category of "you press a couple of buttons on the phone and everything works" lol.

[Protoncek]

As a result, the only remaining solution for BLE is the use of RPi and ready-made analogues like "Home Assistant Blue".

Hm... i guess that it's true if you intend to do more with BLE. However, just for xiaomi temperature and moisture reading my ESP32 with tasmota reads values quite perfectly. No need for anything else. Also, after reboot (or startup of ESP32) all values are shown in a matter of seconds (unlike when using ESPHome, where getting all data could take as long as 5 minutes.

[nagyrobi]

@Protoncek under current ESPHome (2022.3 at time of writing) values show instantly after the node boots up.

[pvvx]

Well that's not the category of "you press a couple of buttons on the phone and everything works" lol.

Isn't there a pre-installed program in these devices? Need something to solder and an extra computer?

There are other options for the price of an ESP32 device - Xiaomi Gateway3. Fully serviced from a smartphone.

[nagyrobi]

No there isn't - on raspberrys at least...

Thanks for the tip, but that costs more than 5 times than an ESP32 Nodemcu, and almost 3 times more than a WT32-ETH01 for this region - but that's again, offtopic.

And there's no open source firmware alternative for it. No chinese cloud managed stuff allowed here, sorry. Has to be able to operate off the cloud from the very first boot. Much more important than servicing from a smartphone.

An RPi is also a solution but a big overkill for just forwarding data.

[pvvx]

And there's no open source firmware alternative for it.

https://github.com/AlexxIT/gw3

[nagyrobi]

Not qualifying: All Xiaomi devices must be linked to an account via the Mi Home app, otherwise they don't send data.

Also it's not a firmware, just an extra binary you can hack onto the device to add some extra functionality.

[nagyrobi]

@pvvx you can try as hard as you want, you won't convince us.

[Protoncek]

@Protoncek under current ESPHome (2022.3 at time of writing) values show instantly after the node boots up.

Is that so? I must test it then. Is there anything that has changed in ESPHome? From my understanding bad drivers are(were?) the main cause for bad working... Regarding xiaomi gateway: i don't know it, but my main condition is also - working LOCALLY. Not with cloud.

[pvvx]

I'm not going to convince. I'm looking for normal solutions, not toys in ESP.

There are many Tuya Smart Plug devices on the market. They have a BK7231x chip installed. It supports BLE and WiFi. Full replacement ESP32 has the ability to build a BLE-WiFi gateway. It is possible to flash from a smartphone. For the mass user, the OTA firmware is not yet available, but publication is possible in the future...

Initial Open Source Integration: [BK7231T] My HTTP server, configurator, MQTT support from Home Assistant

[nagyrobi]

ESPHome project starts to support RP2040. I wouldn't be surprised to cover BK7231x too in the future.

[pvvx]

ESPHome project starts to support RP2040. I wouldn't be surprised to cover BK7231x too in the future.

The ESPHome project should force sellers to sell ESP32 modules with open OTA firmware. And also supply a power supply and a case. Then it might be worth considering.

[nagyrobi]

You're offtopic again.

There are products which come with ESPHome preinstalled. Same with Tasmota. But I wouldn't wanna pay a penny more to have it preinstalled, while I can do it on my own.

And I don't thing forcing anything ever leads to anything good...

[pvvx]

Coercion is due to popularity, price and consumer qualities. ESP32 does not have this today. Only in narrow specific circles, and not for the mass user.

[nagyrobi]

Well that's only how you see it.

[pvvx]

Well that's only how you see it.

I not only see it, but I can participate and correct this process. This is not the first time and there is a lot of experience in such cases. :)

[nagyrobi]

That still doesn't change the fact that using ESP32 as a bridge it is possible, and usable in everyday life.