Closed schlimmchen closed 1 month ago
tobi0171
commented
5 months ago Oh, thank you @schlimmchen for this important and forward-looking information.
Would it be conceivable to also add an example of the pinmap of the ESP32-S3 (44pin) and an adapted frizzing for DIY hobbyists to the wiki? That would probably be very helpful for many who are starting out on the topic and are not sure yet.
Thanks to you, you are ALL doing a really great job.
swingstate
commented
5 months ago That should be possible.
schlimmchen
commented
5 months ago For the record: The limit is hit again after merging the upstream changes from release v24.6.10:
Exception: firmware binary too large: 1969872 > 1966080
Next step: I am going to remove sortables.js from the web application. This removes the ability to sort inverters in the list of inverters (settings and live view). I already know that this is going to carve out around 20k of code (surprisingly much). This gives us headroom once more to merge the upstream changes and leaving around 16k of headroom, sacrificing a feature that hopefully only a very few will miss. My power meter refactoring efforts (ongoing for weeks now) will then use around 11k out of those 16k.
Once those 5k (possibly less) remaining are used up, I don't know how to postpone the inevitable any longer, as I (currently) have no more ideas what to carve out. As I said elsewhere, I am not going to put effort in looking for 500 Bytes here or there so save with hours worth of refactoring.
spcqike
commented
5 months ago As I said elsewhere, I am not going to put effort in looking for 500 Bytes here or there so save with hours worth of refactoring.
that’s totally understandable. Thank you for your effort so far!
One idea: is it possible to store the webviews outside the firmware? Like in AI-on-the-edge, where you put the html into SD card.
I know we don’t have an SD card, but is it possible to put the views in like littleFS filesystem? So that you don’t need to have it twice (in binary firmware and OTA firmware space)?
schlimmchen
commented
5 months ago One idea: is it possible to store the webviews outside the firmware?
That beast is ~200kB. It does not fit the littlefs even if it was empty. There is no other part of the flash we could use. There is no other non-volatile storage.
We could re-partition and designate a new shared partition to hold the web app. That would be a breaking change and require significant effort (different than changing the partition layout for 8MB flash but otherwise keeping things like they are). And it would not last very long. It also creates a new problem: Not only the firmware code must stay within its own bound, but the web app must as well, governed by a different limitation.
I am not convinced we gain anything...
Also: The webapp is tailed to a specific firmware version. The API must be compatible. It makes sense that it is integral part of the firmware.
spcqike
commented
5 months ago Valid points. As I said, just an idea :)
changing to 8MB modules will be easier I guess. That’s why I already have some N16R8 modules on hand. So I’m ready to switch, whenever needed :) (I just need to re-solder everything 🫣)
meisterlampe
commented
5 months ago Well I guess, I have only 4MB Flash, but wanted to check anyway and cannot see it on the page 'System Information'. :) Any ideas, why the information is not shown? I just upgraded to 2024.06.03, deleted the cache, refreshed, tried different browsers and changed the language. I know, there are other ways to get the info. I just want to share the info, that it is not shown in my case. :D
schlimmchen
commented
5 months ago @meisterlampe I am confused... It should be right under the line with the CPU frequency, like this:
You wrote that you cleared the browser cache and refreshed (also with Ctrl + F5?) and tried different browsers. Then I have only one explanation left: you built the firmware yourself and forgot to build the web app. Could that be the case? Otherwise I am puzzled...
schlimmchen
commented
5 months ago I already know that this is going to carve out around 20k of code (surprisingly much).
Well, that was a lie (by the vue vizualizer or whatever the name of that tool was). The actual savings are merely 13k. I tried, and merging the upstream changes as well as adding the power meter refactoring will already fail. That means we can only have one, and to get the other, we need to switch anyways.
Trying not to be a crybaby, but this is becoming a frustrating issue. I don't want to deal with it any more. We had a discussion, we have a solution, I did the announcement. Unfortunately, the implementation will happen much sooner than anticipated:
I will update this issue's description (the announcement) accordingly.
meisterlampe
commented
5 months ago @schlimmchen you've got me, now it is working. Sorry. (sadly 4MB, as expected. Will wait for the OnBattery Pro PCB. But right now everything is working good, so that's ok :) ) Honestly I've never built the webapp before and had no problems. It seems I was always lucky that the webapp_dist was up to date. O:-D
Just to encourage you: Requiring 8MB is the right decision. You said, that the low hanging fruits have been picked. If saving some bytes means way too much effort, just to postpone the inevitable, it makes no sense to ride the dead horse (and it also makes no fun :) )
schlimmchen
commented
5 months ago Honestly I've never built the webapp before and had no problems.
Have you switched to OpenDTU-OnBattery only recently? OpenDTU has up-to-date blobs of the web app in its repo. As I am ignorant and stubborn, I won't do that for OpenDTU-OnBattery. Binary blobs shall not be part of a source code repo, unless they are not an (intermediate) result of building something (manuals, external proprietary firmware, etc.).
Just to encourage you: Requiring 8MB is the right decision. You said, that the low hanging fruits have been picked. If saving some bytes means way too much effort, just to postpone the inevitable, it makes no sense to ride the dead horse (and it also makes no fun :) )
Thanks for understanding!
Gumbagubanga
commented
5 months ago Going 8 MB is a logical choice.
Nonetheless it saddens me as I'm running a WT32-ETH01 for the ethernet port even thou I've pretty much ran out of GPIO pins immediately. Are there any known (affordable) boards with supported ethernet and at least 8 MB flash?
schlimmchen
commented
5 months ago I had yet another idea about how to continue support for devices with 4MB of flash memory. AFAIK we have not discussed it yet. Just now I completed some tests and it seems that this would fly:
We can introduce a new platformio environment, probably named generic_esp32_4mb_no_ota. The second app partition will be removed from the partition table. The freed space is allocated to the remaining single app partition, doubling it in size just like we want to. OTA updates will not be possible when using the firmware build for this environment. The firmware will detect this and the web UI will display a respective message when navigating to the "Firmware Upgrade" view. These devices can only be upgraded by wire. For this reason, there will only be a "factory" binary for this environment (at least as part of the release).
This change is still a breaking change, as the partition layout must be updated. However, once switched to this new type of release, users with ESP32 with only 4MB can continue using their device, with the major caveat that updates are only possible by wire going forward. Users who find this unacceptable need to upgrade to an ESP32(-S3) with 8MB+ flash.
Opinions?
swingstate
commented
5 months ago Settings would still be preserved?
schlimmchen
commented
5 months ago Settings would still be preserved?
Yes, the filesystem partition is still left as it is, so settings are preserved when writing the new factory bin using esptool.
CKone01
commented
5 months ago Excellent workaround instead of scrapping usable devices!
helgeerbe
commented
5 months ago Hi @schlimmchen, I would be OK to flash esp32 in future by wire. But not everybody has easy access to his openDTU-onBattery.
If we go the way forward to have a new plattformio environment. I still think, it could make sense to have a small ota firmware. Just thinking while writing:
Don't know how big this image would be, but I guess this brings us a lot of free memory.
Actually we don't make really use of the cold standby image (no fallback strategy on errors to the previous image). So in my point of view, its just a waste of resources.
We can discuss this within the upstream project. Thomas is facing the same situation more sooner than later.
schlimmchen
commented
5 months ago But not everybody has easy access to his openDTU-onBattery.
I can see that. Those people hopefully are easily convinced to upgrade to an ESP32 with 8 MB of flash to keep the convenience of OTA updates, at least eventually. My idea to provide the non-OTA firmware is merely an attempt to ease the transition for people with 4 MB devices. In the long run I assume that every OpenDTU-OnBattery user switches to an ESP32 with at least 8 MB of flash: Those who start fresh will know to obtain such a device in the first place, and many existing users will miss the OTA functionality, so they'll upgrade, or they are fine updating by wire from time to time, so no OTA update functionality is need for them either.
The process you describe using a minimal firmware makes sense. However, let me iterate disadvantages and problems, partially copied from https://github.com/helgeerbe/OpenDTU-OnBattery/issues/935#issuecomment-2085149009:
#ifdef "everywhere" in the code. These will not only guard whole files, but blocks of code, and also use alternatives, like MessageOutput will only be available in the main firmware, while we must use Serial in the minimal firmware to save space. (2) Maintain a whole different branch and build the minimal firmware from there. (1) is bad because it clutters the code and because it does not work for the web app -- we would still need a standalone second web app. (2) is bad because changes in the main firmware need to be backported into the minimal firmware.It is doable, I guess. Is it worth the headaches: I say that it isn't. It's also not worth the risk that the whole approach simply does not work as expected, after investing dozens of hours into this.
There are inconveniences for the users, too: OTA updates will be a multi-step procedure using this approach, rather than a one-click operation.
How do you @helgeerbe feel about this approach after reading my arguments?
I was unsure about whether or not I should bring this up in the upstream project. As you think so as well, I started a discussion at tbnobody#2081 and asked for Thomas comment.
schlimmchen
commented
5 months ago I spent another hour exploring. This time I went the other way around: I created a new branch based on the current development branch and stripped everything that I could. I managed to build and run this minimal firmware. However, it is 1160kB. This is much larger than my previous estimate. The web app is not yet stripped down, so let's assume the firmware will be 1060kB. Still more than a Megabyte.
Things that I realized cannot be removed and were not accounted for in yesterdays estimate: (1) Ethernet Support: If we provide such a minimal firmware, it also needs to support existing ESP32 that are connected via Ethernet. (2) PinMapping support: required for Ethernet.
My best guess for the larger size compared to yesterdays estimate is that this time, functions that are actually needed to provide the intended functionality, are now included, whereas yesterday I merely tried to tickle the respective lib by using a single function of it.
To implement this breaking change, users still need to update their partition layout using a wired connection. We need to maintain the minimal firmware, invest more time to make it at least a little smaller, and we don't gain very much for the main firmware partition, maybe ~500kB. I still think this is not worth the effort. @helgeerbe Let us know what you think.
spcqike
commented
5 months ago (2) PinMapping support: required for Ethernet.
is this really true? I mean, someone who wants to upgrade an existing firmware, already has a working pinmapping. So there is no need for the full code that belongs to pinmapping itself. As we don’t need to use anything out of it expect the pins for Ethernet.
Would it be possible to like store the Ethernet pins in a different file if and only if Ethernet is used? In the main firmware.
So if minimal firmware boots it only checks whether or not this Ethernet file exists and if so, it loads it. If it doesn’t exist, it will use WiFi. So the minimal firmware wouldn’t need to handle or read everything related to pinmapping.
Idk but tasmotas minimal firmware is just about 259Kb. It doesn’t come with a fancy web interface where you can setup something. Everything you can do is upload another firmware. So it kind of only uses wifi with a very simple and limited html server file and OTA handling.
If we consider dynamic Ethernet ports (meaning they can be connected to whatever pin the user defines), we would need to check for them and their pins, too. So we would need wifi, littlefs, Ethernet and OTA, wouldn’t we?
schlimmchen
commented
5 months ago Not sure that I understand. The code to be able to read a file from the littlefs partition and to interpret it (JSON) has to be there, regardless of how much data is in it (all pins used by the main firmware or just the ones for Ethernet). I was not talking about support for editing the pin mapping, neither to edit the config, in the minimal firmware. Those would be read-only in the minimal firmware that I explored. The config must be read, however, to be able to connect to the configured Wi-Fi network, or to configure for Ethernet, etc.
Have a look in the respective branch.
I have been thinking about this for months now, as I anticipated this issue a long time ago, and I was so happy to have found support to justify the switch to 8 MB devices, and even happier to find that 4 MB devices could be supported when sacrificing OTA updates -- with very little effort. Now the discussion about the minimal firmware approach is starting anew, after I published an announcement and people started buying new hardware and upgrading their setups... I invested a couple of hours into exploration of something that I don't even support, in order to come up with a good estimate of how much flash the "minimal" firmware would need. Now I am spending time to argue with this:
Idk but tasmotas minimal firmware is just about 259Kb
I don't know where you found that number and I did not check how large the Tasmota minimal firmware for "legacy" ESP (ESP8266) is, but the Tasmota safeboot firmware for ESP32 (non-S3) is 813kB:
The safeboot partition is 832kB, see the docs. This is very close to my estimate of 875kB above.
Moreover, I think that a comparison with Tasmota, at least regarding firmware update strategies, makes no sense. Tasmota devices are buried behind some physical switch or inside lamps or something. And most of them don't even have a port to connect to for firmware updates. Also, those devices cannot just be upgraded to an ESP32-S3 with 8MB. The requirement for supporting OTA updates on these devices is obvious. I would never argue about that. I wouldn't use Tasmota on my many ESP devices if I had to dig them up and solder wires to them (Shelly mini series) every time I wanted to perform an update. It makes perfect sense to me that people invest time in having and maintaining the Tasmota safeboot firmware.
The situation for OpenDTU-OnBattery, however, is very different. So different, that I think OTA updates are a convenience, not a requirement. And users may keep this convenience, they merely have to install a different MCU. It's a one-time effort. And as other people stated: 30 bugs for a Fusion board is justifiable given that the cost for all the other parts of a small solar plant are so much higher.
Given that, I still want to have Helge on board and feeling good about how we move forward. And he was right to suggest that we ask upstream for an opinion. Even though I was about to move forward, we will wait for their input on this matter.
schlimmchen
commented
5 months ago @spcqike is right, we can just serve static (or mostly static) HTML pages in the minimal firmware. The minimal firmware will have a completely different look, it will be ugly, and there will be no translations (at least no dynamic ones). Should be okay for something one only sees when performing an OTA update. That should bring us down to ~960kB for the minmal firmware.
I think we should add some sort of API between the main firmware and the minimal firmware, i.e., the minimal firmware uses its own (very limited) set of settings, and each time an OTA update is triggered, the main firmware constructs this from the active config. This allows to change the main config without the need to keep compatible with the minimal firmware. The only thing that might need an update in such a case is the function writing/updating the minimal firmware config. Also, getting rid of ArduinoJson should save code (roughly 33k as estimated earlier).
spcqike
commented
5 months ago Also, getting rid of ArduinoJson should save code (roughly 33k as estimated earlier).
That’s what I thought when writing
Would it be possible to like store the Ethernet pins in a different file if and only if Ethernet is used? In the main firmware.
So to have a file like „Ethernet.conf“ wich is just one like of comma separated PIN numbers (like „12,4,15,16“ or whatever pins are used for Ethernet) this file could be created by pinmapping loading process and holds the Ethernet pins in a certain order. Of course only if Ethernet is used 😄
we could also use some EEPROM bytes to hold these pins. If eeprom handling uses less Programm size than LittleFS, idk. (Even tho at the end eeprom is also just emulated in flash)
But you wrote we need the config file and therefore JSON anyway, as it holds the wifi credentials.
Snoopy-HSS
commented
5 months ago I try to compile the current development and it says: raise Exception("firmware binary too large: %d > %d" % (fw_size, max_size)) Exception: firmware binary too large: 1970624 > 1966080
is it already over with ESP32_generic 4MB ????
schlimmchen
commented
5 months ago But you wrote we need the config file and therefore JSON anyway, as it holds the wifi credentials.
We shall use the approach to translate current settings for the minimal firmware with both the pin mapping (the part which is required to setup Ethernet) as well as critical config values, so no JSON needs to be parsed.
is it already over with ESP32_generic 4MB ????
Well, yes. You currently cannot compile firmware for generic or generic_esp32 or any other ESP32 (non-S3) based on the current development branch. The partition layout must be updated to make more space for the sketch, one way or another. For that reason I disabled those environment in the github action builds.
Since you are compiling yourself, here is the simplest approach you can take: Update partitions_custom_4mb.csv to look like this (remove partition app1 and double app0 in size):
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, 0x10000, 0x3C0000,
spiffs, data, spiffs, 0x3D0000, 0x30000,Compile again and flash using esptool over USB. OTA is broken, continue updating only using esptool.
Other than that you can start stripping out pieces of code that you are not using anyways, like the AC charger code, or whatever you find is not used in your setup.
Snoopy-HSS
commented
5 months ago It's weird. When I compile for the S3 size is smaller !? why is there a 2MB limit when i use the generic_S3 ???
Checking size .pio\build\generic_esp32s3\firmware.elf Advanced Memory Usage is available via "PlatformIO Home > Project Inspect" RAM: [== ] 24.0% (used 78680 bytes from 327680 bytes) Flash: [==========] 95.9% (used 1886133 bytes from 1966080 bytes) Building .pio\build\generic_esp32s3\firmware.bin
In Platform.ini default ENV is still generic_ESP32
Changing partition worked, can also be uploaded Serial via, visual Studio code
schlimmchen
commented
5 months ago It's weird. When I compile for the S3 size is smaller !?
Yes, as the S3 is a slightly different processor with a different instruction set, so the code is can be smaller (more efficient instructions, etc.).
why is there a 2MB limit when i use the generic_S3 ???
I don't know what you mean. The partition table is the same for ESP32-S3 and non-S3.
Snoopy-HSS
commented
5 months ago Flash: [==========] 95.9% (used 1886133 bytes from 1966080 bytes)
Only a little under 2MB availabe. How can i change, that it uses 16MB filesystem when compiling generic_esp32s3
is this part for all boards? board_build.partitions = partitions_custom_4mb.csv
it's set in platform.ini to 4MB and ESP32 per default...
greymda
commented
4 months ago I had yet another idea about how to continue support for devices with 4MB of flash memory. AFAIK we have not discussed it yet. Just now I completed some tests and it seems that this would fly:
We can introduce a new platformio environment, probably named
generic_esp32_4mb_no_ota. The second app partition will be removed from the partition table. The freed space is allocated to the remaining single app partition, doubling it in size just like we want to. OTA updates will not be possible when using the firmware build for this environment. The firmware will detect this and the web UI will display a respective message when navigating to the "Firmware Upgrade" view. These devices can only be upgraded by wire. For this reason, there will only be a "factory" binary for this environment (at least as part of the release).This change is still a breaking change, as the partition layout must be updated. However, once switched to this new type of release, users with ESP32 with only 4MB can continue using their device, with the major caveat that updates are only possible by wire going forward. Users who find this unacceptable need to upgrade to an ESP32(-S3) with 8MB+ flash.
Opinions?
is this option still on the table? @schlimmchen
schlimmchen
commented
4 months ago @greymda This is already implemented and I don't see any reason why this variant of the firmware should go away. There is no release with this variant, yet, but build runs already build this kind of firmware, named as suggested.
greymda
commented
4 months ago @greymda This is already implemented and I don't see any reason why this variant of the firmware should go away. There is no release with this variant, yet, but build runs already build this kind of firmware, named as suggested.
great news! so my 4mb board still has some bright days ahead!
futzicomp
commented
4 months ago Hallo zusammen,
Nochmals vielen Dank für euren Einsatz! Die Aussicht sinnvolle Funktionserweiterungen zukünftig erhalten zu können ohne an die Grenze des ESP mehr zu stoßen lassen mich den Umstieg auf ein ESP32S3 locker verkraften!
Ich habe einen ES32S3 N16R8 bei mir rumliegen. Spaßeshalber habe ich 2024.06.03 aufgespielt und bin etwas verwirrt was die PSRAM Größe angeht.
In der Info Anzeige wird 16MB Flash aber 0MB PSRAM angezeigt, obwohl dieser Chip 16/8MB haben sollte. Kann ich mich auf die Anzeige N16R8 auf dem Chip oder auf die Anzeige von Flash und PSRAM Speicher in der Info verlassen? (habe ich ein Fake Exemplar erwischt?)
Danke Uli
schlimmchen
commented
4 months ago Die Unterstützung für PSRAM muss ausdrücklich eingeschaltet werden. Dafür braucht es dann eine zusätzliche Firmware-Variante. Eine solche könnten wir demnächst mal einführen. Beispiel:
Allerdings bin ich mir der Risiken nicht bewusst. Vielleicht gibt es einen Grund, warum tbnobody noch keine solchen Images anbietet. Außerdem ist ja "die Wahrheit", dass bisher kein PSRAM gebraucht wird, wir kommen ja bisher gut mit dem integrierten RAM aus.
futzicomp
commented
4 months ago Thanks for your answer and again:thanks for your good work! My system is now working flawlessly since one year(!) I am looking forward for cool new developments which are possible with the upgraded Hardware 🙂
schwatter
commented
4 months ago @schlimmchen Tag, meine 8mb Flashsize Esp's sind im Zulauf. Nu dachte ich, bis dahin spiele ich mit meinen 4m Flashsize Esp's. Aber da bootet OpenDTU on Battery nicht. Bzw nur bis zum Release 2023.11.15. Das ist 1.74mb groß. Alles danach macht Probleme. EspHomeFlasher schreibt aber, mein Flash wäre 4mb groß. Kann mich wer erhellen. Falsche angaben? fake Chip?
`Using 'COM5' as serial port. Connecting..... Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting....... Detecting chip type... ESP32 Connecting.....
Chip Info:
Gruß schwatter
spcqike
commented
4 months ago @schwatter das „Problem“ ist die partitionierung. OpenDTU hält eine zweite 2MB Partition für das OTA Update vor. Daher kannst du nur die Hälfte benutzen und auch das Image ist <2MB.
Du kannst hier die non OTA Version nehmen. https://github.com/helgeerbe/OpenDTU-OnBattery/actions/runs/10180804929
schwatter
commented
4 months ago Ah, danke! Vielen Dank für die Erklärung und den Link. Die hatte ich nicht gesehen. Sind die Non-Ota auch irgendwo verlinkt? Oder wo sind die beschrieben?
edit: Die möchte EspHomeFlasher nicht flashen: `Using 'COM5' as serial port. Connecting.... Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting.... Detecting chip type... ESP32 Connecting....
Chip Info:
Gruß schwatter
schwatter
commented
4 months ago Ok, unter Linux mit esptool hat es geklappt.
`root@pi3Pihole:/tmp# esptool.py -p /dev/ttyUSB0 -b 230400 write_flash --flash_size detect --flash_freq 40m 0x0 opendtu-onbattery-generic_esp32_4mb_no_ota.factory.bin esptool.py v4.7.0 Serial port /dev/ttyUSB0 Connecting.... Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting.... Detecting chip type... ESP32 Chip is ESP32-D0WD-V3 (revision v3.1) Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None Crystal is 40MHz MAC: a0:b7:65:4e:f9:dc Uploading stub... Running stub... Stub running... Changing baud rate to 230400 Changed. Configuring flash size... Auto-detected Flash size: 4MB Flash will be erased from 0x00000000 to 0x001fafff... Compressed 2073312 bytes to 1292183... Wrote 2073312 bytes (1292183 compressed) at 0x00000000 in 59.5 seconds (effective 278.7 kbit/s)... Hash of data verified.
Leaving... Hard resetting via RTS pin... root@pi3Pihole:/tmp#`
Danke und Gruß
schwatter
Gumbagubanga
commented
4 months ago Going 8 MB is a logical choice.
Nonetheless it saddens me as I'm running a WT32-ETH01 for the ethernet port even thou I've pretty much ran out of GPIO pins immediately. Are there any known (affordable) boards with supported ethernet and at least 8 MB flash?
Update: I bought an Lilygo T-ETH-Lite S3 board (16 MB Flash, 8 MB PSRAM) with an currently unsupported W5500 ethernet controller. Luckily in an upstream PR https://github.com/tbnobody/OpenDTU/pull/2068 an initial support has been implemented. As @marvincarstensen mentioned this project directly, I'm positive that he tested it against the latest OpenDTU-OnBattery too and we'll finally get ethernet support back on ESP32-S3 boards 👍
Dani1802
commented
3 months ago If anyone needs a case, I remixed an existing one to fit the WROOM-1-N16R8 ESP32-S3-DevKitC-1. It can be found here: https://www.thingiverse.com/thing:6734089
schlimmchen
commented
3 months ago @Dani1802 Thanks, linked in the docs.
schlimmchen
commented
1 month ago We published two releases which implement the changes outlined here. This issue/announcement has been stale for almost two months now, so I will close this issue now. Please open a new issue in case of problems regarding the changed partition layout.
TL;DR
Flash Size Constraints
The flash memory of your ESP32 is partitioned. The legacy partition layout allocates a little less than 2 MB for one copy of the firmware. There is room for a second copy of the firmware, allowing over-the-air (OTA) updates. Overall, the partition layout is currently designed to fit into 4 MB of flash memory.
As OpenDTU-OnBattery is based on OpenDTU, this layout is inherited from said upstream project. OpenDTU-OnBattery is kept from evolving due to the partition's size constraints. Many additional features (compared to the original OpenDTU) require a siginificant amount of additional binary code, which needs space in flash memory.
Short-Term Mitigations
We actually already hit the limit with version 2023.05.03 and had to employ creative solutions to restore compatibility with the existing flash layout. See #958 (and #950) for details.
We might be able to continue reducing the binary firmware image size slightly. However, note that the low-hanging fruit has already been harvested. Keeping compatibility with 4 MB flash will become a struggle in the near future.Long-Term Solution
As discussed in #935, where alternate solutions were also discussed, we will introduce a breaking change with the next release, which redesigns the partition layout for 8 MB (and larger) flash memories. This doubles the amount of space available for the firmware binary, allowing us to continue adding features to the project. Doubling the firmware partitions in size will give headroom for the foreseeable future.
You will then need an ESP32 with at least 8 MB of flash memory in order to install and run OpenDTU-OnBattery.
Alternatively, you can kepp your device with 4MB of flash but have to forfeit OTA updates. This means you will need to upgrade using a new
opendtu-onbattery-generic_esp32_4mb_no_ota.factory.binversion of the firmware and will be constraint to wired updates (usingesptoolor similar tools to flash the binary using the USB connection to your ESP32 board).Timeline
Version 2024.06.03 is the last version to support the legacy partition layout, i.e., ESP32 with 4 MB of flash memory.
We are unable to keep up with the upstream changes in v24.6.10, as adding those increases the firmware size beyond the allowable maximum.
We ran out of options to reduce the firmware size further with an acceptable amount of effort.
There is no specific timeline for the introduction of the breaking change.The project is continuing to evolve and new features are added. Sometimes even fixes add more code to the binary. Merging changes from upstream, including updated libraries, bears a risk of enlarging the firmware binary in size. We might put effort into keeping 4 MB flash devices supported for a while, if the work to do so is managable.Some day in the near future (weeks, maybe a few months), the firmware blob will be too big for the existing partition layout, possibly despite efforts to keep the firmware size at bay.Actions Required Now
Determine Hardware Upgrade Requirement
Determine whether or not you want or need to upgrade your ESP32.
This can be tricky. The variety of ESP32 in the wild can make it hard to pin down the exact ESP32 model and/or board and hence the amount of flash your ESP32 has available.
You do not need an upgrade if you use an OpenDTU Fusion board.
Option 1: Perform Firmware Upgrade
Install OpenDTU-OnBattery 2024.06.03 and navigate to the system's information page of the web UI. It lists the flash memory size in section "Hardware Information". This information is expected to be reliable.
Option 2: Avoid Firmware Upgrade
Option 2.1: Examine Hardware
Have a look at your hardware and note the exact model name engraved on the ESP32 module's RF shield. Consult the appropriate datasheet to find out how much flash your module should have available.
This method leaves significant room for errors. Custom chips/modules are hard to identify.
Option 2.2: Use
esptoolConnect your ESP32 board to a computer with
esptoolinstalled and executeesptool --port <serial_port> flash_id(where<serial_port>often is/dev/ttyUSB0or/dev/ttyACM0on Linux orCOMn(ninteger) on Windows).The output contains a line
Detected flash size: <?>MB. This information is expected to be reliable.Executing this command does not alter memory. The board can be put back into production with a power-on reset.
Select an Upgrade
Depending on your overall setup, selecting a replacement ESP32 (board) can be simple. The "NodeMCU" or "Espressif DevKitC" style development boards are available in many different configurations. If you use one of those and chose to use a pluggable connection to peripherals, in particular to connect to the RF module(s), a replacement development/prototype board could be a good option.
For custom setups or those using soldered connections, you might need to consider replacing more than just the ESP32 board.
OpenDTU Fusion
Consider an OpenDTU Fusion board to be your next driver for OpenDTU-OnBattery. It ships with an ESP32-S3 with (at least) 8 MB of flash memory and (at least) 2 MB of PSRAM. Support for the OpenDTU Fusion board is very likely to be sustained for a long time. Additional hardware to connect peripherals which are typically in use by OpenDTU-OnBattery users (CAN transceiver, 2 ADUM1201 isolators for serial connections (VE.Direct)) is expected to be available in June 2024, while a limited number of working prototypes with a single ADUM1201 are already available.
OnBattery Pro PCB
A community member (@swingstate) is working on making a new board ready for production, which provides ready-to-use ports for typical OpenDTU-OnBattery peripherals. It features an ESP32-S3 with 16 MB of flash and 8 MB PSRAM. If you can wait for the release of this product, you might want to consider it. See #541 and a comment in #935 for details.
Other Boards
See recommendations in the docs.
Resoldering an ESP32 module
If you know how to remove the ESP32 module from your board and solder a new one in, you know what you are doing and you should be good to go to select a suitable ESP32 module.
Actions Required to Upgrade
See the upgrade guide.
Questions and Answers
Can I keep running my OpenDTU-OnBattery using Firmware 2024.06.03 or older?
If you are happy with the way OpenDTU-OnBattery performs, you may very well keep your setup as it is.