Open schlimmchen opened 3 weeks ago
tobi0171
commented
2 weeks ago Oh, thank you @schlimmchen for this important and forward-looking information.
Would it be conceivable to also add an example of the pinmap of the ESP32-S3 (44pin) and an adapted frizzing for DIY hobbyists to the wiki? That would probably be very helpful for many who are starting out on the topic and are not sure yet.
Thanks to you, you are ALL doing a really great job.
swingstate
commented
2 weeks ago That should be possible.
schlimmchen
commented
1 week ago For the record: The limit is hit again after merging the upstream changes from release v24.6.10:
Exception: firmware binary too large: 1969872 > 1966080
Next step: I am going to remove sortables.js from the web application. This removes the ability to sort inverters in the list of inverters (settings and live view). I already know that this is going to carve out around 20k of code (surprisingly much). This gives us headroom once more to merge the upstream changes and leaving around 16k of headroom, sacrificing a feature that hopefully only a very few will miss. My power meter refactoring efforts (ongoing for weeks now) will then use around 11k out of those 16k.
Once those 5k (possibly less) remaining are used up, I don't know how to postpone the inevitable any longer, as I (currently) have no more ideas what to carve out. As I said elsewhere, I am not going to put effort in looking for 500 Bytes here or there so save with hours worth of refactoring.
spcqike
commented
1 week ago As I said elsewhere, I am not going to put effort in looking for 500 Bytes here or there so save with hours worth of refactoring.
that’s totally understandable. Thank you for your effort so far!
One idea: is it possible to store the webviews outside the firmware? Like in AI-on-the-edge, where you put the html into SD card.
I know we don’t have an SD card, but is it possible to put the views in like littleFS filesystem? So that you don’t need to have it twice (in binary firmware and OTA firmware space)?
schlimmchen
commented
1 week ago One idea: is it possible to store the webviews outside the firmware?
That beast is ~200kB. It does not fit the littlefs even if it was empty. There is no other part of the flash we could use. There is no other non-volatile storage.
We could re-partition and designate a new shared partition to hold the web app. That would be a breaking change and require significant effort (different than changing the partition layout for 8MB flash but otherwise keeping things like they are). And it would not last very long. It also creates a new problem: Not only the firmware code must stay within its own bound, but the web app must as well, governed by a different limitation.
I am not convinced we gain anything...
Also: The webapp is tailed to a specific firmware version. The API must be compatible. It makes sense that it is integral part of the firmware.
spcqike
commented
1 week ago Valid points. As I said, just an idea :)
changing to 8MB modules will be easier I guess. That’s why I already have some N16R8 modules on hand. So I’m ready to switch, whenever needed :) (I just need to re-solder everything 🫣)
meisterlampe
commented
1 week ago Well I guess, I have only 4MB Flash, but wanted to check anyway and cannot see it on the page 'System Information'. :) Any ideas, why the information is not shown? I just upgraded to 2024.06.03, deleted the cache, refreshed, tried different browsers and changed the language. I know, there are other ways to get the info. I just want to share the info, that it is not shown in my case. :D
schlimmchen
commented
1 week ago @meisterlampe I am confused... It should be right under the line with the CPU frequency, like this:
You wrote that you cleared the browser cache and refreshed (also with Ctrl + F5?) and tried different browsers. Then I have only one explanation left: you built the firmware yourself and forgot to build the web app. Could that be the case? Otherwise I am puzzled...
schlimmchen
commented
1 week ago I already know that this is going to carve out around 20k of code (surprisingly much).
Well, that was a lie (by the vue vizualizer or whatever the name of that tool was). The actual savings are merely 13k. I tried, and merging the upstream changes as well as adding the power meter refactoring will already fail. That means we can only have one, and to get the other, we need to switch anyways.
Trying not to be a crybaby, but this is becoming a frustrating issue. I don't want to deal with it any more. We had a discussion, we have a solution, I did the announcement. Unfortunately, the implementation will happen much sooner than anticipated:
I will update this issue's description (the announcement) accordingly.
meisterlampe
commented
1 week ago @schlimmchen you've got me, now it is working. Sorry. (sadly 4MB, as expected. Will wait for the OnBattery Pro PCB. But right now everything is working good, so that's ok :) ) Honestly I've never built the webapp before and had no problems. It seems I was always lucky that the webapp_dist was up to date. O:-D
Just to encourage you: Requiring 8MB is the right decision. You said, that the low hanging fruits have been picked. If saving some bytes means way too much effort, just to postpone the inevitable, it makes no sense to ride the dead horse (and it also makes no fun :) )
schlimmchen
commented
1 week ago Honestly I've never built the webapp before and had no problems.
Have you switched to OpenDTU-OnBattery only recently? OpenDTU has up-to-date blobs of the web app in its repo. As I am ignorant and stubborn, I won't do that for OpenDTU-OnBattery. Binary blobs shall not be part of a source code repo, unless they are not an (intermediate) result of building something (manuals, external proprietary firmware, etc.).
Just to encourage you: Requiring 8MB is the right decision. You said, that the low hanging fruits have been picked. If saving some bytes means way too much effort, just to postpone the inevitable, it makes no sense to ride the dead horse (and it also makes no fun :) )
Thanks for understanding!
Gumbagubanga
commented
1 week ago Going 8 MB is a logical choice.
Nonetheless it saddens me as I'm running a WT32-ETH01 for the ethernet port even thou I've pretty much ran out of GPIO pins immediately. Are there any known (affordable) boards with supported ethernet and at least 8 MB flash?
schlimmchen
commented
6 days ago I had yet another idea about how to continue support for devices with 4MB of flash memory. AFAIK we have not discussed it yet. Just now I completed some tests and it seems that this would fly:
We can introduce a new platformio environment, probably named generic_esp32_4mb_no_ota. The second app partition will be removed from the partition table. The freed space is allocated to the remaining single app partition, doubling it in size just like we want to. OTA updates will not be possible when using the firmware build for this environment. The firmware will detect this and the web UI will display a respective message when navigating to the "Firmware Upgrade" view. These devices can only be upgraded by wire. For this reason, there will only be a "factory" binary for this environment (at least as part of the release).
This change is still a breaking change, as the partition layout must be updated. However, once switched to this new type of release, users with ESP32 with only 4MB can continue using their device, with the major caveat that updates are only possible by wire going forward. Users who find this unacceptable need to upgrade to an ESP32(-S3) with 8MB+ flash.
Opinions?
swingstate
commented
6 days ago Settings would still be preserved?
schlimmchen
commented
6 days ago Settings would still be preserved?
Yes, the filesystem partition is still left as it is, so settings are preserved when writing the new factory bin using esptool.
CKone01
commented
5 days ago Excellent workaround instead of scrapping usable devices!
helgeerbe
commented
5 days ago Hi @schlimmchen, I would be OK to flash esp32 in future by wire. But not everybody has easy access to his openDTU-onBattery.
If we go the way forward to have a new plattformio environment. I still think, it could make sense to have a small ota firmware. Just thinking while writing:
Don't know how big this image would be, but I guess this brings us a lot of free memory.
Actually we don't make really use of the cold standby image (no fallback strategy on errors to the previous image). So in my point of view, its just a waste of resources.
We can discuss this within the upstream project. Thomas is facing the same situation more sooner than later.
schlimmchen
commented
5 days ago But not everybody has easy access to his openDTU-onBattery.
I can see that. Those people hopefully are easily convinced to upgrade to an ESP32 with 8 MB of flash to keep the convenience of OTA updates, at least eventually. My idea to provide the non-OTA firmware is merely an attempt to ease the transition for people with 4 MB devices. In the long run I assume that every OpenDTU-OnBattery user switches to an ESP32 with at least 8 MB of flash: Those who start fresh will know to obtain such a device in the first place, and many existing users will miss the OTA functionality, so they'll upgrade, or they are fine updating by wire from time to time, so no OTA update functionality is need for them either.
The process you describe using a minimal firmware makes sense. However, let me iterate disadvantages and problems, partially copied from https://github.com/helgeerbe/OpenDTU-OnBattery/issues/935#issuecomment-2085149009:
#ifdef "everywhere" in the code. These will not only guard whole files, but blocks of code, and also use alternatives, like MessageOutput will only be available in the main firmware, while we must use Serial in the minimal firmware to save space. (2) Maintain a whole different branch and build the minimal firmware from there. (1) is bad because it clutters the code and because it does not work for the web app -- we would still need a standalone second web app. (2) is bad because changes in the main firmware need to be backported into the minimal firmware.It is doable, I guess. Is it worth the headaches: I say that it isn't. It's also not worth the risk that the whole approach simply does not work as expected, after investing dozens of hours into this.
There are inconveniences for the users, too: OTA updates will be a multi-step procedure using this approach, rather than a one-click operation.
How do you @helgeerbe feel about this approach after reading my arguments?
I was unsure about whether or not I should bring this up in the upstream project. As you think so as well, I started a discussion at tbnobody#2081 and asked for Thomas comment.
schlimmchen
commented
4 days ago I spent another hour exploring. This time I went the other way around: I created a new branch based on the current development branch and stripped everything that I could. I managed to build and run this minimal firmware. However, it is 1160kB. This is much larger than my previous estimate. The web app is not yet stripped down, so let's assume the firmware will be 1060kB. Still more than a Megabyte.
Things that I realized cannot be removed and were not accounted for in yesterdays estimate: (1) Ethernet Support: If we provide such a minimal firmware, it also needs to support existing ESP32 that are connected via Ethernet. (2) PinMapping support: required for Ethernet.
My best guess for the larger size compared to yesterdays estimate is that this time, functions that are actually needed to provide the intended functionality, are now included, whereas yesterday I merely tried to tickle the respective lib by using a single function of it.
To implement this breaking change, users still need to update their partition layout using a wired connection. We need to maintain the minimal firmware, invest more time to make it at least a little smaller, and we don't gain very much for the main firmware partition, maybe ~500kB. I still think this is not worth the effort. @helgeerbe Let us know what you think.
spcqike
commented
4 days ago (2) PinMapping support: required for Ethernet.
is this really true? I mean, someone who wants to upgrade an existing firmware, already has a working pinmapping. So there is no need for the full code that belongs to pinmapping itself. As we don’t need to use anything out of it expect the pins for Ethernet.
Would it be possible to like store the Ethernet pins in a different file if and only if Ethernet is used? In the main firmware.
So if minimal firmware boots it only checks whether or not this Ethernet file exists and if so, it loads it. If it doesn’t exist, it will use WiFi. So the minimal firmware wouldn’t need to handle or read everything related to pinmapping.
Idk but tasmotas minimal firmware is just about 259Kb. It doesn’t come with a fancy web interface where you can setup something. Everything you can do is upload another firmware. So it kind of only uses wifi with a very simple and limited html server file and OTA handling.
If we consider dynamic Ethernet ports (meaning they can be connected to whatever pin the user defines), we would need to check for them and their pins, too. So we would need wifi, littlefs, Ethernet and OTA, wouldn’t we?
schlimmchen
commented
3 days ago Not sure that I understand. The code to be able to read a file from the littlefs partition and to interpret it (JSON) has to be there, regardless of how much data is in it (all pins used by the main firmware or just the ones for Ethernet). I was not talking about support for editing the pin mapping, neither to edit the config, in the minimal firmware. Those would be read-only in the minimal firmware that I explored. The config must be read, however, to be able to connect to the configured Wi-Fi network, or to configure for Ethernet, etc.
Have a look in the respective branch.
I have been thinking about this for months now, as I anticipated this issue a long time ago, and I was so happy to have found support to justify the switch to 8 MB devices, and even happier to find that 4 MB devices could be supported when sacrificing OTA updates -- with very little effort. Now the discussion about the minimal firmware approach is starting anew, after I published an announcement and people started buying new hardware and upgrading their setups... I invested a couple of hours into exploration of something that I don't even support, in order to come up with a good estimate of how much flash the "minimal" firmware would need. Now I am spending time to argue with this:
Idk but tasmotas minimal firmware is just about 259Kb
I don't know where you found that number and I did not check how large the Tasmota minimal firmware for "legacy" ESP (ESP8266) is, but the Tasmota safeboot firmware for ESP32 (non-S3) is 813kB:
The safeboot partition is 832kB, see the docs. This is very close to my estimate of 875kB above.
Moreover, I think that a comparison with Tasmota, at least regarding firmware update strategies, makes no sense. Tasmota devices are buried behind some physical switch or inside lamps or something. And most of them don't even have a port to connect to for firmware updates. Also, those devices cannot just be upgraded to an ESP32-S3 with 8MB. The requirement for supporting OTA updates on these devices is obvious. I would never argue about that. I wouldn't use Tasmota on my many ESP devices if I had to dig them up and solder wires to them (Shelly mini series) every time I wanted to perform an update. It makes perfect sense to me that people invest time in having and maintaining the Tasmota safeboot firmware.
The situation for OpenDTU-OnBattery, however, is very different. So different, that I think OTA updates are a convenience, not a requirement. And users may keep this convenience, they merely have to install a different MCU. It's a one-time effort. And as other people stated: 30 bugs for a Fusion board is justifiable given that the cost for all the other parts of a small solar plant are so much higher.
Given that, I still want to have Helge on board and feeling good about how we move forward. And he was right to suggest that we ask upstream for an opinion. Even though I was about to move forward, we will wait for their input on this matter.
schlimmchen
commented
2 days ago @spcqike is right, we can just serve static (or mostly static) HTML pages in the minimal firmware. The minimal firmware will have a completely different look, it will be ugly, and there will be no translations (at least no dynamic ones). Should be okay for something one only sees when performing an OTA update. That should bring us down to ~960kB for the minmal firmware.
I think we should add some sort of API between the main firmware and the minimal firmware, i.e., the minimal firmware uses its own (very limited) set of settings, and each time an OTA update is triggered, the main firmware constructs this from the active config. This allows to change the main config without the need to keep compatible with the minimal firmware. The only thing that might need an update in such a case is the function writing/updating the minimal firmware config. Also, getting rid of ArduinoJson should save code (roughly 33k as estimated earlier).
spcqike
commented
2 days ago Also, getting rid of ArduinoJson should save code (roughly 33k as estimated earlier).
That’s what I thought when writing
Would it be possible to like store the Ethernet pins in a different file if and only if Ethernet is used? In the main firmware.
So to have a file like „Ethernet.conf“ wich is just one like of comma separated PIN numbers (like „12,4,15,16“ or whatever pins are used for Ethernet) this file could be created by pinmapping loading process and holds the Ethernet pins in a certain order. Of course only if Ethernet is used 😄
we could also use some EEPROM bytes to hold these pins. If eeprom handling uses less Programm size than LittleFS, idk. (Even tho at the end eeprom is also just emulated in flash)
But you wrote we need the config file and therefore JSON anyway, as it holds the wifi credentials.
Snoopy-HSS
commented
1 day ago I try to compile the current development and it says: raise Exception("firmware binary too large: %d > %d" % (fw_size, max_size)) Exception: firmware binary too large: 1970624 > 1966080
is it already over with ESP32_generic 4MB ????
schlimmchen
commented
1 day ago But you wrote we need the config file and therefore JSON anyway, as it holds the wifi credentials.
We shall use the approach to translate current settings for the minimal firmware with both the pin mapping (the part which is required to setup Ethernet) as well as critical config values, so no JSON needs to be parsed.
is it already over with ESP32_generic 4MB ????
Well, yes. You currently cannot compile firmware for generic or generic_esp32 or any other ESP32 (non-S3) based on the current development branch. The partition layout must be updated to make more space for the sketch, one way or another. For that reason I disabled those environment in the github action builds.
Since you are compiling yourself, here is the simplest approach you can take: Update partitions_custom_4mb.csv to look like this (remove partition app1 and double app0 in size):
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, 0x10000, 0x3C0000,
spiffs, data, spiffs, 0x3D0000, 0x30000,Compile again and flash using esptool over USB. OTA is broken, continue updating only using esptool.
Other than that you can start stripping out pieces of code that you are not using anyways, like the AC charger code, or whatever you find is not used in your setup.
Snoopy-HSS
commented
1 day ago It's weird. When I compile for the S3 size is smaller !? why is there a 2MB limit when i use the generic_S3 ???
Checking size .pio\build\generic_esp32s3\firmware.elf Advanced Memory Usage is available via "PlatformIO Home > Project Inspect" RAM: [== ] 24.0% (used 78680 bytes from 327680 bytes) Flash: [==========] 95.9% (used 1886133 bytes from 1966080 bytes) Building .pio\build\generic_esp32s3\firmware.bin
In Platform.ini default ENV is still generic_ESP32
Changing partition worked, can also be uploaded Serial via, visual Studio code
schlimmchen
commented
1 day ago It's weird. When I compile for the S3 size is smaller !?
Yes, as the S3 is a slightly different processor with a different instruction set, so the code is can be smaller (more efficient instructions, etc.).
why is there a 2MB limit when i use the generic_S3 ???
I don't know what you mean. The partition table is the same for ESP32-S3 and non-S3.
Snoopy-HSS
commented
1 day ago Flash: [==========] 95.9% (used 1886133 bytes from 1966080 bytes)
Only a little under 2MB availabe. How can i change, that it uses 16MB filesystem when compiling generic_esp32s3
is this part for all boards? board_build.partitions = partitions_custom_4mb.csv
it's set in platform.ini to 4MB and ESP32 per default...
TL;DR
Flash Size Constraints
The flash memory of your ESP32 is partitioned. The legacy partition layout allocates a little less than 2 MB for one copy of the firmware. There is room for a second copy of the firmware, allowing over-the-air (OTA) updates. Overall, the partition layout is currently designed to fit into 4 MB of flash memory.
As OpenDTU-OnBattery is based on OpenDTU, this layout is inherited from said upstream project. OpenDTU-OnBattery is kept from evolving due to the partition's size constraints. Many additional features (compared to the original OpenDTU) require a siginificant amount of additional binary code, which needs space in flash memory.
Short-Term Mitigations
We actually already hit the limit with version 2023.05.03 and had to employ creative solutions to restore compatibility with the existing flash layout. See #958 (and #950) for details.
We might be able to continue reducing the binary firmware image size slightly. However, note that the low-hanging fruit has already been harvested. Keeping compatibility with 4 MB flash will become a struggle in the near future.Long-Term Solution
As discussed in #935, where alternate solutions were also discussed, we will introduce a breaking change with the next release, which redesigns the partition layout for 8 MB (and larger) flash memories. This doubles the amount of space available for the firmware binary, allowing us to continue adding features to the project. Doubling the firmware partitions in size will give headroom for the foreseeable future.
You will then need an ESP32 with at least 8 MB of flash memory in order to install and run OpenDTU-OnBattery.
Timeline
Version 2024.06.03 is the last version to support the legacy partition layout, i.e., ESP32 with 4 MB of flash memory.
We are unable to keep up with the upstream changes in v24.6.10, as adding those increases the firmware size beyond the allowable maximum.
We ran out of options to reduce the firmware size further with an acceptable amount of effort.
There is no specific timeline for the introduction of the breaking change.The project is continuing to evolve and new features are added. Sometimes even fixes add more code to the binary. Merging changes from upstream, including updated libraries, bears a risk of enlarging the firmware binary in size. We might put effort into keeping 4 MB flash devices supported for a while, if the work to do so is managable.Some day in the near future (weeks, maybe a few months), the firmware blob will be too big for the existing partition layout, possibly despite efforts to keep the firmware size at bay.Actions Required Now
Determine Upgrade Requirement
Determine whether or not you need to upgrade your ESP32.
This can be tricky. The variety of ESP32 in the wild can make it hard to pin down the exact ESP32 model and/or board and hence the amount of flash your ESP32 has available.
You do not need an upgrade if you use an OpenDTU Fusion board.
Option 1: Perform Firmware Upgrade
Install OpenDTU-OnBattery 2024.06.03 and navigate to the system's information page of the web UI. It lists the flash memory size in section "Hardware Information". This information is expected to be reliable.
Option 2: Avoid Firmware Upgrade
Option 2.1: Examine Hardware
Have a look at your hardware and note the exact model name engraved on the ESP32 module's RF shield. Consult the appropriate datasheet to find out how much flash your module should have available.
This method leaves significant room for errors. Custom chips/modules are hard to identify.
Option 2.2: Use
esptoolConnect your ESP32 board to a computer with
esptoolinstalled and executeesptool --port <serial_port> flash_id(where<serial_port>often is/dev/ttyUSB0or/dev/ttyACM0on Linux orCOMn(ninteger) on Windows).The output contains a line
Detected flash size: <?>MB. This information is expected to be reliable.Executing this command does not alter memory. The board can be put back into production with a power-on reset.
Select an Upgrade
Depending on your overall setup, selecting a replacement ESP32 (board) can be simple. The "NodeMCU" or "Espressif DevKitC" style development boards are available in many different configurations. If you use one of those and chose to use a pluggable connection to peripherals, in particular to connect to the RF module(s), a replacement development/prototype board could be a good option.
For custom setups or those using soldered connections, you might need to consider replacing more than just the ESP32 board.
OpenDTU Fusion
Consider an OpenDTU Fusion board to be your next driver for OpenDTU-OnBattery. It ships with an ESP32-S3 with (at least) 8 MB of flash memory and (at least) 2 MB of PSRAM. Support for the OpenDTU Fusion board is very likely to be sustained for a long time. Additional hardware to connect peripherals which are typically in use by OpenDTU-OnBattery users (CAN transceiver, 2 ADUM1201 isolators for serial connections (VE.Direct)) is expected to be available in June 2024, while a limited number of working prototypes with a single ADUM1201 are already available.
OnBattery Pro PCB
A community member (@swingstate) is working on making a new board ready for production, which provides ready-to-use ports for typical OpenDTU-OnBattery peripherals. It features an ESP32-S3 with 16 MB of flash and 8 MB PSRAM. If you can wait for the release of this product, you might want to consider it. See #541 and a comment in #935 for details.
Other Boards
You should prefer an ESP32-S3 (note the "S3"). The ESP32-S3 has more GPIOs (not all may be pinned out) and a slightly more efficient core. More importantly, it has native USB support, freeing one hardware UART.
Make sure that the board, to which the ESP32-S3 is soldered, actually uses the native USB capability, such that the third hardware UART becomes available. Look out for a USB to UART transceiver. If there is one on the board, the chip is possibly not connected natively to the USB port. There are boards with two USB connectors, where one does connect natively to the ESP32. In that case, the board is fine.
Flash Memory Size
At least 8 MB.
PSRAM
Treat yourself a chip with embedded PSRAM. Support for PSRAM is likely to arrive in OpenDTU-OnBattery eventually, allowing for stable operation despite multiple concurrent features being enabled and requiring large blocks of heap memory.
Known Good Modules
Look out for boards with one of these ESP32-S3 modules (prefer one with PSRAM):
You will not be able to leverage 16 MB of flash memory with OpenDTU-OnBattery any time soon. However, given the price difference, you might want to go for a version with 16 MB, allowing the module/board to be reused in a different project eventually.
Resoldering an ESP32 module
If you know how to remove the ESP32 module from your board and solder a new one in, you know what you are doing and you should be good to go to select a suitable ESP32 module.
Actions Required Once the Update Hits
The details still need to be worked out.
There will a guide on how to perform the update once the breaking change arrives.
Questions and Answers
Can I keep running my OpenDTU-OnBattery using Firmware 2024.06.03 or older?
If you are happy with the way OpenDTU-OnBattery performs, you may very well keep your setup as it is.