Flash memory use way up with v3.0.0

[moose4lord]

Board

MH ET Live ESP32MiniKit

Device Description

Plain module on a breadboard

Hardware Configuration

none

Version

v3.0.0

IDE Name

Arduino IDE 2.3.2

Operating System

macOS 14.5

Flash frequency

80MHz

PSRAM enabled

no

Upload speed

115200

Description

I have a reasonably complex sketch that uses quite a few large libraries (WiFi, BluetoothLE, ArduinoJson, ESPAsyncWebServer, HTTPClient, FastLED, ESP_HTTPS_OTA). With the 2.0.17 version of the arduino-esp32 core I've always been able to squeeze everything into 4M of flash memory (using an OTA partition scheme with a minimal SPIFFS partition).

Sketch uses 1870801 bytes (95%) of program storage space. Maximum is 1966080 bytes.
Global variables use 74044 bytes (22%) of dynamic memory, leaving 253636 bytes for local variables. Maximum is 327680 bytes.

Unfortunately, with v3.0.0 of arduino-esp32 core my sketch is now too big.

Sketch uses 2051421 bytes (104%) of program storage space. Maximum is 1966080 bytes.
Global variables use 72076 bytes (21%) of dynamic memory, leaving 255604 bytes for local variables. Maximum is 327680 bytes.

I guess I would expect a new major release of arduino-esp32 to require more memory, but a 9% increase seems like a lot. I used avr-nm to look at where all the flash is being used, but I don't see anything that's grossly out of whack (although I'm admittedly not an avr-nm expert and it produces a ton of information that's hard to parse).

Anyway, in trying to hack out some code to make room, avr-nm does show some things I don't understand. For example, here's what avr-nm shows are the largest linked modules for a sketch with empty setup() and loop() functions:

00002071 t mbedtls_sha256_software_process
00003400 T _dtoa_r
00007510 T _svfiprintf_r
00007908 T _vfiprintf_r
00012262 T _svfprintf_r
00012582 T _vfprintf_r

Why are the large 'printf' modules still linked into the executable when there's no print features in the sketch? And why would mbedtls be included when there's no encryption being done? Does the arduino-esp32 core need them behind the scenes for some reason?

If anyone has any tips on reducing the flash memory requirements for v3.0.0, please chime in.

Thanks

Sketch

// this is what I used just to get a feel for what avr-nm would show for a minimal sketch. my real application sketch is way more complicated. :)
void setup() {
  // put your setup code here, to run once:
}

void loop() {
  // put your main code here, to run repeatedly:
}

// Sketch uses 252537 bytes (12%) of program storage space. Maximum is 1966080 bytes.

Debug Message

Sketch too big; see https://support.arduino.cc/hc/en-us/articles/360013825179 for tips on reducing it.

Compilation error: text section exceeds available space in board

Other Steps to Reproduce

Here's the avr-nm command I used to get iinfo about flash memory use:

$AVR_PATH/avr-nm --size-sort -Ctd { path-to-elf-file }

I have checked existing issues, online documentation and the Troubleshooting Guide

• [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[me-no-dev]

Don't forget that you are running on top of ESP-IDF and while it might look like just setup and loop, in fact there is a full blown FreeRTOS underneath and more. The printf things probably come from debug and such and mbedtls from the fact that it could check if something is encrypted (i'm guessing here). The point is that this is not a simple MCU to which you run an empty sketch, but rather an OS that you do not run any real app on. There are many changes between IDF 4.4 and 5.1 (used in 2.x and 3.x) and also many changes in Arduino. Added options for PPP and more that do take some space in LwIP for example.

[AWSW-de]

Hello, i can confirm this issue with ESP32 board manager V3.0.0 with most of my ESP32 projects like reported from users already too: https://github.com/AWSW-de/WordClock-16x16-LED-matrix/issues/10

Tested with these so far: https://github.com/AWSW-de/WordClock-16x16-LED-matrix https://github.com/AWSW-de/WordClock-16x8-LED-matrix https://github.com/AWSW-de/AWSW-PixelClock-32x8

The downgrade of the board manager version to V2.0.17 helped to get rid of this error, so i guess there is a bug in this new release V3.0.0 that should be fixed to prevent many projects from stepping into this error on compiling them.

Thanks for a solution in advance :)

Kind regards AWSW

[me-no-dev]

There is no bug. Things change and evolve and as result require more space. You are welcome to change the partition scheme, flash size or stay at lower version.

[JAndrassy]

@moose4lord, do you need OTA to be HTTPS?

[moose4lord]

Thanks for everyone's comments.

@me-no-dev I understand what you're saying. I'm sure there are many, many changes under the covers, and new and improved features are bound to use more memory. I just wish it wasn't quite so much more. :) And just to clarify, the printf and mbedtls modules, which seemed extraneous to me, are also compiled into an empty sketch with arduino-esp32 core v2.0.17. So this is nothing new.

@JAndrassy I would be fine using HTTP for OTA updates, but it seems that load modules that handle encryption (mbedtls) are automatically compiled into even the smallest sketch, so I'm not sure substituting HTTP for HTTPS would save much space.

I will poke around some more to see if I can find enough optimizations to get my sketch to fit in 4M now. I might have to live without HTTPS OTA, which would be a shame. @me-no-dev suggestion about tweaking the partition scheme might help as well. I don't need a SPIFFS partition, even a small one, so getting rid of that may do the trick.

[me-no-dev]

@moose4lord in that case, just delete the SPIFFS partition and split the space if frees between the two app partitions. That should give you close to 2MB APP partitions

[Jason2866]

@moose4lord The official Arduino version has to meet the expectations to make as most as possible features available. This makes the base "fat" This was one reason for me to fork the project and do my own setup. Removed SPIFFS in IDF and Arduino. Reduced mbedtls to its absolute minimum so wifi is still working (removed Enterprise support). Removed Insights, Rainmaker, TFLite, WifiProv, BLE and (Network) ClientSecure and encryption update from the OTA update routine. Using BearSSL when cryptography routines are needed (https). Better performance than mbedtls and lower resources needed. For BLE using h2zero NimBLE (less resources, feature rich). This results is about 200k less flash usage than the official build and more free Heap. You can try (Platformio only!) since ArduinoIDE is a pain for bigger projects with

platform = https://github.com/tasmota/platform-espressif32/releases/download/2024.05.13/platform-espressif32.zip

The framework is used to build the project Tasmota

[moose4lord]

@me-no-dev I tried to create a custom partition scheme, but it doesn't seem to be working for me. Following this tutorial: https://docs.espressif.com/projects/arduino-esp32/en/latest/tutorials/partition_table.html I created a partitions.csv file in my sketch folder:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1F0000,
app1,     app,  ota_1,   0x200000,0x1F0000,
coredump, data, coredump,0x3F0000,0x10000,

This removes the SPIFFS partition and reallocates that space to the ota partitions, as you sugggested. But when I compile, it ignores this custom partition file and still uses the partition scheme selected in the Tools -> Partition Scheme menu. What did I miss?

[me-no-dev]

it does not ignore it, it just has no way of knowing the max size of firmware. Pick a partition scheme with similar or larger size app partition (like Huge APP for example)

[moose4lord]

Hmmmm, I don't know, I couldn't get the partitions.csv file to work in the sketch folder. I copied the min_spiffs.csv file from the tools/partitions folder, modified it and renamed it to partitions.csv, but it just didn't seem to work.

So I ended up moved my modified .csv file to the tools/partitions folder and edited the boards.txt file to make it available in the Arduino IDE Tools menu. That seems to be working ok, but dang, it's still a tight squeeze.

Sketch uses 2028353 bytes (99%) of program storage space. Maximum is 2031616 bytes.
Global variables use 71420 bytes (21%) of dynamic memory, leaving 256260 bytes for local variables. Maximum is 327680 bytes.

[me-no-dev]

@moose4lord any chance you can provide us with your .elf and .map files from both 2.x and 3.x builds, so we can try to figure out where the additional flash comes from?

[moose4lord]

Here's the entire build folder for each build. ledCubeBLE_3.0.0.zip ledCubeBLE_2.0.17.zip

And some info about each build and the custom partition scheme I created: Arduino-esp32 core v3.0.0 Sketch uses 2051421 bytes (100%) of program storage space. Maximum is 2031616 bytes. Global variables use 72076 bytes (21%) of dynamic memory, leaving 255604 bytes for local variables. Maximum is 327680 bytes.

Arduino-esp32 core v2.0.17 Sketch uses 1870801 bytes (92%) of program storage space. Maximum is 2031616 bytes. Global variables use 74044 bytes (22%) of dynamic memory, leaving 253636 bytes for local variables. Maximum is 327680 bytes.

Contents of custom partition scheme file (no_spiffs.csv):

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1F0000,
app1,     app,  ota_1,   0x200000,0x1F0000,
coredump, data, coredump,0x3F0000,0x10000,
#
# add these lines to boards.txt for the "MH ET LIVE ESP32MiniKit" board
# mhetesp32minikit.menu.PartitionScheme.no_spiffs=No SPIFFS (Large APPS with OTA)
# mhetesp32minikit.menu.PartitionScheme.no_spiffs.build.partitions=no_spiffs
# mhetesp32minikit.menu.PartitionScheme.no_spiffs.upload.maximum_size=2031616
#
# delete the ~/Library/Application Support/arduino-ide/Local Storage/leveldb folder
# to workaround the bug that prevents updates to board.txt from showing up in the IDE GUI

[me-no-dev]

@moose4lord you can see size comparison between the different examples we have here: https://github.com/P-R-O-C-H-Y/arduino-esp32/blob/gh-pages/SIZES_TEST.md

Looks like the network stack(WiFi/LwIP/ETH/PPP) added about 170KB to the firmware, BT added 20K, USB added another 20K and empty grew by about 5K.

[moose4lord]

@me-no-dev Interesting size comparison chart. Although I don't believe the ESP32/examples/Camera/CameraWebServer data. It shows a 474KB decrease in flash usage (for the older ESP32 device), which looked really suspicious to me. I ran the comparison and got a 151KB INCREASE in flash usage, which seemed to jibe with my experience. On the other hand, I ran the Ethernet example sketch comparisons too, and they really did have a 279KB decrease in flash usage, which surprised me.

[me-no-dev]

@moose4lord face detection/recognition has changed and has been disabled for ESP32 and ESP32-S2

[me-no-dev]

@moose4lord we can save 64KB from NetworkClientSecure if certificate bundle is not used: https://github.com/espressif/arduino-esp32/pull/9763

Other increases in flash size:

• 120KB come from various WiFi related things (MbedTLS, phy, supplicant, etc.)
• 12KB come from our Core
• 24KB come from LwIP (PPP, NAPT, etc.)
• 20+KB come from other system updates

[moose4lord]

@me-no-dev I saw your pull requests. That's awesome! Thank you.

[me-no-dev]

@moose4lord can we close this now?

[VojtechBartoska]

I am considering this as solved as PRs have been merged and 3.0.1 version released, if needed you can reopen. Thanks for pointing this out @moose4lord!

[horendus]

Will this be included in 3.0.2?

[Jason2866]

@horendus yes, it is already in 3.0.1

[me-no-dev]

Will this be included in 3.0.2?

It is in 3.0.1

[moose4lord]

After switching to the new no_fs.csv partition table, I'm now left with the question of how to update the partition table remotely. I'm currently using HTTPS_OTA_Update to do remote updates. After Googling a bit I came across this recent blog post: https://blog.toit.io/changing-an-esp32-partition-table-over-the-air-276c86feeba8 That looks interesting, but also dangerous. Anyone know if there's a more idiot-proof option?

[Jason2866]

@moose4lord It IS complicated and there is a high chance to generate a brick. Anyways not possible to do with standard Arduino since you are not allowed to write in any flash area. Arduino is compiled with the option "Dangerous writes abort" So either you write the repartitoning routine with IDF or compile the Arduino libs yourself with the option set "Dangerous writes allowed" and code the needed routines.

Most probably it is easier to compile the Arduino libs yourself and strip the option/features you don't need to save flash space.

[moose4lord]

@Jason2866 Looks like you're correct. I tried updating the partition table as described in the blog post, using arduino-esp core v3.0.1, and it generated an abort() when the code tried to erase the partition table. :( Oh well, back to the drawing board.