Provide size-optimized builds of binary libraries (IDFGH-13133)

[rojer]

Is your feature request related to a problem?

Binary libraries comprise significant percentage of the resulting binary size, yet compilation flags don't apply to them

Describe the solution you'd like.

For those projects that prioritize size over performance (CONFIG_COMPILER_OPTIMIZATION_SIZE=y), binary libraries compiled with -Os should be provided

Describe alternatives you've considered.

There aren't any

Additional context.

Flash size is a big issue for us, yet binary blobs are a black box that keeps growing and growing.

[rojer]

yes, recompiling blobs with -Os would reduce IRAM usage.

[MaxwellAlan]

Hi @rojer

Plz provide some more details for this feat req.

• Chips
• IDF versions
• WiFi or Bluetooth ?
• etc

maybe you can refer to https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/performance/size.html

Thanks.

[rojer]

@MaxwellAlan we have no way of knowing which settings were used to compile the blobs, and no way to influence that either. IDF's default optimization is -Og, i sure hope that wasn't the case for blobs... currently only one set of binary libraries is provided, with its build settings unkown to us. ideally, a set corresponding to all the available optimization settings should be provided. but i personally care about -Os the most. this applies to all chips, all IDF versions, wifi, bluetooth libs as well as Newlib (the source for this one is open but it's still supplied as a binary library when building user apps).

[igrr]

a set corresponding to all the available optimization settings should be provided

We have considered this, but it's not practical due to the increased size of the libraries we need to distribute. I think all the wireless-related libraries are already built with -Os, but I'll leave it to @MaxwellAlan to confirm this.

As for newlib, it is the default setting in crosstool-NG to compile it with size optimization enabled.

[rojer]

@igrr ok Ivan, thanks for clarification. if blobs are already size-optimized... on the one hand this fulfils my request, on the other it also takes away some hope i had :)

size gains in IDF is relentless. going from 4.4 to 5.2.1 was a huge jump, i had to pull a number of hares out of a hat to make it fit our space budget - with no new features enabled, i might add (ok, WPA3-SAE was one). now 5.2.1 -> 5.2.2 adds another 14K in wireless libs, for what? nobody knows. some ble-wifi coex stuff, seemingly... -std=gnu++17 -> -std=gnu++20 is extra 20K for us, because of some stdc++ changes. sorry, i needed to vent :)

[rojer]

@igrr as for dealing with binaries in repo - that's if they are stored inline. in mongoose os days, we also ha d anumber of binary libs, we recognized repo bloat issue and distributed them as github anf gitlab releases. some api glue code is needed, but overall it's not too bad. you are welcome to take inspiration if you like, the code we used to publish and fetch them is open :)

[igrr]

you are welcome to take inspiration if you like, the code we used to publish and fetch them is open

Thanks, we already have the intention of moving these over from submodules into a different distribution format — probably managed component releases, since we already have that. The main issue to deal with is still supporting offline installations, as part of the user base is behind firewalls and they can't access the internet.

with no new features enabled, i might add (ok, WPA3-SAE was one). now 5.2.1 -> 5.2.2 adds another 14K in wireless libs, for what? nobody knows. some ble-wifi coex stuff, seemingly...

I think we should definitely check if this increase can be managed better anyhow (i.e. reduced :)), or the size can be reduced if some of the options are disabled. For example, a while ago we have reduced the size of the Wi-Fi code that gets linked into the binary if only STA mode was enabled. Perhaps something similar can be done here. I would suggest rephrasing the ticket to describe your original goal — reducing the size of Wi-Fi code that gets linked into your application. It would help Wi-Fi developers find optimization ideas if you also attach an example .map file or somehow indicate the APIs/features you use, and which you don't need.

[rojer]

well, this was one thing we could do. let's wait for @MaxwellAlan to confirm current wireless library build settings and if they are -Os, we can call this one done. i'll follow up with other requests, especially the C++ standard change once i understand better under what conditions it occurs.

[rojer]

The main issue to deal with is still supporting offline installations, as part of the user base is behind firewalls and they can't access the internet.

we solved that by providing a flag to check for the required blobs before fetching releases. either we fetch them ourselves, or the user does and supplies them in a specified --binary-libs-dir=....

[rojer]

for the 5.2.1 -> 5.2.2 blob growth i filed https://github.com/espressif/esp-idf/issues/14098, including a size diff, i'll appreciate if someone can take a quick look at that. 5.2.2 actually came with a very nice reduction in size for ESP32, which makes ESP32-C3 growth more surprising.