search

Hieromon / AutoConnect

An Arduino library for ESP8266/ESP32 WLAN configuration at runtime with the Web interface
https://hieromon.github.io/AutoConnect/
MIT License
903 stars 189 forks source link

ESP8266 with 1MB of flash: not enough space for OTA updates with v1.1.7 #227

Closed fbattarra closed 4 years ago

fbattarra commented 4 years ago

First of all, thank you so much for your incredibly useful job! I love AutoConnect and I am looking forward for the v1.2 for

I'm working with PlatformIO and I recently tried to move from a WeMos D1 mini to an ESP-01S, stumbling into an issue which is not so trivial to me.

when uploading firmware via PlatformIO / USB, esptool write_flash parameters (afaik) compresses the binary (Compressed 477952 bytes to 333143), while when I just compile my code in order to have a .bin to upload OTA the compression seems not among the triggered tasks and the final file has the uncompressed size (that's 3.79 Mbit for a 4Mbit board).

I don't know whether I could be able, by better investigating how to compress my build via PlatformIO, I eventually would be able to overcome the issue I am facing: my code (which at the moment is sightly more than an HelloWorld ) nicely fits when uploaded via the IDE / USB, but on an OTA / Browser update the "not enough space" appears.

On the WeMos board I don't have any issue, but it's obvious as there I can count on 32Mb.

I was relying a lot on the OTA update for ESP-01S board, as I need to remove its header pins once programmed for the first time; do you have any hint on what I am actually doing wrong? :)

[EDIT 20200531] As found on OTA Basic Requirements for Arduino framework, OTA is doable only if there's enough room to store the new firmware on the flash before having it replacing the old one (and, hey: this makes sense, actually). As my bin is more than half the size of the flash (Flash: [====== ] 62.2% (used 473792 bytes from 761840 bytes)) , there won't be enough space on the FS to upload the same binary via OTA. Compressing it (how?) could then be a solution, should one manage to have at the end twice the space on the flash, couldn't it?

Another possible solution I've found (which has been made for ESPEasy) could be a two-steps OTA update: a small image that has been build just to

You then would always have to use a two-step process to renew using OTA on 1MB modules, starting with this uploader image as an in-between step to final, big, actual firmware images.

Unless I'm wrong, then, nowadays an ESP breakout board with 1MB of flash won't actually support OTA updates of firmware containing AutoConnect library (& bundled dependencies). This remark can relate to #203 request, can't it?

Can you please advice, and/or add a note on the OTA feature related to the actual flash size of the ESP board and/or foresee a (guide to a) smart two-steps OTA updater?

All my gratitude, once more, for your fantastic job!

Hieromon commented 4 years ago

@fbattarra Thank you for using AutoConnect. Bloating of the .bin size is a serious concern. I know the description for uploading a compressed binary file or filesystem in the same section you mentioned. https://arduino-esp8266.readthedocs.io/en/latest/ota_updates/readme.html#compression Also, there are ideas on how to implement the above method with PlatformIO in the following forum topics of PlatformIO. https://community.platformio.org/t/esp8266-ota-upload-vs-compressed-ota-upload/13487/5 However, I haven't tried this method yet. For now, I think these methods are the hope to rescue the OTA from the bloated size of the .bin.

fbattarra commented 4 years ago

@Hieromon Heartfelt thanks for your reply and your hints!

As from the forum topics, as I'm already using Expressif8266@2.5.1, I eventually needed nothing but gzipping (-9 for maximum compression) the bin once compiled with PlatformIO.

I obtained 333135 bytes instead of 477952 (which was stated as 62.2% of 761840 availables), but it remains still too big for an OTA update (where I should not exceed the remaining 37.8% / 288048).

Please note that, so far, other than AutoConnect I am using only ESP8266mDNS in my sketch, so there really is no room for saving space by trying to remove something.

That said, what about the other possible solution, uploading OTA a first, very lightweight special transitional firmware containing no AutoConnect, but only what's needed to retrieve from the EEPROM working WiFi credential (saved by the previous, AutoConnect-based fw) and perform another OTA update with much more space available? A tinier fw should free a bunch of flash, allowing then to upload, in the 2nd step, the actual new firmware (gzipped).

It seems a nerdy, brainy way to overcome a board's flash limits for OTA update, where the fw doesn't leave enough space to store a newer version of itself, but it could indeed be the smarter solution available.

Could you please, then, point me towards the code where it reads the EEPROM looking for previously stored SSIDs ? I'd then try to take inspiration from the .ino offered for ESPEasy for the same purpose and modify it in order to work for AutoConnect... :)

[EDIT 20200602] Speaking of my case (ESP-01S with 1MB of flash, PlatformIO IDE), I found someone pointing to something perfectible on PlatformIO json file for the board esp01_im, and adding on platformio.ini the line board_build.ldscript = "eagle.flash.1m64.ld" (overriding the json where it's instead “ldscript”: “eagle.flash.1m256.ld”) I have now my sketch using Flash: [===== ] 46.3% (used 473888 bytes from 1023984 bytes) instead of 62.2% of 761840. Here, the available LD scripts (as from PIO 3.6.0 doc for ESP8266)

If my own code and libraries will be tiny enough, and with the help of gzip, I should now be able to OTA update AutoConnect on boards with 1MB of flash.

I can already confirm you it's properly working if i update OTA the same firmware (gzipped: 333289 KB). Good vibrations :)

Hieromon commented 4 years ago

@fbattarra Thank you for the investigation. The method you presented as a solution is not really much different from the implementation of the AutoConnectOTA class. I think it will be revealed if you explore the AutoConnectOTA.cpp.

The location of the firmware to replace with the bootstrap based on the ESP8266 flash layout is deterministic. And the source binary that the bootstrap copies must fit in the OTA update area. To implement your proposed method, AutoConnect seems to need to write the binary uploaded to the file system directly to the boot area of the Sketch without bootstrapping, but is it possible to do it by the Sketch itself without using esptool.py?

fbattarra commented 4 years ago

@Hieromon in the meanwhile I found even another "tiniest fw for OTA updates", supposed to let as much free space as possible in the flash for uploading OTA firmwares that are more than half the size of the flash. It somewhat works, even though I found no proper error management there, so you start with your project containing AutoConnect leaving not enough room for an OTA update of itself, then you use the OTA update feature to update a more lightweight firmware and, eventually, you upload your real new firmware OTA because not OTA update area is... bigger.

Therefore, it seems that after the override of esp01_1m board specifications for PIO I don't need anymore a firmware-in-the-middle so I won't investigate further the topic - even because 1MB boards are already the bare minimum for projects using AutoConnect and, so far, I am cool with the solution working on PIO IDE.

I am sorry to have not an answer to your question because of my lack of expertise: you mention "OTA update area" and, apart from what I find on the documentation (and, I admit, I haven't yet read carefully) it's unclear to me how to juggle with the memory management.

I'm now with 550096 KBytes free (although I cannot tell how many for the "OTA area") and a 333289 KB firmware to upload... I feel plenty of room and not anymore in the need of having an intermediate fw to upload up to 640KB worthy firmwares OTA :) That's why I closed the issue myself :)

Thank you so much once more for the amazing library and your active support!