ESP32-FlashZ

an arduino library that provides zlib compressed OTA update feature for esp32

EXAMPLES | CHANGELOG | | |

Unlike esp8266, esp32's bootloader does not (yet) support compressed image updates. But esp32 has a miniz lib decompressor in ROM, so it can be used to decompress zlib packed stream on the fly during OTA update and write decompressed data to SPI flash. This could be done both for firmware and filesystem image.

I wrote this lib just for fun to get the idea of miniz functions hidden in esp32 ROM. FlashZ lib is integrated into EmbUI framework as OTA update handler. The code inspired by esptool that does the same with it's serial flasher stub.

Features

• low code overhead, deco algo is already present in ROM
• both firmware and filesystem compressed images upload supported
• compatible with ESP32 WebServer, autodetect compressed/non-compressed images
• compatible with ESPAsyncWebServer), autodetect compressed/non-compressed images
• stream decompression, i.e. via http client (fetch and flash compressed image from remote URL)
• PlatformIO integration via post_flash.py script that automates on-the-fly compression and OTA upload for your project
• on-the-fly compress/decompress upload from browser example via pako js lib (tnx to @playmiel for contribution)
• esp32-flashz is used in BLEOTA library to upload updates to esp32 via BlueTooth

Requirements

• 32k of heap memory during decompression for dict data
• derives from Arduino's UpdaterClass to perform flash operation on inflated data
• firmware images must be compressed in zlib stream (not a gzip format file!). There are many ways to do this, i.e.
  • use pigz tool with -z flag
  • use python's zlib module (check post_flash.py script for Platformio)
  • use perl's Compress::Zlib module
  • use pako js lib

Check examples to get more idea on how to intergate this lib into platformio projects

Compatibility

Comparison

some basic tests

Using FlashZ lib

FlashZ Lib consists of a low level FlashZ singleton that derives from a built-in Arduino's UpdateClass class and provides additional methods to handle zlib compressed data. It tries to maintain same API as Update.

FlashZ::beginz suposed to be called instead of UpdateClass::begin call to intialize update procedure. It allocates zlib Inflator structures and dictionary memory. For uncompressed data FlashZ::begin could be used as usual with UpdaterClass.

FlashZ::writez is called to inflate and flash zlib compressed block of data. bool final flag is used to signal last piece of data input.

FlashZ::writezStream can read a standart Stream class objects, decompress and write decompressed stream to flash. NOTE: total stream length must be known in advance, so that Inflator insures a proper end of stream is reached on decompression.

FlashZ::abortz or FlashZ::endz must be called to end the update and release dynamically allocated Inflator memory.

To stich FlashZ with networking and OTA updates here is a FlashZhttp class. This is not a complete OTA updater solution but more of a reference implementation example. Any real-life projects could easily implement something similar with more features, bells and whistles. FlashZhttp class integrates WebServer or AsyncWebServer file upload feature with FlashZ low level methods. Also it can initiate streamed download via http client from a remote URL (only plain http). FlashZhttp methods includes some heuristic in attempt to autodetect file image format and type, so that it can handle both compressed and uncompressed images transparently. But for compressed file it can't autodetect between firmware and FS image, so it need some metadata to differetiate. This is implemented via additional POST data fields.

Build-time options

By default AsyncWebServer support is not build into lib, do not want to intorduce dependency for external lib. To get AsyncWebServer support, FlashZ lib must be build with FZ_WITH_ASYNCSRV flag. This could be done via PlatformIO build_flags. AsyncWebServer and ESP32 WebServer support options are mutually exclusive due to some definitions clashing. See EXAMPLES projects for further details.

By default FlashZhttp class within the lib includes support for HTTP Client to be able to download and flash an image from a remote URL. It uses Arduino's HTTPClient.h lib for that, which by default includes SSL support, even if client code does not meant to be using it. SSL support makes a huge impact on resulting firmware image size, it grows in about 100k. If you do not need SSL support in HTTP Client for the sake of reducing resulting image size you can define HTTPCLIENT_NOSECURE to disable SSL and allow linker exclude SSL-related code from the resulting firmware image. This flag is available only since Arduino Core v3.0.2. Added in PR #9893. For previous versions of Arduino core you can define FZ_NOHTTPCLIENT build flag to completely disable HTTP Client support in this lib and reduce firmware size.

Also you should always specify NO_GLOBAL_UPDATE build flag for your project to prevent Arduino's UpdateClass creating it's instance by default. FlashZ uses it's own instance of a derived class and default one just wastes your memory (about 180 bytes). See arduino-esp32/pull#8500

On-the-fly compression of uploaded images via pako js lib

An example of implementing firmware updating web page with embedded js code that compresses raw uploaded images on-the-fly could be found in asyncserver-flashz-pakojs.

Integration with PlatformIO

It is pretty easy to integrate PlatformIO with HTTP OTA update via post build scripting. Python's zlib module could be used to compress firmware image after building and http-client module to upload a compressed image to ESP32 board Over-the-Air. See a reference implementation in post_flashz.py example. It relies on FlashZhttp class methods to process POST form data but could be adjusted easily. Additional platformio.ini variables are used to set remote address of a board. Uploading compressed firmware/FS is done automagicaly via simple pio run -t upload. (MCU must be connected to network and reachable).

Using CLI tools for updates

I'm a linux user and prefer to use cli tools for automating tasks rather than web browser. So here are some oneliners for ESP32-FlashZ updating

• compress and upload firmware image to ESP

pigz -9kzc .pio/build/esp32c3/firmware.bin | curl -v http://$ESPHOST/update -F file=@-

• compress and upload File System image to ESP

pigz -9kzc .pio/build/esp32-s2/littlefs.bin | curl -v http://$ESPHOST/update -F "img=fs" -F file=@-

• trigger esp32's firmware self-update from a remote host

curl http://$ESPHOST/update -F "img=fw" -F "url=http://$REMOTE/download/firmware.bin.zz"

• trigger esp32's FS self-update from a remote host

curl http://$ESPHOST/update -F "img=fs" -F "url=http://$REMOTE/download/littlefs.bin.zz"

License

Since I get the idea from a esptool code, this lib inherits esptool's GNU General Public License v2.0