Support esptool.py merge_bin

[probonopd]

What kind of issue is this?

• [x] Feature Request. Start by telling us what problem you’re trying to solve. Often a solution already exists! Don’t send pull requests to implement new features without first getting our support. Sometimes we leave features out on purpose to keep the project small.

---

For esp8266, we get one file that can be flashed at offset 0x0.

For ESP32, we currently get multiple files that need to be flashed at various offsets. This is cumbersome.

Since some time, esptool.py supports making a merged flash file:

https://github.com/espressif/esptool/commit/5cca25598c1f00d32fdf59003da619ac7b03866d

For example:

esptool.py --chip esp32 merge_bin -o merged-flash.bin --flash_mode dio --flash_size 4MB 0x1000 bootloader.bin 0x8000 partition-table.bin 0x10000 app.bin

Will create a file merged-flash.bin with the contents of the other 3 files. This file can be later be written to flash with esptool.py write_flash 0x0 merged-flash.bin.

It would be nice if we could make a merged flash file directly in PlatformIO.

This has been requested/discussed before: https://community.platformio.org/t/export-of-binary-firmware-files-for-esp32-download-tool/9253/

It was suggested there to run pio run -v -t upload but this of course fails on a headless build server and is cumbersome.

[probonopd]

Manual workaround:

find . -name '*.bin'
# ./.pioenvs/esp32_4MB_cam_wifi/partitions.bin
# ./.pioenvs/esp32_4MB_cam_wifi/bootloader.bin
# ./.pioenvs/esp32_4MB_cam_wifi/firmware.bin

cd ./.pioenvs/esp32_4MB_cam_wifi/
~/.platformio/packages/tool-esptoolpy/esptool.py --chip esp32 merge_bin -o merged-flash.bin --flash_mode dio --flash_size 4MB 0x1000 bootloader.bin 0x8000 partitions.bin 0x10000 firmware.bin
cd -

It says Wrote 0x158b90 bytes to file merged-flash.bin, ready to flash to offset 0x0 and the resulting firmware image can be flashed like this:

python3 -m esptool --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size 4MB  0x0 firmware.bin 

[ivankravets]

Do you mean pio run -t merge_bin target?

[Jason2866]

Manual workaround:

find . -name '*.bin'
# ./.pioenvs/esp32_4MB_cam_wifi/partitions.bin
# ./.pioenvs/esp32_4MB_cam_wifi/bootloader.bin
# ./.pioenvs/esp32_4MB_cam_wifi/firmware.bin

cd ./.pioenvs/esp32_4MB_cam_wifi/
~/.platformio/packages/tool-esptoolpy/esptool.py --chip esp32 merge_bin -o merged-flash.bin --flash_mode dio --flash_size 4MB 0x1000 bootloader.bin 0x8000 partitions.bin 0x10000 firmware.bin
cd -

It says Wrote 0x158b90 bytes to file merged-flash.bin, ready to flash to offset 0x0 and the resulting firmware image can be flashed like this:

python3 -m esptool --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size 4MB  0x0 firmware.bin 

This workaround is error prone since you hard code flash mode and size. Invested some time to write a platformio pio script (for Tasmota) to correctly generate a merged firmware bin.

So YES this would help a lot, for many projects, which offers pre compiled firmwares. Using merged bin firmwares is much easier (and way less error prone) with https://github.com/esphome/esp-web-tools for example

[valeros]

I believe a simple extra scripts should solve the problem. Something like this:

[env:esp32dev]
platform = espressif32
framework = arduino
board = esp32dev
extra_scripts =
    merge_firmware.py

merge_firmware.py:

Import("env")

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"
MERGED_BIN = "$BUILD_DIR/${PROGNAME}_merged.bin"
BOARD_CONFIG = env.BoardConfig()

def merge_bin(source, target, env):
    # The list contains all extra images (bootloader, partitions, eboot) and
    # the final application binary
    flash_images = env.Flatten(env.get("FLASH_EXTRA_IMAGES", [])) + ["$ESP32_APP_OFFSET", APP_BIN]

    # Run esptool to merge images into a single binary
    env.Execute(
        " ".join(
            [
                "$PYTHONEXE",
                "$OBJCOPY",
                "--chip",
                BOARD_CONFIG.get("build.mcu", "esp32"),
                "merge_bin",
                "--fill-flash-size",
                BOARD_CONFIG.get("upload.flash_size", "4MB"),
                "-o",
                MERGED_BIN,
            ]
            + flash_images
        )
    )

# Add a post action that runs esptoolpy to merge available flash images
env.AddPostAction(APP_BIN , merge_bin)

# Patch the upload command to flash the merged binary at address 0x0
env.Replace(
    UPLOADERFLAGS=[
            f
            for f in env.get("UPLOADERFLAGS")
            if f not in env.Flatten(env.get("FLASH_EXTRA_IMAGES"))
        ]
        + ["0x0", MERGED_BIN],
    UPLOADCMD='"$PYTHONEXE" "$UPLOADER" $UPLOADERFLAGS',
)

[Jason2866]

the merge command should include the needed flash mode settings. code sniplet:

    flash_size = env.BoardConfig().get("upload.flash_size", "4MB")
    flash_freq = env.BoardConfig().get("build.f_flash", "40000000L")
    flash_freq = str(flash_freq).replace("L", "")
    flash_freq = str(int(int(flash_freq) / 1000000)) + "m"
    flash_mode = env.BoardConfig().get("build.flash_mode", "dio")
    memory_type = env.BoardConfig().get("build.arduino.memory_type", "qio_qspi")

    if flash_mode == "qio" or flash_mode == "qout":
        flash_mode = "dio"
    if memory_type == "opi_opi" or memory_type == "opi_qspi":
        flash_mode = "dout"
    cmd = [
        "--chip",
        chip,
        "merge_bin",
        "-o",
        new_file_name,
        "--flash_mode",
        flash_mode,
        "--flash_freq",
        flash_freq,
        "--flash_size",
        flash_size,
    ]

[probonopd]

Why hardcode the numbers?

[probonopd]

Do you mean pio run -t merge_bin target?

Yes. Does it already exist and I missed it?

[Jason2866]

Why hardcode the numbers?

What do you mean with harcoded numbers?

[probonopd]

4MB, 40000000L.

[Jason2866]

That are the defaults ONLY used when not set in the boards config.

[probonopd]

Do you mean pio run -t merge_bin target?

Yes, or maybe even make it the default just like it is for the esp8266. What I don't know is: Do we still need the unmerged image for something, e.g., for OTA? Or can the merged one be used for that, too?

[Jason2866]

Do you mean pio run -t merge_bin target?

Yes, or maybe even make it the default just like it is for the esp8266. What I don't know is: Do we still need the unmerged image for something, e.g., for OTA? Or can the merged one be used for that, too?

For OTA you can not use the merged bin. For OTA the plain firmware file is needed.

[probonopd]

Strange that it works for esp8266 but not for esp32 - do you know the reason @Jason2866?

[Jason2866]

esp8266 built firmware is just one file. There are no other files.

[probonopd]

Yes, the same is what I'd like for esp32.

[Jason2866]

No, it is not. The esp8266 is completely different. There is just one file at all. The file after compile is linked and can be flashed as it is. Esp32 is compiled and linked and need additional files when flashing on an empty esp32. For OTA this other files are not possible to be there. That has nothing to do with Platformio. ESP32 != ESP8266 You can't have the same behaviour. Like girls and boys ;-)

[alex-code]

This would be a useful action to have so you can build release binaries for use with web serial flashing. Would be handy to have an option to merge in the SPIFFS bin too.

[probonopd]

Indeed, SPIFFS/LittleFS should ideally also be merged.

[Jason2866]

@alex-code We do all this (and a few things more for Tasmota needs) in this pio script. https://github.com/arendst/Tasmota/blob/development/pio-tools/post_esp32.py It is an enhanced version what valeros showed.

[probonopd]

It would be great if there was a pio run -t ... command to do this "out of the box" in PlatformIO.

[dewenni]

I believe a simple extra scripts should solve the problem. Something like this:

[env:esp32dev]
platform = espressif32
framework = arduino
board = esp32dev
extra_scripts =
    merge_firmware.py

merge_firmware.py:

Import("env")

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"
MERGED_BIN = "$BUILD_DIR/${PROGNAME}_merged.bin"
BOARD_CONFIG = env.BoardConfig()

def merge_bin(source, target, env):
    # The list contains all extra images (bootloader, partitions, eboot) and
    # the final application binary
    flash_images = env.Flatten(env.get("FLASH_EXTRA_IMAGES", [])) + ["$ESP32_APP_OFFSET", APP_BIN]

    # Run esptool to merge images into a single binary
    env.Execute(
        " ".join(
            [
                "$PYTHONEXE",
                "$OBJCOPY",
                "--chip",
                BOARD_CONFIG.get("build.mcu", "esp32"),
                "merge_bin",
                "--fill-flash-size",
                BOARD_CONFIG.get("upload.flash_size", "4MB"),
                "-o",
                MERGED_BIN,
            ]
            + flash_images
        )
    )

# Add a post action that runs esptoolpy to merge available flash images
env.AddPostAction(APP_BIN , merge_bin)

# Patch the upload command to flash the merged binary at address 0x0
env.Replace(
    UPLOADERFLAGS=[
            f
            for f in env.get("UPLOADERFLAGS")
            if f not in env.Flatten(env.get("FLASH_EXTRA_IMAGES"))
        ]
        + ["0x0", MERGED_BIN],
    UPLOADCMD='"$PYTHONEXE" "$UPLOADER" $UPLOADERFLAGS',
)

@valeros Thanks for this example script, It would be nice to use this, but if I use this, I get the following error:

TypeError: 'NoneType' object is not iterable:
  File "/Users/xxx/.platformio/penv/lib/python3.11/site-packages/platformio/builder/main.py", line 182:
    env.SConscript(env.GetExtraScripts("post"), exports="env")
  File "/Users/xxx/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/Users/xxx/.platformio/packages/tool-scons/scons-local-4.5.2/SCons/Script/SConscript.py", line 285:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/Users/xxx/Documents/PlatformIO/Projects/ESP_Buderus_KM271/merge_firmware.py", line 36:
    UPLOADERFLAGS=[

could you help me to resolve this?

[Jason2866]

@dewenni Try this one

Import("env")

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"
MERGED_BIN = "$BUILD_DIR/${PROGNAME}_merged.bin"
BOARD_CONFIG = env.BoardConfig()

def merge_bin(source, target, env):
    # The list contains all extra images (bootloader, partitions, eboot) and
    # the final application binary
    flash_images = env.Flatten(env.get("FLASH_EXTRA_IMAGES", [])) + ["$ESP32_APP_OFFSET", APP_BIN]

    # Run esptool to merge images into a single binary
    env.Execute(
        " ".join(
            [
                "$PYTHONEXE",
                "$OBJCOPY",
                "--chip",
                BOARD_CONFIG.get("build.mcu", "esp32"),
                "merge_bin",
                "--fill-flash-size",
                BOARD_CONFIG.get("upload.flash_size", "4MB"),
                "-o",
                MERGED_BIN,
            ]
            + flash_images
        )
    )

# Add a post action that runs esptoolpy to merge available flash images
env.AddPostAction(APP_BIN , merge_bin)

# Patch the upload command to flash the merged binary at address 0x0
env.Replace(
    UPLOADERFLAGS=[
        ]
        + ["0x0", MERGED_BIN],
    UPLOADCMD='"$PYTHONEXE" "$UPLOADER" $UPLOADERFLAGS',
)

[dewenni]

@valeros that works! Thank you. In the meanwhile I tried to help myself with chatGPT :-) As a result I could create a script that also works for me. => platformio_release.py

But your script is a bit more flexible because it reed the board and chip type.
What I also mention is, that the merged bin file of your script is about 4MB and the merged bin file of my script is 1.1MB what is roughly the sum of the single bin files.

Can you explain why?

[MIKHANYA]

@dewenni Try this one

Import("env")

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"
MERGED_BIN = "$BUILD_DIR/${PROGNAME}_merged.bin"
BOARD_CONFIG = env.BoardConfig()

def merge_bin(source, target, env):
    # The list contains all extra images (bootloader, partitions, eboot) and
    # the final application binary
    flash_images = env.Flatten(env.get("FLASH_EXTRA_IMAGES", [])) + ["$ESP32_APP_OFFSET", APP_BIN]

    # Run esptool to merge images into a single binary
    env.Execute(
        " ".join(
            [
                "$PYTHONEXE",
                "$OBJCOPY",
                "--chip",
                BOARD_CONFIG.get("build.mcu", "esp32"),
                "merge_bin",
                "--fill-flash-size",
                BOARD_CONFIG.get("upload.flash_size", "4MB"),
                "-o",
                MERGED_BIN,
            ]
            + flash_images
        )
    )

# Add a post action that runs esptoolpy to merge available flash images
env.AddPostAction(APP_BIN , merge_bin)

# Patch the upload command to flash the merged binary at address 0x0
env.Replace(
    UPLOADERFLAGS=[
        ]
        + ["0x0", MERGED_BIN],
    UPLOADCMD='"$PYTHONEXE" "$UPLOADER" $UPLOADERFLAGS',
)

It's cool that I found your discussion because I faced the same problem. The need for one firmware file for ESP32. This may be a little off-topic, but when compiling the code with your script file merge_firmware.py included in /src at the beginning I get a warning, could you help? Any thoughts ?

warning: Calling missing SConscript without error is deprecated.
Transition by adding must_exist=False to SConscript calls.
Missing SConscript 'merge_firmware.py'
File "C:\Users\myname\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 180, in <module>

[MIKHANYA]

@dewenni Try this one

Import("env")

APP_BIN = "$BUILD_DIR/${PROGNAME}.bin"
MERGED_BIN = "$BUILD_DIR/${PROGNAME}_merged.bin"
BOARD_CONFIG = env.BoardConfig()

def merge_bin(source, target, env):
    # The list contains all extra images (bootloader, partitions, eboot) and
    # the final application binary
    flash_images = env.Flatten(env.get("FLASH_EXTRA_IMAGES", [])) + ["$ESP32_APP_OFFSET", APP_BIN]

    # Run esptool to merge images into a single binary
    env.Execute(
        " ".join(
            [
                "$PYTHONEXE",
                "$OBJCOPY",
                "--chip",
                BOARD_CONFIG.get("build.mcu", "esp32"),
                "merge_bin",
                "--fill-flash-size",
                BOARD_CONFIG.get("upload.flash_size", "4MB"),
                "-o",
                MERGED_BIN,
            ]
            + flash_images
        )
    )

# Add a post action that runs esptoolpy to merge available flash images
env.AddPostAction(APP_BIN , merge_bin)

# Patch the upload command to flash the merged binary at address 0x0
env.Replace(
    UPLOADERFLAGS=[
        ]
        + ["0x0", MERGED_BIN],
    UPLOADCMD='"$PYTHONEXE" "$UPLOADER" $UPLOADERFLAGS',
)

It's cool that I found your discussion because I faced the same problem. The need for one firmware file for ESP32. This may be a little off-topic, but when compiling the code with your script file merge_firmware.py included in /src at the beginning I get a warning, could you help? Any thoughts ?

warning: Calling missing SConscript without error is deprecated.
Transition by adding must_exist=False to SConscript calls.
Missing SConscript 'merge_firmware.py'
File "C:\Users\myname\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 180, in <module>

Sorry, my mistake, needed to place .py file not in the /src folder, but simply in the root of the project.

[hamishcunningham]

I've been using this script for a while to produce a merged .bin and it works well, thanks!

The problem that it also breaks upload, so while pio run or etc. works, pio run -t upload causes this error:

Wrote 0x400000 bytes to file /home/ubuntu/project/.pio/build/adafruit_feather_esp32s3/firmware_merged.bin, ready to flash to offset 0x0
Configuring upload protocol...
AVAILABLE: cmsis-dap, esp-bridge, esp-builtin, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...

Warning! Please install `99-platformio-udev.rules`. 
More details: https://docs.platformio.org/en/latest/core/installation/udev-rules.html

Auto-detected: /dev/ttyACM1
Forcing reset using 1200bps open/close on port /dev/ttyACM1
Waiting for the new upload port...
Uploading .pio/build/adafruit_feather_esp32s3/firmware.bin
usage: esptool [-h]
               [--chip {auto,esp8266,esp32,esp32s2,esp32s3beta2,esp32s3,esp32c3,esp32c6beta,esp32h2beta1,esp32h2beta2,esp32c2,esp32c6}]
               [--port PORT] [--baud BAUD]
               [--before {default_reset,usb_reset,no_reset,no_reset_no_sync}]
               [--after {hard_reset,soft_reset,no_reset,no_reset_stub}]
               [--no-stub] [--trace] [--override-vddsdio [{1.8V,1.9V,OFF}]]
               [--connect-attempts CONNECT_ATTEMPTS]
               {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,merge_bin,get_security_info,version}
               ...
esptool: error: argument operation: invalid choice: '0x0' (choose from 'load_ram', 'dump_mem', 'read_mem', 'write_mem', 'write_flash', 'run', 'image_info', 'make_image', 'elf2image', 'read_mac', 'chip_id', 'flash_id', 'read_flash_status', 'write_flash_status', 'read_flash', 'verify_flash', 'erase_flash', 'erase_region', 'merge_bin', 'get_security_info', 'version')
*** [upload] Error 2

Any ideas? Tnx!

[hamishcunningham]

Removing the last (env.Replace) statement fixes it :) There's a bit of waste in the upload process (which doesn't use the merged bin), but I can live with that :)

[AllanOricil]

It must consider spiffs.bin as well

[DavidSchinazi]

I ran into the same issue and landed here. Thanks everyone for the tips! Based on that information, I was able to create a custom PlatformIO target that I think fits the bill. To enable it in your project, first add the following extra_scripts line to your platformio.ini file:

[env]
extra_scripts = merge-bin.py

Then in the same directory as the platformio.ini file, add this merge-bin.py:

#!/usr/bin/python3

# Adds PlatformIO post-processing to merge all the ESP flash images into a single image.

import os

Import("env", "projenv")

board_config = env.BoardConfig()
firmware_bin = "${BUILD_DIR}/${PROGNAME}.bin"
merged_bin = os.environ.get("MERGED_BIN_PATH", "${BUILD_DIR}/${PROGNAME}-merged.bin")

def merge_bin_action(source, target, env):
    flash_images = [
        *env.Flatten(env.get("FLASH_EXTRA_IMAGES", [])),
        "$ESP32_APP_OFFSET",
        source[0].get_abspath(),
    ]
    merge_cmd = " ".join(
        [
            '"$PYTHONEXE"',
            '"$OBJCOPY"',
            "--chip",
            board_config.get("build.mcu", "esp32"),
            "merge_bin",
            "-o",
            merged_bin,
            "--flash_mode",
            board_config.get("build.flash_mode", "dio"),
            "--flash_freq",
            "${__get_board_f_flash(__env__)}",
            "--flash_size",
            board_config.get("upload.flash_size", "4MB"),
            *flash_images,
        ]
    )
    env.Execute(merge_cmd)

env.AddCustomTarget(
    name="mergebin",
    dependencies=firmware_bin,
    actions=merge_bin_action,
    title="Merge binary",
    description="Build combined image",
    always_build=True,
)

Then all you need to do is run pio run -t mergebin to get a merged file.

If you want the file written to a specific location, pass in the MERGED_BIN_PATH environment variable:

MERGED_BIN_PATH=firmware-merged.bin pio run -t mergebin