esptool <4.8 builds invalid ESP32 NuttX firmware image silently. (ESPTOOL-939)

cederom commented 1 month ago

Operating System

FreeBSD, Linux (and probably others).

Esptool Version

4.7.0.

Python Version

3.8, 3.11.

Chip Description

ESP32.

Device Description

ESP32-DevKitC with ESP32-WROOM-32E.

Hardware Configuration

ESP32-DevKitC with ESP32-WROOM-32E.
NuttX ~12.7.0 with esp32-devkitc:nsh configuration.

How is Esptool Run

NuttX build script (see logs below).

Full Esptool Command Line that Was Run

./tools/configure.sh -B esp32-devkitc:nsh; gmake flash

Esptool Output

This is for version 4.7.0:

All seems to build fine, bootloader is created, final firmware image is created, flashed, but hangs on boot.
4.7.0 is the system package version on FreeBSD. Other OS / users may have older versions too and that leads to invalid firmware creation.

% /usr/bin/time -h ./tools/configure.sh -B esp32-devkitc:nsh
        3,89s real              1,77s user              2,41s sys

% /usr/bin/time -h gmake -j8
Create version.h
Cloning Espressif HAL for 3rd Party Platforms
Clone: chip/esp-hal-3rdparty LN: platform/board to /XXX/nuttx-apps.git/platform/dummy
Register: nsh
Register: sh
Espressif HAL for 3rd Party Platforms: 20690e67695f0a8170a19ec99e2e9a13b620e94d
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmp
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.7.0
Creating esp32 image...
ROM segments hidden - only RAM segments are visible to the ROM loader!
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
        39,74s real             1m31,54s user           1m2,00s sys

%  /usr/bin/time -h gmake flash
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmpLD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.7.0
Creating esp32 image...
ROM segments hidden - only RAM segments are visible to the ROM loader!
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
esptool.py -c esp32 -p /dev/cuaU0 -b 115200  write_flash -fs detect -fm dio -ff 40m 0x1000 nuttx.bin
esptool.py v4.7.0
Serial port /dev/cuaU0
Connecting.........
Chip is ESP32-D0WD-V3 (revision v3.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: XXX
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00033fff...
Compressed 206984 bytes to 72429...
Wrote 206984 bytes (72429 compressed) at 0x00001000 in 6.7 seconds (effective 247.0 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
        15,25s real             3,07s user              3,17s sys

% cu -l /dev/cuaU0 -s 115200
Connected
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffb1f40,len:1388
ho 0 tail 12 room 4
load:0x40080000,len:20320
entry 0x40082650
*** Booting NuttX ***
dram: lma 0x00001020 vma 0x3ffb1f40 len 0x56c    (1388)
iram: lma 0x00001594 vma 0x40080000 len 0x4f60   (20320)
padd: lma 0x00006508 vma 0x00000000 len 0xaaf0   (43760)
imap: lma 0x00011000 vma 0x400e0000 len 0x10d78  (68984)
padd: lma 0x00021d80 vma 0x00000000 len 0xf298   (62104)
dmap: lma 0x00031020 vma 0x3f400020 len 0x2848   (10312)
total segments stored 6

>>> NO NSH HERE DEVICE HANGS <<<

This is for version 4.8.1:

As above all build, firmware image creation, flashing works fine, but firmware works as expected.
This is done in Python venv where esptool 4.8.1 is installed and used in separation from the system packages.

(venv3.9embedded) pip install -U esptool

(venv3.9embedded) /usr/bin/time -h gmake -j8
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmpLD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.8.1
Creating esp32 image...
Image has only RAM segments visible. ROM segments are hidden and SHA256 digest is not appended.
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
        4,19s real              2,96s user              3,44s sys

(venv3.9embedded) /usr/bin/time -h gmake flash
        15,26s real             3,03s user              3,24s sys

(venv3.9embedded) cu -l /dev/cuaU0 -s 115200
Stale lock on cuaU0 PID=4016... overriding.
Connected
ts Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffb1f40,len:1388
ho 0 tail 12 room 4
load:0x40080000,len:20320
entry 0x40082650
*** Booting NuttX ***
dram: lma 0x00001020 vma 0x3ffb1f40 len 0x56c    (1388)
iram: lma 0x00001594 vma 0x40080000 len 0x4f60   (20320)
padd: lma 0x00006508 vma 0x00000000 len 0x9af0   (39664)
imap: lma 0x00010000 vma 0x400e0000 len 0x10d78  (68984)
padd: lma 0x00020d80 vma 0x00000000 len 0xf298   (62104)
dmap: lma 0x00030020 vma 0x3f400020 len 0x2848   (10312)
total segments stored 6

NuttShell (NSH) NuttX-12.7.0
nsh> uname -a
NuttX 12.7.0 10e44f8915-dirty Oct  4 2024 22:22:13 xtensa esp32-devkitc
nsh> free
                 total       used       free    maxused    maxfree  nused  nfree
      Umem:     323568       6424     317144       6800     186696     22      3

Initially version 4.3 was installed but that returned clear build error:

% /usr/bin/time -h ./tools/configure.sh -B esp32-devkitc:nsh
        3,92s real              1,98s user              2,23s sys

% /usr/bin/time -h gmake -j8
 /usr/bin/time -h gmake -j8
Create version.h
Cloning Espressif HAL for 3rd Party Platforms
Clone: chip/esp-hal-3rdparty LN: platform/board to /XXX/nuttx-apps.git/platform/dummy
Register: nsh
Register: sh
Espressif HAL for 3rd Party Platforms: 20690e67695f0a8170a19ec99e2e9a13b620e94d
CPP:  /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld-> /XXX/nuttx.git/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld.tmpLD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
usage: esptool [-h] [--chip {auto,esp8266,esp32,esp32s2,esp32s3beta2,esp32s3,esp32c3,esp32c6beta,esp32h2beta1,esp32h2beta2,esp32c2,esp32c6}] [--port PORT] [--baud BAUD] [--before {default_reset,usb_reset,no_reset,no_reset_no_sync}] [--after {hard_reset,soft_reset,no_reset,no_reset_stub}] [--no-stub] [--trace]
               [--override-vddsdio [{1.8V,1.9V,OFF}]] [--connect-attempts CONNECT_ATTEMPTS]
               {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,merge_bin,get_security_info,version} ...
esptool: error: unrecognized arguments: --ram-only-header
gmake: *** [tools/Unix.mk:558: nuttx] Error 2
        40,96s real             1m32,23s user           1m1,82s sys

% pip show esptool
Name: esptool
Version: 4.3
Summary: A serial utility to communicate & flash code to Espressif chips.
Home-page: https://github.com/espressif/esptool/
Author: Fredrik Ahlberg (themadinventor) & Angus Gratton (projectgus) & Espressif Systems
Author-email:
License: GPLv2+
Location: /XXX/venv3.9embedded/lib/python3.9/site-packages
Requires: bitstring, cryptography, ecdsa, pyserial, reedsolo
Required-by: #N/A

More Information

This problem was found during NuttX 12.7.0-RC0 pre-release testing.
CI scripts on NuttX repo were updated to use esptool 4.8 dev.
This is probably related to new version of ESP32 bootloader from ESP-IDF?
We would like to know why build fails silently on versions <4.8 and what would be the way to approach and fix.
- Perfect solution points to automatic adaptation of bootloader version to available esptool.
- Is there a esptool - bootloader compatibility matrix we could implement to adapt bootloader release/version/tag/commit to available esptool version?
- Should we use 4.8 only and error build on older versions?
Internal NuttX problem report is here: https://github.com/apache/nuttx/issues/13824/
We would like to fix the problem very soon before 12.7.0 is released not to discourage ESP32 users where things do not work out of the box with a broken firmware.
Any hints welcome, thank you :-)

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

[X] I confirm I have read the troubleshooting guide.

radimkarnis commented 1 month ago

Hello @cederom,

Taking a brief look at this - I believe this is a result of some of the recent fixes (https://github.com/espressif/esptool/commit/4c5874a3c497ce3bd2e3289d5e367db4f4ac89b5, https://github.com/espressif/esptool/commit/4394a658c40b73cc6d593797a42b665e68f84cf7, https://github.com/espressif/esptool/commit/e87cc3ec62c08046503092b8f58bf62ebadcbcac) in the --ram-only-header option in v4.8.

These changes have been implemented by the Zephyr team. Hopefully, they can shed some more light on this - cc @almir-okato, @sylvioalves PTAL.

what would be the way to approach and fix

Currently, I can only recommend that you use the latest esptool. We don't plan to backport any of these changes and they are still somewhat "experimental".

Sorry, I don't have more answers at the moment, but I will look into this!

cederom commented 1 month ago

Thank you @radimkarnis :-) Question is why it worked before^TM ? :D

As we are close to a release probably we will throw build error if esptool <4.8 in that case.

Do you know if this kind of "simple boot" can be achieved without ram only flags? Or any other way to make things work again with older esptool if the fixes are not backported to older releases? Or 4.8 is the only way to go?

sylvioalves commented 1 month ago

@cederom, "Simple Boot" bootloader was created to override esp-idf bootloader dependency (used in both NuttX and Zephyr RTOS). It does not require esp-idf partition-table and can run integrated with the application as a single bin. No need of multiple .bin files as needed in previous implementation.
Allowing such custom bootloader required changes in the way esptool merges sections (and other stuff).

Unless there is an issue with esptool >=4.8, I guess this can be closed. NuttX team will handle it in there.

cederom commented 1 month ago

Perfect! Thank you @radimkarnis @sylvioalves :-)

espressif / esptool