Bootloader and partition table are not flashed if build and upload are performed in two steps

platformio / platform-espressif32

Espressif 32: development platform for PlatformIO

https://registry.platformio.org/platforms/platformio/espressif32

Apache License 2.0

894 stars 600 forks source link

Bootloader and partition table are not flashed if build and upload are performed in two steps #1142

Open 5p4k opened 1 year ago

5p4k commented 1 year ago

We have an ESP32 project with CI/CD set up in such a way that in one stage we build the firmware, and in a separate stage we upload that to a device and run the tests. Therefore what happens is equivalent to running the following commands:

# instead of pio test -vv
pio test -vv --without-uploading --without-testing
pio test -vv --without-building

or for a regular application:

# instead of pio run -t upload -t monitor
pio run
pio run -t nobuild -t upload
pio run -t monitor

This workflow appears to work, but in fact is broken because at the upload stage, it will exclusively flash the main firmware, without transferring bootloader, partition table, and any other partition. This goes undetected unless of course the main app partition is moved from its original offset.

Also, note that pio run -t nobuild -t monitor also does not work, the monitor target is not started.

MVE

example-partition-table.tar.gz

Consider the attached application, which does absolutely nothing, but moves the factory partition from 0x10000 to 0x20000. On a regular pio run -t upload, it produces the following output:

Configuring flash size...
Flash will be erased from 0x00001000 to 0x00007fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x00020000 to 0x00059fff...
Compressed 26368 bytes to 16444...

0x00001000: this is the hardcoded offset of the bootloader, corresponds to .pio/build/esp32/bootloader.bin
0x00008000: this is the hardcoded offset of the partition table, corresponds to .pio/build/esp32/partitions.bin
0x00020000: this is the actual application, at the offset requested, corresponds to .pio/build/esp32/firmware.bin

Now if we try to separate the two commands, and run

# make sure to erase the flash first so that we do not have the old partition data under the current app
pio run -t erase
pio run
pio run -t nobuild -t upload
# Cannot do -t nobuild -t monitor, it does not work (bug?)
pio run -t monitor

We see as output an incorrect offset for the firmware:

Configuring flash size...
Flash will be erased from 0x00010000 to 0x00049fff...
Compressed 237264 bytes to 99067...

That is the offset for the default partition table, and is missing flashing the bootloader and the partitions table proper. Indeed when we start the monitor, the ESP32 resets indefinitely with the message

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371 
ets Jun  8 2016 00:22:57

I am not sure whether this is a ESP-IDF issue or a platformio issue, but if you can give directions on how to determine that I can do some debugging.

PlatformIO Core, version 6.1.8a5

 - framework-espidf @ 3.50002.230512 (5.0.2) 
 - tool-cmake @ 3.16.4 
 - tool-esptoolpy @ 1.40501.0 (4.5.1) 
 - tool-mkfatfs @ 2.0.1 
 - tool-mklittlefs @ 1.203.210628 (2.3) 
 - tool-mkspiffs @ 2.230.0 (2.30) 
 - tool-ninja @ 1.7.1 
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0) 
 - toolchain-xtensa-esp32 @ 11.2.0+2022r1

5p4k commented 1 year ago

I just realized I can increase verbosity. The command line passed by pio run to esptool.py indeed differs:

# with pio run -t upload -vvv
~/.platformio/penv/bin/python \
    ~/.platformio/packages/tool-esptoolpy/esptool.py \
    --chip esp32 \
    --port /dev/ttyUSB0 \
    --baud 460800 \
    --before default_reset \
    --after hard_reset write_flash \
    -z \
    --flash_mode dio \
    --flash_freq 40m \
    --flash_size 4MB \
    0x1000 ~/Desktop/test/.pio/build/esp32/bootloader.bin \
    0x8000 ~/Desktop/test/.pio/build/esp32/partitions.bin \
    0x20000 .pio/build/esp32/firmware.bin

# with pio run -t nobuild -t upload -vvv
~/.platformio/penv/bin/python \
    ~/.platformio/packages/tool-esptoolpy/esptool.py \
    --chip esp32 \
    --port /dev/ttyUSB0 \
    --baud 460800 \
    --before default_reset \
    --after hard_reset write_flash \
    -z \
    --flash_mode dio \
    --flash_freq 40m \
    --flash_size 4MB \
    0x10000 .pio/build/esp32/firmware.bin

5p4k commented 1 year ago

Upon further inspection, the espidf build framework parses the partitions table and sets the environment variables FLASH_EXTRA_IMAGES

https://github.com/platformio/platform-espressif32/blob/a47ecbfca0654f57004052cd90ce5f97214c40fa/builder/frameworks/espidf.py#L1497

as well as ESP32_APP_OFFSET

https://github.com/platformio/platform-espressif32/blob/a47ecbfca0654f57004052cd90ce5f97214c40fa/builder/frameworks/espidf.py#L1614

This does not happen when the target is nobuild, because it does not call env.BuildProgram(). I believe that nobuild should

parse the partitions table and at least set ESP32_APP_OFFSET
depend on partitions.bin as well as bootloader.bin
correctly set FLASH_EXTRA_IMAGES as done in espidf.py

However I am not sure how the code should be arranged to do this.

Jason2866 commented 1 year ago

@LizardM4 We had a similar problem, since we use a factory firmware at address 0x10000 and the firmware is at 0xe0000. To solve i modified the framework and we use a pio python script after the build process which combines all parts to a "factory" image which can be flashed at address 0x0. The modified forked framework is here https://github.com/tasmota/platform-espressif32 and the script which generates the combined firmware is here https://github.com/arendst/Tasmota/blob/development/pio-tools/post_esp32.py furthermore we use modified boards.json where we can define to which address the parts should go. Seen here https://github.com/arendst/Tasmota/blob/development/boards/esp32.json

You are completely correct with your finding with the handling of the partition offset. It is fixed and does not allow flexible use.

5p4k commented 1 year ago

Thank you @Jason2866 , this might come in handy especially later during development, if we want to release pre-built packaged firmware. For now forking the platform and board info would be too much effort, however I managed to work around this by writing this pre-script.

The script checks if nobuild is among the targets (this is the case when pio test is called with the option --without-building), and if so, pre-populates the SCons environment variables ESP32_APP_OFFSET and FLASH_EXTRA_IMAGES.

It does more or less the same as espidf.py:

uses $BUILD_DIR/config/sdkconfig.json to obtain the partition table address.
uses the board type to determine the bootloader address.
uses parttool.py from ESP-IDF to obtain the offset of the boot partition where firmware.elf should be flashed
uses parttool.py to detect if an otadata partition is present (in our project we use ota, and this generates one further ota_data_initial.bin partition file which has to be flashed too)
updates the variables above in env, flashing bootloader.bin, partitions.bin, ota_data_initial.bin if present, and setting the correct offset for firmware.bin
since it can only handle bootloader, partitions table and ota data, it also scans the partitions json for any other custom user entry, and warns the user if any is found (look for ensure_partitions: warning)

For anyone else having this issue,

download the script and add it next to platformio.ini
add in your environment section extra_scripts = pre:ensure_partitions.py
if you need to choose which artifacts to keep across build stages, you need
1. .pio/build/${PIO_ENV}/*.bin
2. .pio/build/${PIO_ENV}/config/sdkconfig.json
3. .pio/build/project.checksum (this might not be strictly necessary, you would have to try, it might trigger a rebuild)

For anyone looking to fix this properly,

the script could be a starting point (I tried to keep it tidy). At the very least something like that should be added to main.py here in platformio-espressif32.

However, for that to be a "complete" solution, it should handle also other data partitions type that the user might want to flash. I suppose that it should also add a dependency from target_firm on the bin files with the partitions data, and some code restructuring between main.py and framework/espidf.py might be necessary to avoid code duplication.

Another alternative worth considering is to use the file .pio/build/${PIO_ENV}/flasher_args.json. This seems to be created by ESP-IDF and contains offsets and bin files. However the bin file location differs (e.g. bootloader/bootloader.bin, or ”app”.bin instead of firmware.bin), and the offset of the main app does not seem to be correct. If these aspects can be fixed, flasher_args.json could drive the upload target independently on whether the build is taking place or not.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. Please provide more details or it will be closed if no further activity occurs. Thank you for your contributions.

5p4k commented 1 year ago

Just to make sure this doesn't go stale, it's very much still a problem for which I posted a workaround. It needs however developers' attention.

valeros commented 1 year ago

Hi @LizardM4, thanks for pointing it out. In a nutshell, this behavior is somewhat expected, mainly because the framework build script (that includes FLASH_EXTRA_IMAGES) is not executed when you specify the nobuild target, hence those additional binaries are not added to the upload command. As for the offset value, there is a special configuration option board_upload.offset_address = 0xVALUE that allows using an arbitrary offset value for the application binary.

Overall, I'd recommend you to stick to the default workflow that includes build step until this issue is resolved.

Doohan23 commented 11 months ago

I think I'm suffering this issue too.

Basically when I upload the "firmware.bin" file using "pio run -t nobuild -t upload - v" command, it is loaded in the board (an ESP32):

...
Writing at 0x000dce8c... (100 %)
Wrote 849824 bytes (564593 compressed) at 0x00010000 in 13.6 seconds (effective 501.7 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

but when the board restarts, it executes the previous program, not the new one.

If I upload the program using "upload" task (so the program is built and upload), it works properly.

Last time I used this (on March or so) I could upload a firmware without built it without problem

Now I'm using Visual Code 1.83.1 and Platformio Core 6.1.11-Home 3.4.4 THe board is an ESP32-WROOM-32 and, in Platformio, I use it as "AZ-Delivery ESP-32 Dev Kit C V4" with "Espressif 32 v5.2.0"

justoke commented 10 months ago

I've got a code base about 2 years old which has been working fine. I just installed fresh install of PlatformIO and VS Code on a new Windows PC. Although the project builds and outputs the bin file, the command line has all the supporting files as usual, the binary once released to an ESP32 Dev module fails to run and just emits the RTCWDT_RTC_RESET in a continuous cycle. Only when I take a binary built on the old PC and flash the device does it work as it did before:

I am using min_spiffs.csv

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1E0000,
app1,     app,  ota_1,   0x1F0000,0x1E0000,
spiffs,   data, spiffs,  0x3D0000,0x20000,
coredump, data, coredump,0x3F0000,0x10000,

Command line release

"C:\Users\sb\.platformio\penv\Scripts\python.exe" 
"C:\Users\sb\.platformio\packages\tool-esptoolpy\esptool.py" --chip esp32 --port "COM3" 
--baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio 
--flash_freq 40m --flash_size 4MB 
0x1000 E:\PlatformIO\Projects\kctech\.pio\build\esp32dev_4M\bootloader.bin 
0x8000 E:\PlatformIO\Projects\kctech\.pio\build\esp32dev_4M\partitions.bin 
0xe000 C:\Users\sb\.platformio\packages\framework-arduinoespressif32\tools\partitions\boot_app0.bin 
0x10000 .pio\build\esp32dev_4M\firmware1.bin

Build and Release

Processing esp32dev_4M (platform: espressif32; board: esp32dev; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (6.4.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 @ 3.20011.230801 (2.0.11)
 - tool-esptoolpy @ 1.40501.0 (4.5.1)
 - toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch5
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ strict
Building in release mode
Retrieving maximum program size .pio\build\esp32dev_4M\firmware.elf
Checking size .pio\build\esp32dev_4M\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  17.8% (used 58416 bytes from 327680 bytes)
Flash: [========= ]  85.4% (used 1679413 bytes from 1966080 bytes)
Configuring upload protocol...
AVAILABLE: cmsis-dap, esp-bridge, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...
Using manually specified: COM3
Uploading .pio\build\esp32dev_4M\firmware.bin
esptool.py v4.5.1
Serial port COM3
Connecting....
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: d4:d4:da:40:a1:98
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00001000 to 0x00005fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00010000 to 0x001abfff...
Compressed 17536 bytes to 12203...
Writing at 0x00001000... (100 %)
Wrote 17536 bytes (12203 compressed) at 0x00001000 in 0.7 seconds (effective 188.4 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 146...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (146 compressed) at 0x00008000 in 0.3 seconds (effective 82.2 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 47...
Writing at 0x0000e000... (100 %)
Wrote 8192 bytes (47 compressed) at 0x0000e000 in 0.3 seconds (effective 249.4 kbit/s)...
Hash of data verified.
Compressed 1685168 bytes to 1178648...
Writing at 0x00010000... (1 %)
.....
Writing at 0x001a605a... (100 %)
Wrote 1685168 bytes (1178648 compressed) at 0x00010000 in 32.9 seconds (effective 409.5 kbit/s)...
Hash of data verified.

Reset cycle ad infinitum

ets Jun 8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
ets Jun  8 2016 00:22:57

I have tried different boards, erasing the flash and using the Espressif Download tool as an alternative to PlatformIO just to try but the problem seems to be that the offsets in the binary are incorrect or something has changed that seems to have broken the release binary.

Any suggestions greatly appreciated. I'm stuck on this one for the last 24 hours without being any closer to a resolution.

Jason2866 commented 10 months ago

@justoke The problem you encounter is a different one. Without providing the full setup or better a link to the whole project no help possible.