Failure to upload filesystem image to ESP32-S3 (ESPTOOL-538)

CoryCharlton commented 1 year ago

Operating System

Windows 10

Esptool Version

v4.2.1 and v4.3

Python Version

Python 3.9.10

Chip Description

ESP32-S3

Device Description

Unexpected Maker FeatherS3 and TinyS3 (another user also reported the same issue with the ProS3): https://esp32s3.com/index.html#esp32s3_info

Hardware Configuration

Nothing is connected

How is Esptool Run

Windows terminal, PlatformIO (VS Code)

Full Esptool Command Line that Was Run

C:\Users\ccharlton\Downloads\esptool-v4.3-win64\esptool-v4.3-win64\esptool.exe --chip esp32s3 --port "COM10" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 16MB 13172736 .pio\build\debug\spiffs.bin

Esptool Output

esptool.py v4.3
Serial port COM10
Connecting....
Chip is ESP32-S3 (revision v0.1)
Features: WiFi, BLE
Crystal is 40MHz
MAC: f4:12:fa:42:03:2c
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00c90000 to 0x00ffffff...
Compressed 3604480 bytes to 6617...
Writing at 0x00c90000... (100 %)

A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.

More Information

No response

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

[X] I confirm I have read the troubleshooting guide.

dobairoland commented 1 year ago

@CoryCharlton I'm sorry but I cannot download the trace. It times out for me. Could you please just attach to the GH issue?

I wonder if adding the --no-compress argument helps?

CoryCharlton commented 1 year ago

@dobairoland Here's the trace:

trace.txt

Here's the output and trace from replacing -z with --no-compress (the result was the same failure):

output_no-compress.txt trace_no-compress.txt

pauldupuyjr commented 1 year ago

I am also seeing this error with an Unexpected Maker ProS3 board using PlatformIO 2.5.5 (and platform_packages = tool-esptoolpy@~1.40300.0) when executing the "Upload Filesystem Image" task:

A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.
*** [uploadfs] Error 2

In Arduino IDE 1.8.19 using arduino-esp32fs-plugin 2.0.7 the result is:

->Writing at 0x00910000... (100 %)
->
->A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.
SPIFFS Upload failed!

However, in PlatformIO, when I "Erase Flash", add "board_build.partitions = default.csv" to platformio.ini configuration, and execute "Upload Filesystem Image" it seems to succeed:

Building in release mode
Building FS image from 'data' directory to build\esp32s3\spiffs.bin
/upload-test.txt
Looking for upload port...
Using manually specified: COM15
Forcing reset using 1200bps open/close on port COM15
Waiting for the new upload port...
Uploading build\esp32s3\spiffs.bin
esptool.py v4.3
Serial port COM15
Connecting...
Chip is ESP32-S3 (revision v0.1)
Features: WiFi, BLE
Crystal is 40MHz
MAC: f4:12:fa:42:0c:20
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00290000 to 0x003fffff...
Compressed 1507328 bytes to 2855...
Writing at 0x00290000... (100 %)
Wrote 1507328 bytes (2855 compressed) at 0x00290000 in 11.4 seconds (effective 1061.8 kbit/s)...
Hash of data verified.

CoryCharlton commented 1 year ago

Interesting, using default.csv did seem to work although it doesn't use the entire 16MB flash so I don't have enough room for my files.

Here's the partition layouts I was using:

default_16MB.csv * My board uses this by default large_spiffs_16MB.csv no_ota.csv

And for comparison the default.csv that works:

default.csv

CoryCharlton commented 1 year ago

Looks like the issue is possibly related to the default 16MB partition tables. I used the following custom partition table and it worked fine:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     36K,     20K,
factory,  app,  factory, 64K,     4M,
spiffs,   data, spiffs,  ,        10M,

~~Looks like this is actually an issue for https://github.com/espressif/arduino-esp32~~ (or is it?)

pauldupuyjr commented 1 year ago

Thanks @CoryCharlton for providing a working partition table example and for helping to solve this problem! Cheers,

CoryCharlton commented 1 year ago

So this might still be an issue on the esptool side (based on this comment).

The specific issue appears to be related to the partition table completely filling up the 16MB flash. For instance I broke the working partition table mentioned here by extending the spiffs partition to ~12MB in order to completely use all available space.

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBF0000,

But I can make it work again by shrinking the spiffs partition by 1KB:

# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBEFC00,

dobairoland commented 1 year ago

@CoryCharlton @SuGlider I don't think this is an issue with esptool nor Arduino.

If you take a look at the trace then you can see at the end that some serial data left in the buffer:

    70695f666c617368 2e6320323730200d | pi_flash.c 270 .
    0a                                | .

Invalid head of packet (0x6F) - 0x6f is ASCII for o. So actually serial communication is interrupted by an assertion from ROM (opi_flash.c, I guess 270 is the line number).

Honestly, I don't know what esptool could do about this. There is not enough information to print something more useful.

I'll try to ask around.....

dobairoland commented 1 year ago

It was pointed out to me that this is a duplicate of https://github.com/espressif/esptool/issues/745.

dobairoland commented 1 year ago

I'm able to write 16kB to the end of the flash region on an ESP32-S3 with 8MB flash as opposed to 16MB reported here.

`--> python -m esptool write_flash --no-compress 0x7fc000 ~/Work/zeroes.bin
esptool.py v4.3
Found 2 serial ports
Serial port /dev/ttyUSB0
Connecting...
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (revision v0.0)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e0:11:78
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x007fc000 to 0x007fffff...
Wrote 16384 bytes at 0x007fc000 in 1.6 seconds (84.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
.-(~/esp/esp-idf)----------------------------------------------------------------------------------------------------------------------------------------------------------(dragon@RolandPC)-
`--> python -m esptool write_flash --compress 0x7fc000 ~/Work/zeroes.bin
esptool.py v4.3
Found 2 serial ports
Serial port /dev/ttyUSB0
Connecting....
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (revision v0.0)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e0:11:78
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x007fc000 to 0x007fffff...
Compressed 16384 bytes to 39...
Wrote 16384 bytes (39 compressed) at 0x007fc000 in 0.2 seconds (effective 630.0 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

skieast commented 1 year ago

I looked at this a bit and I think the issue is in the flasher_stub code. The routines that read_flash and write_flash change the spi method when above the first 16M flash. I think that this is happening 1 byte early. Meaning in the first 16M of flash. When I change the lines of code that check I am able to read and write to then end. For example, commented out line is original.

    #if defined(ESP32S3)
      if (addr + len-1 > 0x00ffffff)
      // if (addr + len > 0x00ffffff)
        res = SPIRead4B(1, SPI_FLASH_FASTRD_MODE, addr, buf, n);
      else
        res = SPIRead(addr, (uint32_t *)buf, n);
    #else
      res = SPIRead(addr, (uint32_t *)buf, n);
    #endif // ESP32S3

Evan5935 commented 1 year ago

So this might still be an issue on the esptool side (based on this comment).

The specific issue appears to be related to the partition table completely filling up the 16MB flash. For instance I broke the working partition table mentioned here by extending the spiffs partition to ~12MB in order to completely use all available space.
# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBF0000,
But I can make it work again by shrinking the spiffs partition by 1KB:
# Name,   Type, SubType,  Offset,   Size,  Flags
nvs, data, nvs, 0x9000, 0x5000,
factory, app, factory, 0x10000, 0x400000,
spiffs, data, spiffs, 0x410000, 0xBEFC00,

can I get you to make me a partition table like the one you found worked for you? i have no idea what I am doing, but just need a 2mb app 12 mb fatfs or spiffs or whatever so I can get a project running on my ProS3 by UM

Thank You!

CoryCharlton commented 1 year ago

can I get you to make me a partition table like the one you found worked for you? i have no idea what I am doing, but just need a 2mb app 12 mb fatfs or spiffs or whatever so I can get a project running on my ProS3 by UM

Thank You!

Here's a partition table with room for a 4MB app and a ~12MB SPIFFS partition: partitions.csv

Place the partitions.csv in your project directory next to the platformio.ini file. Edit your platformio.ini to add the following line:

board_build.partitions = partitions.csv

Complete platformio.ini example from one of my projects:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env]
platform = espressif32
board = um_feathers3
board_build.partitions = partitions.csv
framework = arduino
monitor_port = COM14
upload_port = COM14
lib_deps = earlephilhower/ESP8266Audio@^1.9.7

[env:debug]
build_type = debug
build_flags = -DARDUINO_USB_MODE=1 -DCORE_DEBUG_LEVEL=5

Then use the PlatformIO commands to perform the following steps: Erase Flase, Build Filesystem Image, Upload Filesystem Image

I'm on the Unexpected Maker Discord as Roland Deschain if you run in to any issues.

beckmx commented 1 year ago

@skieast hello there, I looked for those lines and I found them with .c extension, is that correct? I mean I changed those values but I got the same response, should I recompile esptool?

skieast commented 1 year ago

You will need to recompile the flasher_stub routines that are in the flasher_stub folder. That is C code, the instructions are in the folder. Consists of downloading the ESP8266 compiler and sdk and the ESP32 idf. I don't think its really worth it at this time because you are only limited to not being able to read/write the last byte of 16M flash.

From README.md

# To Build

If you want to build the stub to test modifications or updates, here's how:

* You will need an ESP8266 gcc toolchain (xtensa-lx106-elf-) and toolchains for ESP32 and later chips (xtensa-esp32-elf-, riscv32-esp-elf-) on your PATH.

* Set the environment variables SDK_PATH to the path to an ESP8266 IoT NON-OS SDK directory (last stub was built with SDK v1.5.1).

* Set the environment variable IDF_PATH to the path to an ESP-IDF directory.

* Set any other environment variables you'd like to override in the Makefile.

* To build type `make`

Activating an ESP-IDF environment takes care of most of these steps (only the ESP8266 gcc toolchain has to be manually added to PATH).

# To Test

To test the built stub, you can run `make embed`, which will update the stubs in `esptool.py` to the newly compiled ones. Or there are some convenience wrappers to make testing quicker to iterate on:

* Running `esptool_test_stub.py` is the same as running `esptool.py`, only it uses the just-compiled stubs from the build directory.

* Running `run_tests_with_stub.py` is the same as running `test/test_esptool.py`, only it uses the just-compiled stubs from the build directory.

beckmx commented 1 year ago

Well I couldnt get it to compile but!! @CoryCharlton sample partition definately worked for me

Jason2866 commented 1 year ago

@beckmx You can try my fork https://github.com/Jason2866/esptool I did the change and let github actions build the stub.

henrygab commented 1 year ago

I don't think its really worth it at this time because you are only limited to not being able to read/write the last byte of 16M flash.

I do hope this gets reviewed, validate, and merged ... because I have a few ESP32-S3 with 32MB of flash, and I've had to artificially limit use of those to <16MB.

CoryCharlton commented 1 year ago

Confirming that this commit resolves my issue with 16M flash.

I was going to close this issue but it looks like you're keeping it open for an issue with 32M flashes that is not the same as my original issue.

Feel free to close this if I am mistaken.

espressif / esptool