Using all 16MB of flash in PlatformIO

[KamranAghlami]

Hi,

I made the board definition file for T-Display-S3 with following contents:

$HOME/.platformio/platforms/espressif32/boards/t-display-s3.json

{
  "build": {
    "arduino": {
      "ldscript": "esp32s3_out.ld",
      "partitions": "default_8MB.csv",
      "memory_type": "qio_opi"
    },
    "core": "esp32",
    "extra_flags": [
      "-DARDUINO_ESP32S3_DEV",
      "-DBOARD_HAS_PSRAM",
      "-DARDUINO_USB_MODE=1",
      "-DARDUINO_USB_CDC_ON_BOOT=1",
      "-DARDUINO_RUNNING_CORE=1",
      "-DARDUINO_EVENT_RUNNING_CORE=1"
    ],
    "f_cpu": "240000000L",
    "f_flash": "80000000L",
    "flash_mode": "qio",
    "hwids": [
      [
        "0x303A",
        "0x1001"
      ]
    ],
    "mcu": "esp32s3",
    "variant": "esp32s3"
  },
  "connectivity": [
    "wifi",
    "bluetooth"
  ],
  "debug": {
    "default_tool": "esp-builtin",
    "onboard_tools": [
      "esp-builtin"
    ],
    "openocd_target": "esp32s3.cfg"
  },
  "frameworks": [
    "arduino",
    "espidf"
  ],
  "name": "T-Display-S3",
  "upload": {
    "flash_size": "8MB",
    "maximum_ram_size": 327680,
    "maximum_size": 8388608,
    "require_upload_port": true,
    "speed": 921600
  },
  "url": "https://github.com/Xinyuan-LilyGO/T-Display-S3",
  "vendor": "LilyGO"
}

And this is my project configuration file:

$PROJECT_ROOT/.platformio.ini

[env:T-Display-S3]
platform = espressif32
board = t-display-s3
framework = arduino

monitor_speed = 115200

lib_deps =
  lvgl/lvgl @ ^8.3.2

build_flags =
  -DLV_CONF_INCLUDE_SIMPLE
  -Iinclude

Everything seems to work fine but notice that the flash size is defined as 8MB, so I tried making following changes:

"partitions": "default_16MB.csv"
"flash_size": "16MB"
"maximum_size": 16777216

Now the flash size is recognized as 16MB but "Upload Filesystem Image" option fails with following output:

 *  Executing task: $HOME\.platformio\penv\Scripts\platformio.exe run --target uploadfs --environment T-Display-S3 

Processing T-Display-S3 (platform: espressif32; board: t-display-s3; framework: arduino)
--------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/t-display-s3.html
PLATFORM: Espressif 32 (5.2.0) > T-Display-S3
HARDWARE: ESP32S3 240MHz, 320KB RAM, 16MB Flash
DEBUG: Current (esp-builtin) On-board (esp-builtin) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 @ 3.20005.220925 (2.0.5)
 - tool-esptoolpy @ 1.40201.0 (4.2.1)
 - tool-mkfatfs @ 2.0.1
 - tool-mklittlefs @ 1.203.210628 (2.3)
 - tool-mkspiffs @ 2.230.0 (2.30)
 - toolchain-riscv32-esp @ 8.4.0+2021r2-patch3
 - toolchain-xtensa-esp32s3 @ 8.4.0+2021r2-patch3
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 35 compatible libraries
Scanning dependencies...
Dependency Graph
|-- lvgl @ 8.3.2
|-- TFT_eSPI @ 2.4.71
|   |-- SPI @ 2.0.0
|   |-- SPIFFS @ 2.0.0
|   |   |-- FS @ 2.0.0
|   |-- FS @ 2.0.0
|   |-- LittleFS @ 2.0.0
|   |   |-- FS @ 2.0.0
Building in release mode
Building FS image from 'data' directory to .pio\build\T-Display-S3\spiffs.bin
/file00.txt
/file01.txt
/folder00/file00.txt
/folder00/file01.txt
/folder01/file00.txt
/folder01/file01.txt
Looking for upload port...
Auto-detected: COM6
Uploading .pio\build\T-Display-S3\spiffs.bin
esptool.py v4.2.1
Serial port COM6
Connecting...
Chip is ESP32-S3
Features: WiFi, BLE
Crystal is 40MHz
MAC: 68:b6:b3:21:87:dc
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Flash will be erased from 0x00c90000 to 0x00ffffff...
Compressed 3604480 bytes to 6805...
Writing at 0x00c90000... (100 %)

A fatal error occurred: Invalid head of packet (0x6F): Possible serial noise or corruption.
*** [uploadfs] Error 2
==================================================== [FAILED] Took 29.88 seconds ====================================================

 *  The terminal process "$HOME\.platformio\penv\Scripts\platformio.exe 'run', '--target', 'uploadfs', '--environment', 'T-Display-S3'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it. 

Does any one know what is going on here? Any kind of help is sincerely appreciated. 🙏

[mmMicky]

First you need to look at a chip with eight pins next to the chip. Check whether the silk screen inside is 25Q128. If so, it is 16MB FLASH. I have no clue as to how to fix the problem you raised.

[KamranAghlami]

Thanks for your reply,

The flash chip is 16MB as advertised, and with my board file the compilation and uploading takes place successfully, even the esp api call returns 16MB as flash size.

My issue is that the "Upload Filesystem Image" option doesn't work anymore.

[mmMicky]

Please check whether the space size of the file system meets your needs. Whether it is possible to modify the partition table appropriately to increase the space of the file system. When you compile it should show you how much space it takes up.

[KamranAghlami]

So I figured a way around the problem. This used to be the content of partitions file:

$HOME/.platformio/packages/framework-arduinoespressif32/tools/partitions/default_16MB.csv

# Name,   Type, SubType, Offset,   Size,     Flags,
nvs,      data, nvs,     0x9000,   0x5000,
otadata,  data, ota,     0xe000,   0x2000,
app0,     app,  ota_0,   0x10000,  0x640000,
app1,     app,  ota_1,   0x650000, 0x640000,
spiffs,   data, spiffs,  0xc90000, 0x370000,

I reduced size of spiffs partition by 1 (from 0x370000 to 0x36ffff).

Now it works as if It was overflowing from end of flash, I was expecting the default partitions file coming from PlatformIO to be tested many times by now and to work by default since it uses the same calculation on other 16MB partition files as well.

If any one were to shed some light on the issue I'd be relived.

[tyeth]

The only options I've found for 16mb in the arduino ide were for FATFS not spiffs. This is the offering under Partition Scheme if I am using an S3 with 16MB flash selected

and these are the partition files from platformio:

Maybe try large_spiffs_16MB.csv instead, it's contents are:

You also want to increase the ram size, 8mb if it's the chip I've got, but not sure if that maximum_ram_size is same as PSRAM, I've been copying the Espressif ESP32s3-Box as I think it's the 16mb S3 N8R8 or whatever with 16mb flash and 8mb ram.

-- Edit -- Max_ram_size is fine https://github.com/platformio/platform-espressif32/issues/486

[Jason2866]

It is probably a bug in esptool.py. Opened Issue is here

[KamranAghlami]

@Jason2866 Thanks for referencing the issue on esptool side, glad to see it being investigated. 🙏

[KamranAghlami]

I'm closing this since it doesn't seem to be a misconfiguration on LilyGO's side.

[Jason2866]

Bug in esptool.py is solved. You can use 16MB when using latest development version from github.

[KamranAghlami]

Excellent! Thanks for letting me know. 👍