v3.1.1 LittleFS not flashing or reading

dimecho commented 1 year ago

Basic Infos

[x] This issue complies with the issue POLICY doc.
[x] I have read the documentation at readthedocs and the issue is not addressed there.
[x] I have tested that the issue is present in current master branch (aka latest git).
[x] I have searched the issue tracker for a similar issue.
[x] If there is a stack dump, I have decoded it.
[x] I have filled out all fields below.

Platform

Hardware: [ESP-12]
Core Version: [3.1.0/3.1.1]
Development Env: [Arduino IDE]
Operating System: [Windows|MacOS]

Settings in IDE

Module: [Generic ESP8266 Module|Nodemcu]
Flash Mode: [dout]
Flash Size: [4MB/2MB]
lwip Variant: [v2 Lower Memory]
Reset Method: [nodemcu]
Flash Frequency: [40Mhz]
CPU Frequency: [80Mhz]
Upload Using: [OTA|SERIAL]
Upload Speed: [115200]

Problem Description

LittleFS not flashing. Tried different methods ESP8266HTTPUpdateServer(Web-browser /update), OTA (WiFi with #include ), esptool.py (Serial 115200). Both signed and unsigned binary. The upload worked in v3.0.2!

Broken in v3.1.0 and v3.1.1.

Some more troubleshooting, if upload FS using v3.0.2 (works for reading as well) then flash the sketch with v3.1.1 using (Erase Flash: Only Sketch) FS not readable.

I suspect the flashing might quietly work it is just no readable. Did the starting address change? Or did LittleFS block-size change?

Debug Messages

v3.0.2 (Working)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v0009afa0
~ld

v3.1.0/3.1.1 (Not Working)

load 0x4010f000, len 3424, room 16 
tail 0
chksum 0x2e
load 0x3fff20b8, len 40, room 8 
tail 0
chksum 0x2b
csum 0x2b
v0009aa80
~ld

LittleFS build

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 640000 ./build/flash-littlefs.bin

mcspr commented 1 year ago

I suspect the flashing might quietly work it is just no readable. Did the starting address change? Or did LittleFS block-size change?

By 'Debug messages' we mean you have to use 'Debug level' and 'Debug port' menu entries, what we see above is just the usual bootloader stuff. That would likely answer the first question. Printing filesystem info from the working example would answer the second one, if you suspect FS setup issues.

dimecho commented 1 year ago

Did more investigation using Hex dump "fs region"

esptool.py --port $serial --baud 115200 read_flash 0x200000 0x300000 dump.bin

I think the issue is with "write" after all. Comparing good to bad HEX, littlefs headers are missing? at 0x000000 and 0x002000 the rest of the content after 0x004000 looks identical (good).

littlefs-bad-headers

mcspr commented 1 year ago

Where do we get these offsets from? Is it our .ld or something custom?

Assuming that you have looked at updater logs and nothing seems suspicious, you can also double-check raw addresses from the resulting .elf (...and so there is no need to flash & boot the firmware)

LittleFS, by default, uses these four for its constructor. IDE config also has those, see boards.txt for lines starting with e.g. menu.eesz.4M1M=...; these are used when you are flashing or erasing

Trying to build with arduino-cli and 4m1m split

> arduino-cli compile -b esp8266com:esp8266:d1_mini:eesz=4M1M --show-properties | grep -Pe '^build.(flash_ld|path|core)'
build.core=esp8266
build.core.path=/home/runner/Arduino/hardware/esp8266com/esp8266/cores/esp8266
build.flash_ld=eagle.flash.4m1m.ld
build.path=/tmp/arduino-sketch-6E47FB208C9EABB9A285B538F5914278

> nm /tmp/arduino-sketch-6E47FB208C9EABB9A285B538F5914278/arduino8826.ino.elf | grep _FS_
00002000 A _FS_block
405fa000 A _FS_end
00000100 A _FS_page
40500000 A _FS_start

> grep -e '^d1_mini.menu.eesz.4M1M' /home/runner/Arduino/hardware/esp8266com/esp8266/boards.txt
d1_mini.menu.eesz.4M1M=4MB (FS:1MB OTA:~1019KB)
d1_mini.menu.eesz.4M1M.build.flash_size=4M
d1_mini.menu.eesz.4M1M.build.flash_ld=eagle.flash.4m1m.ld
d1_mini.menu.eesz.4M1M.build.spiffs_pagesize=256
d1_mini.menu.eesz.4M1M.build.rfcal_addr=0x3FC000
d1_mini.menu.eesz.4M1M.build.spiffs_start=0x300000
d1_mini.menu.eesz.4M1M.build.spiffs_end=0x3FA000
d1_mini.menu.eesz.4M1M.build.spiffs_blocksize=8192

> cat /home/runner/Arduino/hardware/esp8266com/esp8266/cores/esp8266/tools/sdk/ld/eagle.flash.4m1m.ld
/* Flash Split for 4M chips */
/* sketch @0x40200000 (~1019KB) (1044464B) */
/* empty  @0x402FEFF0 (~2052KB) (2101264B) */
/* spiffs @0x40500000 (~1000KB) (1024000B) */
/* eeprom @0x405FB000 (4KB) */
/* rfcal  @0x405FC000 (4KB) */
/* wifi   @0x405FD000 (12KB) */

MEMORY
{
  dport0_0_seg :                        org = 0x3FF00000, len = 0x10
  dram0_0_seg :                         org = 0x3FFE8000, len = 0x14000
  irom0_0_seg :                         org = 0x40201010, len = 0xfeff0
}

PROVIDE ( _FS_start = 0x40500000 );
PROVIDE ( _FS_end = 0x405FA000 );
PROVIDE ( _FS_page = 0x100 );
PROVIDE ( _FS_block = 0x2000 );
PROVIDE ( _EEPROM_start = 0x405fb000 );
/* The following symbols are DEPRECATED and will be REMOVED in a future release */
PROVIDE ( _SPIFFS_start = 0x40500000 );
PROVIDE ( _SPIFFS_end = 0x405FA000 );
PROVIDE ( _SPIFFS_page = 0x100 );
PROVIDE ( _SPIFFS_block = 0x2000 );

INCLUDE "local.eagle.app.v6.common.ld"

earlephilhower commented 1 year ago

I've just run sanity tests onboard using the test script https://github.com/earlephilhower/arduino-pico/tree/master/libraries/LittleFS/examples/FSUpload (it's for the RPi Pico but runs fine on the ESP8266) which involves uploading, reading, and updating a LittleFS image onboard.

On my D1 Mini it runs fine and has no issue with accessing the uploaded filesystem image. The image is built using the mklittlefs from x86_64-linux-gnu.mklittlefs-30b7fc1.220621.tar.gz and the LittleFS library 2.5.1 that is in 3.1.x.

dimecho commented 1 year ago

For some reason v3.1.1 is very strict about block_count = flash_size/block_size math. Not sure how it worked before, but it look like it did not care about the ratio in v3.0.2

Going to

~/Library/Arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/LittleFS/src/

Edit two files lfs_util.c, lfs.c

#define LFS_NAME_MAX 32
//#define LFS_NO_DEBUG
//#define LFS_NO_WARN
//#define LFS_NO_ERROR

Then I can see the debug

lfs.c:4188:error: Invalid block count (126 != 253)

...OK so my flash is 4MB with 2MB OTA but I build image for 1MB (640KB actually) because ESP8266HTTPUpdateServer needs half space to store the upload.

Bad idea in v3.1.1

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 640000 flash-littlefs.bin

It would be waste of space to build mklittlefs for 2MB as most of it will be FFFFFFF and it will not fit using ESP8266HTTPUpdateServer upload.

...OK so it wants 253 blocks (8192 * 253) = 2072576

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 2072576 flash-littlefs.bin

...good but it is not half the flash, will not fit for ESP8266HTTPUpdateServer _upload_max: 0x000FEFF0 (1044464)

Lets try the max upload size

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 1044464 flash-littlefs.bin

... back to square one, nothing reads on boot. lfs.c:4188:error: Invalid block count (127 != 253)

So lets divide block_count by two in LittleFS.h

line71:   _lfs_cfg.block_count = _size / _blockSize / 2;

Upload works! I can read half the files then it crashes (yes I just made half my flash usable with proper block-size math for flash-littlefs.bin)

So I am thinking to get ESP8266HTTPUpdateServer working with 2MB flash and 1MB upload flash-littlefs.bin I need to build it for 2MB but gzip for upload build.extra_flags=-DATOMIC_FS_UPDATE

dimecho commented 1 year ago

Oh wait!!! This actually works! for full flash size 2072576

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 2072576 flash-littlefs.bin

I remember it used to be that ESP8266HTTPUpdateServer needed half the size, but I guess it is dynamic now.

Issue resolved :)

dimecho commented 1 year ago

But wait this creates a "chicken-egg" situation.

ESP8266HTTPUpdateServer example is broken with signature because it adds 260 bytes (256 byte signature + 4 byte length) to .bin file.

If we try 2072576 - 260 = 2072316

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 2072316 flash-littlefs.bin

This puts us back to lfs.c:4188:error: Invalid block count

and using working size 2072576 + 260 = 2072836

signing.py --mode sign --privatekey ./private.key --bin lash-littlefs.bin --out flash-littlefs.signed

Not Enough Space flash-littlefs.signed file over OTA size 2072576

dimecho commented 1 year ago

Not sure if this is acceptable solution for v3.1.1. In order to fix LittleFS binary signing support, we sacrifice a block.

Patch LittleFS.h

#if ARDUINO_SIGNING
_lfs_cfg.block_count = _size / _blockSize - 1;
#else
_lfs_cfg.block_count = _size / _blockSize;
#endif

2072576 - 8192 = 2064384

mklittlefs -c ./data/ -b 8192 -p 256 -d 5 -s 2064384 flash-littlefs.bin
signing.py --mode sign --privatekey private.key --bin flash-littlefs.bin --out flash-littlefs.signed

d-a-v commented 1 year ago

Did you try to gzip -9 flash-littlefs.bin before signing and uploading ?

dimecho commented 1 year ago

Did you try to gzip -9 flash-littlefs.bin before signing and uploading ?

No, there is not much benefit in gzip as it seems to work with 4MB 1MB OTA (wasting half for extracting?) and I can use 4MB 2MB OTA for full FS. But once it extracts it is same story Not Enough Space if you go over OTA size with 260 byte signing.

But if you subtract "-1" in LittleFS.h, works with signing.

esp8266 / Arduino