search

maxgerhardt / pio-esp32-esp8266-filesystem-downloader

Script for expanding PlatformIO's ability to download LittleFS and SPIFFS filesystems from ESP32 and ESP8266 chips and their contents
35 stars 9 forks source link

Download of LittleFS: Assertion in littlefs/lfs.c failing #6

Open Miq1 opened 2 years ago

Miq1 commented 2 years ago

What I did:

That resulted in

 *  Executing task: C:\Users\Micha\.platformio\penv\Scripts\platformio.exe run --target downloadfs --environment nodemcuv2 

Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (4.0.1) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 @ 3.30002.0 (3.0.2)
 - tool-esptool @ 1.413.0 (4.13)
 - tool-esptoolpy @ 1.30000.201119 (3.0.0)
 - toolchain-xtensa @ 2.100300.210717 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 52 compatible libraries
Scanning dependencies...
Dependency Graph
|-- DHT sensor library for ESPx @ 1.18.0
|-- eModbus @ 1.6.0+sha.5c2d656
|   |-- ESPAsyncTCP @ 1.2.2
|-- ESP8266WiFi @ 1.0
|-- ESPAsyncTCP @ 1.2.2
|-- ESP8266mDNS @ 1.2
|   |-- ESP8266WiFi @ 1.0
|-- ArduinoOTA @ 1.0
|   |-- ESP8266WiFi @ 1.0
|   |-- ESP8266mDNS @ 1.2
|   |   |-- ESP8266WiFi @ 1.0
|-- LittleFS @ 0.1.0
|-- ESP8266WebServer @ 1.0
|   |-- ESP8266WiFi @ 1.0
Building in release mode
Auto-detected: COM3
command_download_fs(["downloadfs"], [])
Entrypoint
Retrieving filesystem info for ESP8266.
FS_START: 0x300000
FS_END: 0x3fa000
FS_PAGE: 0x100
FS_BLOCK: 0x2000
Recognized LittleFS filesystem.
Parsed FS info: FS type FSType.LITTLEFS Start 0x300000 Len 1024000 Page size 256 Block size 8192 Tool: C:\Users\Micha\.platformio\packages\tool-mklittlefs\mklittlefs
Executing flash download command.
"C:\Users\Micha\.platformio\penv\Scripts\python.exe" "C:\Users\Micha\.platformio\packages\tool-esptoolpy@1.30000.201119\esptool.py" --chip esp8266 --port "COM3" --baud 115200 --before default_reset --after hard_reset read_flash 0x300000 0xfa000 "C:\Users\Micha\Documents\PlatformIO\Projects\Dewair\downloaded_fs_0x300000_0xfa000.bin"
esptool.py v3.0
Serial port COM3
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 48:3f:da:7e:48:34
Uploading stub...
Running stub...
Stub running...
1024000 (100 %)
1024000 (100 %)
Read 1024000 bytes at 0x300000 in 92.5 seconds (88.5 kbit/s)...
Hard resetting via RTS pin...
Downloaded filesystem binary.
Download was okay: True. File at: C:\Users\Micha\Documents\PlatformIO\Projects\Dewair\downloaded_fs_0x300000_0xfa000.bin
Executing extraction command: "C:\Users\Micha\.platformio\packages\tool-mklittlefs\mklittlefs" -b 8192 -p 256 --unpack "unpacked_fs" "C:\Users\Micha\Documents\PlatformIO\Projects\Dewair\downloaded_fs_0x300000_0xfa000.bin"
Assertion failed!

Program: C:\Users\Micha\.platformio\packages\tool-mklittlefs\mklittlefs.exe
File: littlefs/lfs.c, Line 84

Expression: block < lfs->cfg->block_count
Unpacked filesystem.
Extracted 0 file(s) from filesystem.

My platformio.ini looks like:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
board_build.filesystem = littlefs
monitor_speed=115200
monitor_filters = esp8266_exception_decoder, default
lib_deps =
    beegee-tokyo/DHT sensor library for ESPx @ ^1.18
    eModbus = https://github.com/eModbus/eModbus
    ESP8266WiFi
    ESPAsyncTCP
    ESP8266mDNS
    ArduinoOTA
    LittleFS
    ESP8266WebServer
build_flags =
    -DLOG_LEVEL=6
extra_scripts = download_fs.py

The upload looked good to me:

 *  Executing task: C:\Users\Micha\.platformio\penv\Scripts\platformio.exe run --target uploadfs --environment nodemcuv2 

Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (4.0.1) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 @ 3.30002.0 (3.0.2)
 - tool-esptool @ 1.413.0 (4.13)
 - tool-esptoolpy @ 1.30000.201119 (3.0.0)
 - tool-mklittlefs @ 1.203.210628 (2.3)
 - tool-mkspiffs @ 1.200.0 (2.0)
 - toolchain-xtensa @ 2.100300.210717 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 52 compatible libraries
Scanning dependencies...
Dependency Graph
|-- DHT sensor library for ESPx @ 1.18.0
|-- eModbus @ 1.6.0+sha.5c2d656
|   |-- ESPAsyncTCP @ 1.2.2
|-- ESP8266WiFi @ 1.0
|-- ESPAsyncTCP @ 1.2.2
|-- ESP8266mDNS @ 1.2
|   |-- ESP8266WiFi @ 1.0
|-- ArduinoOTA @ 1.0
|   |-- ESP8266WiFi @ 1.0
|   |-- ESP8266mDNS @ 1.2
|   |   |-- ESP8266WiFi @ 1.0
|-- LittleFS @ 0.1.0
|-- ESP8266WebServer @ 1.0
|   |-- ESP8266WiFi @ 1.0
Building in release mode
Auto-detected: COM3
Building file system image from 'data' directory to .pio\build\nodemcuv2\littlefs.bin
/config.html
Looking for upload port...
Using manually specified: COM3
Uploading .pio\build\nodemcuv2\littlefs.bin
esptool.py v3.0
Serial port COM3
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 48:3f:da:7e:48:34
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Compressed 1024000 bytes to 4952...
Writing at 0x00300000... (100 %)
Wrote 1024000 bytes (4952 compressed) at 0x00300000 in 0.5 seconds (effective 18166.2 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
maxgerhardt commented 2 years ago

Block numbers and pagesize arguments look good, as well as the length. Address looks fine too, it reads from the same address that stuff was written to. Can you upload the C:\Users\Micha\Documents\PlatformIO\Projects\Dewair\downloaded_fs_0x300000_0xfa000.bin file for inspection?

Miq1 commented 2 years ago

downloaded_fs_0x300000_0xfa000.zip Here we go.

maxgerhardt commented 2 years ago

Well that's weird, given that file, I can immediately list its contents..

"C:\Users\Max\.platformio\packages\tool-mklittlefs\mklittlefs.exe" -b 8192 -p 256  -l downloaded_fs_0x300000_0xfa000.bin
19270   /config.html    Thu Oct 13 08:44:23 2022
965     /set.js Thu Jan 01 00:00:00 1970
256     /settings.bin   Thu Jan 01 00:00:00 1970
Creation time:  Fri Oct 14 15:39:26 2022

But unpacking indeed fails with the same error message as yours. I'll try and understand why..

Miq1 commented 2 years ago

If that is a hint: I looked into the downloaded .bin file with a hex editor and found a file that supposedly should have been altered in the meantime being still at the state of the first write to it, despite the fact I did a fresh download after deleting the previous.

maxgerhardt commented 2 years ago

That may be natural for LittleFS, not doing an erase unless absolutely necessary to overwrite data. I'll keep it in mind though.

maxgerhardt commented 2 years ago

Wow, I just downloaded the latest source code for mklittlefs from https://github.com/earlephilhower/mklittlefs, compiled in debug mode, ready to step-by-step debug the point where it fails, and what do you know, that latest version can unpack the binary just fine?! 

Directory ./unpack_dir/ does not exists. Try to create it.
config.html      > ./unpack_dir/config.html     size: 19270 Bytes
set.js   > ./unpack_dir/set.js  size: 965 Bytes
settings.bin     > ./unpack_dir/settings.bin    size: 256 Bytes

unpack_dir.zip

I need to check what the difference between this and the PIO packaged version is..

My compiled version now is mklittlefs.zip

Miq1 commented 2 years ago

I seem to have stirred up something?

If that matters, the settings.bin file is written binary, so it may contain any byte value - could that cause problems?

maxgerhardt commented 2 years ago

No that shouldn't matter at all. If you save binary files, then those must be preseverd exactly as is. Same for the html and js files. As referenced in the issue, although I can "extract" the files now, they don't match what I see in the littlefs .bin image. 

>type unpack_dir\*

unpack_dir\config.html

��3������3���3������h�3�Yd����3���3���3�e�����3�
!��3�60305�c�"r��3������
�3�60305ML����3�x�3�$������{;��������3�p�3�{;��3��3�P1;����3��3�PA����������3�3��2��0�3�0���3�@\Registry\Machine\System\CurrentControlSet\Control\Nls\Sorting\Versions��T�0�3�Q�3������Q�3�0�3�l�3�h�3����T� �Wh���2���3�P�3���3������p�3��3������0�3�H�3���3��������3�B����3�@F�T�000603xx�2�;����,�3�Q������3���T��y���0F�T�1d�"r��3���2���3���T��Wh���T�Ad�"r��3���T���3��3�������3���3�,�3�A�����3�������3���T� �h�3���3�������3�*���Z�"r��3���2�@�3�(
unpack_dir\set.js

��T���$�T��$�T�3P�T��f�T�
��$�T��    � �T��P�T���)�T�P�T�H�T��0�����T�#��B�T��B�T������T�'�T�k������T��b���������T�@Y����T���T������0�
}C���3��`p�C�%C�����T����3�0��������@�  �@яC�p<��|C��b������Zt�C�
unpack_dir\settings.bin

@@яC�p<�3�
�|C��b������Zt�C�
�3�@����
}C���3�0�C�

The first two cannot be correct when I clearly see human-readable HTML code in the .bin file.

Miq1 commented 2 years ago

At least I am able to re-read the binary file obviously. I will try to open and read the set.js file as well to make sure it is nothing caused by my application.

Jason2866 commented 1 year ago

Use newer mklittlefs (not in pio Registry). The issue was that the needed setting -s was not given in the script. Newer mklittlefs does not need -s option anymore for extracting files.

Jason2866 commented 1 year ago

Uploaded a updated mklittlefs for esp32x MCUs. Extracting files does now work for me. Before never got this going. https://registry.platformio.org/tools/tasmota/tool-mklittlefs

maxgerhardt commented 1 year ago

Thanks for the hints, I will retest this. I think we also talked about this in https://github.com/earlephilhower/mklittlefs/pull/31. If it's just -s that's missing, that's a small change to make it work properly.

Jason2866 commented 1 year ago

@maxgerhardt in my tests -s was not enough. Esp32 littlefs is build in IDF with LFS_NAME_MAX 64and the earlephilower mklittlefs is built for esp8266 with LFS_NAME_MAX 32. Thats why i built a extra mklittlefs for ESP32x with LFS_NAME_MAX 64 -> https://github.com/Jason2866/mklittlefs_esp32

Henning-Lenz commented 1 year ago

Due to the ancient version of "tool-mklittlefs"/mklittlefs in platformio (dating back to Feb 11, 2021, auto FS size retrieval not included yet), I ran into the same issue (_littlefs/lfs.c:84: lfs_bd_read: Assertion `block < lfs->cfg->blockcount' failed).

For me (currently working on esp8266) it was enough to pimp download_fs.py#L61 with an additional "-s", str(self.length), argument to hand over the size known to the download script to mklittlefs for unpacking images different from the default length correctly.

Jason2866 commented 10 months ago

For esp8266 it is enough to add "-s" as mentioned from @Henning-Lenz I have refactored the script for Tasmota (only using LittleFS) and with the changed mklittlefs for esp32 it is working now for esp8266 and esp32 (core2 and core3 alpha). https://github.com/arendst/Tasmota/blob/development/pio-tools/custom_target.py