Broken data after the update from 1.5.3

[butaikis]

Hi. I use your esp_littlefs 1.5.3 library in my project (PLatformIO, Platform Espressif 32). I use a data storage system using JSON (ArduinoJson.h). I decided to update the esp_littles library to the latest version and found that after that the data was read broken or offset. What could be the problem?

[huming2207]

You need to provide more details, e.g. are you using SD card or internal SPI NOR flash partition? What's the ESP-IDF version?

[butaikis]

@huming2207, Hello At the moment

[env:esp-wrover-kit]
platform = espressif32
board = esp-wrover-kit
framework = esp-idf, arduino

Platform version, Espressif framework - 6.7.0 The project uses Arduino version 2.0.16 (based on IDF version 4.4.7)

At the moment, esp_littlefs v1.5.3 sees only its own changes (if you changed the text and saved it in SPIFFS). And if you update the library to the latest version 1.14.4, there will be no changes there.

It turns out that I cannot update esp_littlefs in my project because I will lose data.

UPD

One of the files stops reading from SPIFFS altogether when it has updated the library

(ArduinoJson.h) Failed read file /datas.json Error: Empty Input

[BrianPugh]

are you reading/writing to a binary image on your computer? or on-device? In v1.6.0 we upgraded LittleFS core from v2.5.1 to v2.7.0. This upgrades the on-disk minor version from lfs2.0 -> lfs2.1. If you have other tools that use an older version of LittleFS, they cannot read lfs2.1.

[butaikis]

@BrianPugh I started writing my project with esp_littlefs version 1.5.3 Data in SPIFFS is read and written only from ESP32.

[BrianPugh]

Can you try other versions of esp_littlefs to try and isolate which one broke your environment?

[butaikis]

@BrianPugh

1. espressif32 v6.6.0 and esp_littlefs 1.5.3 = OK
2. espressif32 v6.6.0 and esp_littlefs 1.5.4 = OK
3. espressif32 v6.6.0 and esp_littlefs 1.5.5 = OK
4. espressif32 v6.6.0 and esp_littlefs 1.6.0 = ERR
5. espressif32 v6.6.0 and esp_littlefs 1.7.0 = ERR
6. espressif32 v6.6.0 and esp_littlefs 1.8.0 = ERR
7. espressif32 v6.6.0 and esp_littlefs 1.10.0 = ERR

ERR it is One file is not readable at all (ArduinoJson returns an EmptyInput), and in another file, in some JSON fields, the data is not what it should be. (Here the data is displayed, which was edited using esp_littlefs version higher than 1.5.4, why this happens is not clear)

espressif32 v6.7.0 with this version, it is not possible to compile the project with older versions of esp_littlefs at all, an error

D:/Develop_VS_Code/.platformio/packages/framework-arduinoespressif32/libraries/LittleFS/src/LittleFS.cpp: In member function 'bool fs::LittleFSFS::begin(bool, const char*, uint8_t, const char*)':
D:/Develop_VS_Code/.platformio/packages/framework-arduinoespressif32/libraries/LittleFS/src/LittleFS.cpp:89:5: error: 'esp_vfs_littlefs_conf_t' has no non-static data member named 'partition'
     };
     ^
Compiling .pio\build\esp-wrover-kit\WiFiGeneric.cpp.o
*** [.pio\build\esp-wrover-kit\LittleFS.cpp.o] Error 1

It turns out that today I cannot use the espressif32 version higher than 6.6.0 with esp_littlefs, perhaps there is a solution to the problem?

[BrianPugh]

so this coincides with the LittleFS on-disk upgrade (v2.0 -> v2.1). However, this should NOT result in unreadable data. LittleFS should update the disk version adhoc. However, once using a newer version (i.e. esp_littlefs >= v1.6.0), the files will become unreadable to older versions of littlefs.

[butaikis]

@BrianPugh Are there any solutions to the problem so that I can use the Espressif 32 version higher than 6.6.0 ?

[BrianPugh]

I don't use arduino/platformio, so I don't have too much insight. If it's using the partition field of esp_vfs_littlefs_conf_t, that was included in esp_littlefs v1.8.0.

[butaikis]

@BrianPugh

I understand. Perhaps this log will help you figure out the reason? Maybe it's the size of the file, this file is larger than all the others in the storage, the size is 14436 bytes

V (7525) esp_littlefs: ---------------------<<< Sem Give [loopTask]
V (7531) esp_littlefs: Opening /scens.json
V (7534) esp_littlefs: O_RDONLY
V (7537) esp_littlefs: ------------------------ Sem Taking [loopTask]
V (7543) esp_littlefs: --------------------->>> Sem Taken [loopTask]
V (7550) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7555) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7560) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7565) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7571) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7576) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7581) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7586) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7591) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7596) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7602) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7607) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7612) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7617) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7623) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7628) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7633) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7638) esp_littlefs: ---------------------<<< Sem Give [loopTask]
V (7644) esp_littlefs: Done opening /scens.json
V (7648) esp_littlefs: ------------------------ Sem Taking [loopTask]
V (7654) esp_littlefs: --------------------->>> Sem Taken [loopTask]
V (7661) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7666) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7671) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7676) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7682) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7687) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7692) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7697) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7702) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7708) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7713) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7718) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7723) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7728) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7734) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7739) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7744) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7749) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7754) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7760) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7765) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7770) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7775) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7780) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7786) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7791) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7796) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7801) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7806) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7812) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7817) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7822) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7827) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7833) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7838) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7843) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7848) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7853) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7859) spi_flash: allocate temp buffer: 0x3ffb26e0 (512)
V (7864) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7869) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7874) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7879) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7885) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7890) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7895) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7900) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7905) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7911) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7916) spi_flash: allocate temp buffer: 0x3ffb26e0 (128)
V (7921) esp_littlefs: ---------------------<<< Sem Give [loopTask]
V (7927) esp_littlefs: ------------------------ Sem Taking [loopTask]
V (7933) esp_littlefs: --------------------->>> Sem Taken [loopTask]
V (7939) spi_flash: allocate temp buffer: 0x3ffb2770 (128)
V (7944) spi_flash: allocate temp buffer: 0x3ffb2770 (128)
V (7950) esp_littlefs: ---------------------<<< Sem Give [loopTask]
V (7955) esp_littlefs: Failed to read file "/scens.json". Error  (-84)
V (7962) esp_littlefs: ------------------------ Sem Taking [loopTask]
V (7968) esp_littlefs: --------------------->>> Sem Taken [loopTask]
V (7974) esp_littlefs: Clearing FD
V (7977) esp_littlefs: ---------------------<<< Sem Give [loopTask]

[BrianPugh]

Error -84 corresponds to LFS_ERR_CORRUPT, so it seems like the file you are reading is corrupt.

[butaikis]

The problem was solved like this.

1. Install the esp_littlefs version on which everything worked. (1.5.5)
2. We make the firmware so that all data is read in RAM.
3. Format the littlefs storage
4. Save all the data back to the storage

After that, I was able to upgrade to the latest version of the library and all the files worked as they should.