search

bblanchon / ArduinoJson

📟 JSON library for Arduino and embedded C++. Simple and efficient.
https://arduinojson.org
MIT License
6.7k stars 1.12k forks source link

deserialize file from LittleFS crash #2095

Closed hitecSmartHome closed 4 months ago

hitecSmartHome commented 4 months ago

Hi! The library is working flawlessly so far and thank you very much for the amazing work. I had a crash where I wanted to deserialize a file and the logs points me to either LittleFS or ArduinoJSON. I wonder if you can see any sense in the logs.

==================== THREAD 1 (TCB: 0x3ffcca78, name: 'loopTask') =====================
#0  0x400839c0 in panic_abort (details=0x3ffcc3e0 "assert failed: 0x401c1bdf") at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components\esp_system\panic.c:408
#1  0x4008a134 in esp_system_abort (details=0x3ffcc3e0 "assert failed: 0x401c1bdf") at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components\esp_system\esp_system.c:137
#2  0x4008f1a8 in __assert_func (file=<optimized out>, line=0, func=0x0, expr=0x0) at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components
ewlib\assert.c:47
#3  0x401c1be2 in lfs_file_read (lfs=0x3ffbe3d4, file=0x0, buffer=0x3f80e5b4, size=128) at managed_components\esp_littlefs\src\littlefs\lfs.c:6126
#4  0x401bdb11 in vfs_littlefs_read (ctx=0x3ffbe2f0, fd=<optimized out>, dst=0x3f80e5b4, size=128) at managed_components\esp_littlefs\src\esp_littlefs.c:1545
#5  0x401f5d15 in esp_vfs_read (r=<optimized out>, fd=0, dst=0x3f80e5b4, size=128) at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components\vfs\vfs.c:457
#6  0x4000bdbe in ?? ()
#7  0x40001128 in ?? ()
#8  0x400594ec in ?? ()
#9  0x40268fbd in _fread_r (ptr=0x3ffccae4, buf=<optimized out>, size=1, count=1, fp=0x3f80e7b8) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fread.c:228
#10 0x40269050 in fread (buf=0x3ffcc5a0, size=1, count=1, fp=0x3f80e7b8) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fread.c:266
#11 0x401c5e0e in VFSFileImpl::read (this=0x3f81862c, buf=0x3ffcc5a0 "`f\034\200\340\305\374?<\306\374?D\306\374?", size=1) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/vfs_api.cpp:408
#12 0x401a751e in fs::File::read (this=<optimized out>, buf=0x3ffcc5a0 "`f\034\200\340\305\374?<\306\374?D\306\374?", size=1) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/FS.cpp:82
#13 0x401a7530 in fs::File::readBytes (this=0x3ffcc6b0, buffer=0x3ffcc5a0 "`f\034\200\340\305\374?<\306\374?D\306\374?", length=1) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/FS.h:63
#14 0x40271e59 in ArduinoJson::V700PB2::detail::Reader<fs::File, void>::read (this=0x3ffcc630) at lib/ArduinoJson-7.x/src/ArduinoJson/Deserialization/Readers/ArduinoStreamReader.hpp:17
#15 ArduinoJson::V700PB2::detail::Latch<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::load (this=0x3ffcc630) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/Latch.hpp:38
#16 ArduinoJson::V700PB2::detail::Latch<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::current (this=0x3ffcc630) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/Latch.hpp:30
#17 ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::current (this=0x3ffcc620) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:47
#18 0x40137871 in ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::skipSpacesAndComments (this=0x3ffcc620) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:581
#19 0x40137bb9 in ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::skipSpacesAndComments (this=0x3ffcc620) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:579
#20 ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::parseVariant<ArduinoJson::V700PB2::detail::AllowAllFilter> (this=0x3ffcc620, variant=..., filter=..., nestingLimit=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:67
#21 0x40137e61 in ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::parse<ArduinoJson::V700PB2::detail::AllowAllFilter> (nestingLimit=..., variant=..., this=0x3ffcc620, filter=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:31
#22 ArduinoJson::V700PB2::detail::doDeserialize<ArduinoJson::V700PB2::detail::JsonDeserializer, ArduinoJson::V700PB2::JsonDocument&, ArduinoJson::V700PB2::detail::Reader<fs::File, void>, ArduinoJson::V700PB2::detail::DeserializationOptions<ArduinoJson::V700PB2::detail::AllowAllFilter> > (dst=..., reader=..., options=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Deserialization/deserialize.hpp:57
#23 0x40138011 in ArduinoJson::V700PB2::detail::deserialize<ArduinoJson::V700PB2::detail::JsonDeserializer, ArduinoJson::V700PB2::JsonDocument&, fs::File&, , void>(ArduinoJson::V700PB2::JsonDocument&, fs::File&) (input=..., dst=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Deserialization/deserialize.hpp:69
#24 ArduinoJson::V700PB2::deserializeJson<ArduinoJson::V700PB2::JsonDocument&, fs::File&> (dst=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:679
#25 FileSystem::read (this=0x3ffb43c0 <db>, path=0x3f40228c "/graphs.json", doc=...) at src/FileSystem/FileSystemReads.cpp:138
#26 0x400e68b4 in Graphs::get (this=<optimized out>, type=0x3f408868 "heating", id=0x3ffe350c <error: Cannot access memory at address 0x3ffe350c>, graphData=...) at src/Components/Graph/Graph.cpp:8
#27 0x4011f4ea in Thermostat::handleGraph (this=0x3f823968, detail=...) at c:\users\pc\.platformio\packages     oolchain-xtensa-esp32@8.4.0+2021r2-patch5\xtensa-esp32-elf\include\c++\8.4.0\bits/basic_string.h:2290
#28 0x4011f6a8 in Thermostat::checkGraph (this=0x3f823968) at src/Components/Thermostat/GraphHandler.cpp:13
#29 0x4012516d in <lambda(int, int)>::operator() (__closure=0x3ffcc940, prev=<optimized out>, now=<optimized out>) at src/Components/Thermostat/Thermostat.cpp:26
#30 std::_Function_handler<void(int, int), Thermostat::Thermostat(ArduinoJson::V700PB2::JsonObject&)::<lambda(int, int)> >::_M_invoke(const std::_Any_data &, int &&, int &&) (__functor=..., __args#0=@0x3ffcc910: 59, __args#1=@0x3ffcc914: 0) at c:\users\pc\.platformio\packages oolchain-xtensa-esp32@8.4.0+2021r2-patch5\xtensa-esp32-elf\include\c++\8.4.0\bits/std_function.h:297
#31 0x401975ec in std::function<void (int, int)>::operator()(int, int) const (this=0x3ffcc940, __args#0=<optimized out>, __args#1=<optimized out>) at c:\users\pc\.platformio\packages  oolchain-xtensa-esp32@8.4.0+2021r2-patch5\xtensa-esp32-elf\include\c++\8.4.0\bits/std_function.h:687
#32 0x401976b4 in Time::emitMinChange (now=0, prev=59, this=<optimized out>) at src/Time/TimeEvents.cpp:45
#33 Time::emitMinChange (this=<optimized out>, prev=59, now=0) at src/Time/TimeEvents.cpp:39
#34 0x40197836 in Time::monitorMinChange (this=0x3ffb3d7c <hshTime>) at src/Time/TimeEvents.cpp:154
#35 0x4019785f in Time::handleEvents (this=0x3ffb3d7c <hshTime>) at src/Time/TimeEvents.cpp:205
#36 0x40196599 in Time::loop (this=0x3ffb3d7c <hshTime>) at src/Time/Time.cpp:211
#37 0x401a14e0 in loop () at src/main.cpp:91
#38 0x401c49d8 in loopTask (pvParameters=<optimized out>) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:50

The function which initiated the crash

boolean Graphs::get(const char* type, const char* id, JsonArray& graphData) {
    JsonDocument graphs;

    int size = db.read(GRAPHS_PATH,graphs);
    if( size <= 0 ){ return false; }

    JsonArray graphsArray = graphs[type].as<JsonArray>();
    for(JsonObject graph : graphsArray ){
        if( graph["id"] == id ){
            JsonArray savedData = graph["data"].as<JsonArray>();
            return graphData.set(savedData);
        }
    }
    return false;
}

The LittleFS read function

int FileSystem::read(const char* path, JsonDocument& doc){
    File file = userFS.open(path);
    if(!file){
        #if DB_DEBUG
            logger.write("[FileSystem] - (getJson) File not found: %s\n",path);
        #endif
        return -1;
    }
    DeserializationError error = deserializeJson(doc, file);
    file.close();
    if( error == DeserializationError::Ok ){
        #if DB_DEBUG
            logger.write("[FileSystem] - (getJson) Message read from file: %s\n",path);
        #endif
        return doc.size();
    }
    return -1;
}

Sorry for the ticket, it is problably not ArduinoJSON related crash but I don't know.

I'm using ArduinoJSON V7 with ESP32 Arduino as an IDF component.

hitecSmartHome commented 4 months ago

The get method of the graphHandler is called like this

JsonDocument doc;
JsonArray graphArr = doc.to<JsonArray>();
if( !graphHandler.get("heating",detail.id,graphArr) ){
    #if THERMOSTAT_DEBUG
        logger.write("[Thermostat] - Failed to get graph %s\n",detail.name.c_str());
    #endif
    return;
}
bblanchon commented 4 months ago

Hi @hitecSmartHome,

The stack trace indicates that an assertion failed in fs::File::read(), which strongly suggests that the problem comes from the FS library. What version of the ESP32 are you using? Can you try another version?

Best regards, Benoit

hitecSmartHome commented 4 months ago

I have an ESP32-Wrover-E with 8mb psram and 16mb flash. I use LittleFS from the start, a really long time ago. Arduino is a component of IDF. IDF is 4.4.6 Arduino is 2.0.14 ArduinoJson is the latest V7

I have no other ESP which has PS_RAM. I can write a basic script to a no-psram ESP hovewer but this crash is no easily reproducable. This function runs every minute and I have the same ESPs runnin for days without this crash.

hitecSmartHome commented 4 months ago

Strange it seems that it only happens when I use ArduinoJson with LittleFS. It also happened when flash read

int FileSystem::read(const char* path, JsonDocument& doc){
    File file = userFS.open(path);
    if(!file){
        #if DB_DEBUG
            logger.write("[FileSystem] - (getJson) File not found: %s\n",path);
        #endif
        return -1;
    }
    DeserializationError error = deserializeJson(doc, file);
    file.close();
    if( error == DeserializationError::Ok ){
        #if DB_DEBUG
            logger.write("[FileSystem] - (getJson) Message read from file: %s\n",path);
        #endif
        return doc.size();
    }
    return -1;
}

Crash:

==================== CURRENT THREAD STACK =====================
#0  0x400839c0 in panic_abort (details=0x3ffcc3c0 "assert failed: 0x401c720b") at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components\esp_system\panic.c:408
#1  0x4008a134 in esp_system_abort (details=0x3ffcc3c0 "assert failed: 0x401c720b") at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components\esp_system\esp_system.c:137
#2  0x4008f1a8 in __assert_func (file=<optimized out>, line=0, func=0x0, expr=0x0) at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components
ewlib\assert.c:47
#3  0x401c720e in lfs_file_read (lfs=0x3ffbe870, file=0x0, buffer=0x3f82688c, size=128) at managed_components\esp_littlefs\src\littlefs\lfs.c:6126
#4  0x401c313d in vfs_littlefs_read (ctx=0x3ffbe78c, fd=<optimized out>, dst=0x3f82688c, size=128) at managed_components\esp_littlefs\src\esp_littlefs.c:1545
#5  0x401f6189 in esp_vfs_read (r=<optimized out>, fd=0, dst=0x3f82688c, size=128) at C:\Users\Pc\.platformio\packages\framework-espidf@3.40406.240122\components\vfs\vfs.c:457
#6  0x4000bdbe in ?? ()
#7  0x40001128 in ?? ()
#8  0x400594ec in ?? ()
#9  0x40269409 in _fread_r (ptr=0x3ffccaf4, buf=<optimized out>, size=1, count=1, fp=0x3ffd1a8c) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fread.c:228
#10 0x4026949c in fread (buf=0x3ffcc580, size=1, count=1, fp=0x3ffd1a8c) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fread.c:266
#11 0x401c083a in VFSFileImpl::read (this=0x3f8260b4, buf=0x3ffcc580 "\214\020\034\200\300\305\374?\034\306\374?$\306\374?", size=1) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/vfs_api.cpp:408
#12 0x401a797e in fs::File::read (this=<optimized out>, buf=0x3ffcc580 "\214\020\034\200\300\305\374?\034\306\374?$\306\374?", size=1) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/FS.cpp:82
#13 0x401a7990 in fs::File::readBytes (this=0x3ffcc690, buffer=0x3ffcc580 "\214\020\034\200\300\305\374?\034\306\374?$\306\374?", length=1) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/FS.h:63
#14 0x402722a5 in ArduinoJson::V700PB2::detail::Reader<fs::File, void>::read (this=0x3ffcc610) at lib/ArduinoJson-7.x/src/ArduinoJson/Deserialization/Readers/ArduinoStreamReader.hpp:17
#15 ArduinoJson::V700PB2::detail::Latch<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::load (this=0x3ffcc610) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/Latch.hpp:38
#16 ArduinoJson::V700PB2::detail::Latch<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::current (this=0x3ffcc610) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/Latch.hpp:30
#17 ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::current (this=0x3ffcc600) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:47
#18 0x40137959 in ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::skipSpacesAndComments (this=0x3ffcc600) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:581
#19 0x40137ca1 in ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::skipSpacesAndComments (this=0x3ffcc600) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:579
#20 ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::parseVariant<ArduinoJson::V700PB2::detail::AllowAllFilter> (this=0x3ffcc600, variant=..., filter=..., nestingLimit=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:67
#21 0x40137f49 in ArduinoJson::V700PB2::detail::JsonDeserializer<ArduinoJson::V700PB2::detail::Reader<fs::File, void> >::parse<ArduinoJson::V700PB2::detail::AllowAllFilter> (nestingLimit=..., variant=..., this=0x3ffcc600, filter=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:31
#22 ArduinoJson::V700PB2::detail::doDeserialize<ArduinoJson::V700PB2::detail::JsonDeserializer, ArduinoJson::V700PB2::JsonDocument&, ArduinoJson::V700PB2::detail::Reader<fs::File, void>, ArduinoJson::V700PB2::detail::DeserializationOptions<ArduinoJson::V700PB2::detail::AllowAllFilter> > (dst=..., reader=..., options=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Deserialization/deserialize.hpp:57
#23 0x401380f9 in ArduinoJson::V700PB2::detail::deserialize<ArduinoJson::V700PB2::detail::JsonDeserializer, ArduinoJson::V700PB2::JsonDocument&, fs::File&, , void>(ArduinoJson::V700PB2::JsonDocument&, fs::File&) (input=..., dst=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Deserialization/deserialize.hpp:69
#24 ArduinoJson::V700PB2::deserializeJson<ArduinoJson::V700PB2::JsonDocument&, fs::File&> (dst=...) at lib/ArduinoJson-7.x/src/ArduinoJson/Json/JsonDeserializer.hpp:679
#25 FileSystem::read (this=0x3ffb43c0 <db>, path=0x3f40228c "/graphs.json", doc=...) at src/FileSystem/FileSystemReads.cpp:138
#26 0x400e697e in Graphs::get (this=<optimized out>, type=0x3f406918 "shader", id=..., graphData=...) at src/Components/Graph/Graph.cpp:8
#27 0x4010c4d2 in Shader::handleGraph (this=0x3f8515ec, detail=...) at src/Components/Shader/GraphHandler.cpp:22
#28 0x4010c670 in Shader::checkGraph (this=0x3f8515ec) at src/Components/Shader/GraphHandler.cpp:13
#29 0x40113a5d in <lambda(int, int)>::operator() (__closure=0x3ffcc950, prev=<optimized out>, now=<optimized out>) at src/Components/Shader/Shader.cpp:14
#30 std::_Function_handler<void(int, int), Shader::Shader(ArduinoJson::V700PB2::JsonObject&)::<lambda(int, int)> >::_M_invoke(const std::_Any_data &, int &&, int &&) (__functor=..., __args#0=@0x3ffcc920: 59, __args#1=@0x3ffcc924: 0) at c:\users\pc\.platformio\packages       oolchain-xtensa-esp32@8.4.0+2021r2-patch5\xtensa-esp32-elf\include\c++\8.4.0\bits/std_function.h:297
#31 0x401977c0 in std::function<void (int, int)>::operator()(int, int) const (this=0x3ffcc950, __args#0=<optimized out>, __args#1=<optimized out>) at c:\users\pc\.platformio\packages     oolchain-xtensa-esp32@8.4.0+2021r2-patch5\xtensa-esp32-elf\include\c++\8.4.0\bits/std_function.h:687
#32 0x40197888 in Time::emitMinChange (now=0, prev=59, this=<optimized out>) at src/Time/TimeEvents.cpp:45
#33 Time::emitMinChange (this=<optimized out>, prev=59, now=0) at src/Time/TimeEvents.cpp:39
#34 0x40197a0a in Time::monitorMinChange (this=0x3ffb3d7c <hshTime>) at src/Time/TimeEvents.cpp:154
#35 0x40197a33 in Time::handleEvents (this=0x3ffb3d7c <hshTime>) at src/Time/TimeEvents.cpp:205
#36 0x4019676d in Time::loop (this=0x3ffb3d7c <hshTime>) at src/Time/Time.cpp:211
#37 0x401a1754 in loop () at src/main.cpp:91
#38 0x401bf404 in loopTask (pvParameters=<optimized out>) at C:/Users/Pc/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:50
bblanchon commented 4 months ago

According to the two stack traces, here is the assertion that fails: https://github.com/littlefs-project/littlefs/blob/4dd30c1b8f1b416633fe63a338ede8934b6449a9/lfs.c#L6126

I don't know why it fails, but I suggest that you ask the LittleFS development team. Before doing that, however, you should upgrade the LittleFS library to the latest version. I think you are currently running version 2.9.1, shipped with esp_littlefs, but a 2.9.3 is available. Also, make sure your filesystem is not simply corrupted.

Good luck.

hitecSmartHome commented 4 months ago

Thank you very much. Will try on there.