How to deserialize large arrays without running out of RAM?

[liamw9534]

I have a JSON document which has a sizeable object array and I would like to ideally deserialize the array iteratively if at all possible. I took a brief look at the filters but this doesn't appear to do what I want.

Programatically, I was thinking that perhaps you would need to set a filter option that basically informs the deserializer that you want to deserialize one or more array properties iteratively into a separate json document. And then you would have some API call to access the deserialization iterator objects and then expand them into a separate JsonDocument, one by one.

The goal is to only have one object of the large array resident in RAM at any given moment, where it is processed and then released from RAM before moving onto the next one.

[bblanchon]

Hi @liamw9534,

Did you read How to deserialize a very large document?

Best regards, Benoit

[liamw9534]

Ha, that's very cool. I hadn't noticed that. My input is a File object from FsFat which I don't believe is a Stream derivative although I suspect it would be possible to wrap it as a Stream object since it supports single byte access and seek operations.

On Sat, 16 Mar 2024 at 13:01, Benoît Blanchon @.***> wrote:

Hi @liamw9534 https://github.com/liamw9534,

Did you read How to deserialize a very large document? https://arduinojson.org/v7/how-to/deserialize-a-very-large-document/

Best regards, Benoit

— Reply to this email directly, view it on GitHub https://github.com/bblanchon/ArduinoJson/issues/2069#issuecomment-2001980487, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTRST2JXPVLIHHLCKFJ6S3YYQ7A5AVCNFSM6AAAAABEYDCNNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBRHE4DANBYG4 . You are receiving this because you were mentioned.Message ID: @.***>

[bblanchon]

I'm unsure which library you're referring to, but most Arduino file abstractions implement the Stream interface. Examples include SD, ESP32 FS, ESP8266 FS, Arduino Audio Tools, RasPi Bridge.

If the Stream interface is not supported, you can write a custom reader. You can find an example for the C files API (stdio.h) in Mastering ArduinoJson in the Advanced Techniques chapter.

[liamw9534]

That's most helpful. I managed to figure it all out! I'm not using SD but instead SdFat which doesn't have the Stream abstraction on its File objects but is pretty easy to add.

On Sun, 17 Mar 2024 at 09:56, Benoît Blanchon @.***> wrote:

I'm unsure which library you're referring to, but most Arduino file abstractions implement the Stream interface. Examples include SD https://github.com/arduino-libraries/SD/blob/1.2.4/src/SD.h#L28, ESP32 FS https://github.com/espressif/arduino-esp32/blob/3.0.0-alpha3/libraries/FS/src/FS.h#L47, ESP8266 FS https://github.com/esp8266/Arduino/blob/3.1.2/cores/esp8266/FS.h#L52, Arduino Audio Tools https://github.com/pschatzmann/arduino-audio-tools/blob/v0.9.7/src/AudioLibs/Desktop/File.h#L25, RasPi Bridge https://github.com/me-no-dev/RasPiArduino/blob/0.0.1/libraries/Bridge/src/FileIO.h#L32 .

If the Stream interface is not supported, you can write a custom reader https://arduinojson.org/news/2019/11/01/version-6-13-0/#custom-reader. You can find an example for the C files API (stdio.h) in Mastering ArduinoJson https://arduinojson.org/book/ in the Advanced Techniques chapter.

— Reply to this email directly, view it on GitHub https://github.com/bblanchon/ArduinoJson/issues/2069#issuecomment-2002387649, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTRST2ZMOJYAW6ENX3M5JTYYVSE7AVCNFSM6AAAAABEYDCNNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGM4DONRUHE . You are receiving this because you were mentioned.Message ID: @.***>

[bblanchon]

Assuming you're talking about the SdFat library, it does support the Stream abstraction, even if it's not very explicit. Let me explain:

1. File is a typedef for File32, ExFile, or FsFile
2. Let's take File32 for example; it derives from StreamFile
3. StreamFile derives from stream_t
4. stream_t is a typedef for Stream

Conclusion: File32 implements the Stream interface, and so do ExFile and FsFile. Therefore, you can pass the file instance directly to deserializeJson().

[liamw9534]

Thanks, I'll check it out.

On Mon, 18 Mar 2024 at 08:20, Benoît Blanchon @.***> wrote:

Assuming you're talking about the SdFat https://github.com/greiman/SdFat library, it does support the Stream abstraction, even if it's not very explicit. Let me explain:

1. File is a typedef for File32, ExFile, or FsFile https://github.com/greiman/SdFat/blob/2.2.2/src/SdFat.h#L453-L458
2. Let's take File32 for example; it derives from StreamFile https://github.com/greiman/SdFat/blob/2.2.2/src/FatLib/FatFile.h#L1023
3. StreamFile derives from stream_t https://github.com/greiman/SdFat/blob/2.2.2/src/common/ArduinoFiles.h#L61
4. stream_t is a typedef for Stream https://github.com/greiman/SdFat/blob/2.2.2/src/common/SysCall.h#L46

Conclusion: File32 implements the Stream interface, and so do ExFile and FsFile. Therefore, you can pass the file instance directly to deserializeJson().

— Reply to this email directly, view it on GitHub https://github.com/bblanchon/ArduinoJson/issues/2069#issuecomment-2003173621, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTRSTZKGQBDN7IJTRWJCVTYY2PUPAVCNFSM6AAAAABEYDCNNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBTGE3TGNRSGE . You are receiving this because you were mentioned.Message ID: @.***>

[liamw9534]

Confirmed that it does indeed work correctly as a Stream. Thanks again!

On Mon, 18 Mar 2024 at 09:24, Liam Wickins @.***> wrote:

Thanks, I'll check it out.

On Mon, 18 Mar 2024 at 08:20, Benoît Blanchon @.***> wrote:

Assuming you're talking about the SdFat https://github.com/greiman/SdFat library, it does support the Stream abstraction, even if it's not very explicit. Let me explain:

1. File is a typedef for File32, ExFile, or FsFile https://github.com/greiman/SdFat/blob/2.2.2/src/SdFat.h#L453-L458
2. Let's take File32 for example; it derives from StreamFile https://github.com/greiman/SdFat/blob/2.2.2/src/FatLib/FatFile.h#L1023
3. StreamFile derives from stream_t https://github.com/greiman/SdFat/blob/2.2.2/src/common/ArduinoFiles.h#L61
4. stream_t is a typedef for Stream https://github.com/greiman/SdFat/blob/2.2.2/src/common/SysCall.h#L46

Conclusion: File32 implements the Stream interface, and so do ExFile and FsFile. Therefore, you can pass the file instance directly to deserializeJson().

— Reply to this email directly, view it on GitHub https://github.com/bblanchon/ArduinoJson/issues/2069#issuecomment-2003173621, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTRSTZKGQBDN7IJTRWJCVTYY2PUPAVCNFSM6AAAAABEYDCNNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBTGE3TGNRSGE . You are receiving this because you were mentioned.Message ID: @.***>

[bblanchon]

You're welcome, @liamw9534. Thank you for using ArduinoJson! Don't forget to cast a star to support the project :wink: