How to manage big json using HTTPClient ?

[Smanar]

Describe the issue
Hello, on big json, I have the common error EmptyInput during deserialization.

Environment
ESP32, ATM the model 2432S028R

Reproduction
Here is a small snippet that demonstrate the problem. On same server but with smaller Json, all is fine, so I think the code is fine. I have just issue after a size.

        client.useHTTP10(true);
        client.setTimeout(500);
        client.begin(url);

        DynamicJsonDocument doc(buffer);
        DeserializationError err;
        err = deserializeJson(doc, client.getString());

        if (err)
        {
            Serial.printf("JSON PARSE: %s\n", client.getString());
            Serial.printf("Can't deserializeJson JSON : %s\n",err.c_str());
            Serial.printf("content Length : %d\n", client.getSize());
            return false;
        }

Program output

JSON PARSE: 
Can't deserializeJson JSON : EmptyInput
content Length : 38814

I have tried with using (see on your documentation)

err = deserializeJson(doc, client.getStream());

It work better, I can't expain why, if you have an explanation ?

But after a size the issue remain, If I m right String are limited to 65k, so it mean it's not possible to manage big json even the device have enought memory (and I haven't reach this limit yet) ? I know the issue is more from the lib HTTPClient (the buffer is realy empty), but I hope I m not the first one with this problem ...

[bblanchon]

Hi @Smanar,

Indeed, on ESP32, String has a maximum capacity of 65,535 characters.

Even if there were no size limit, String would still require a long contiguous buffer, which might not be available if the heap is fragmented.

To answer your question, String fails to allocate a large buffer to store the whole response, so getString() returns an empty string. getStream() works because deserializeJson() reads characters progressively.

Best regards, Benoit

[Smanar]

Ha yes, so it can explain too why the issue is random, depend of fragmentation and device memory. So the only solution possible ATM is using "Deserialization in chunks" described here https://arduinojson.org/v7/how-to/deserialize-a-very-large-document/ ? (And found a way for the HTTPClient lib don't stream all the data in one time, but packet by packet)

Edit: Have found a way using psram here https://www.upesy.fr/blogs/tutorials/get-more-ram-on-esp32-with-psram

[bblanchon]

ArduinoJson 7 works better than version 6 when the heap is fragmented. Did you check your memory consumption with the ArduinoJson Assistant?

[Smanar]

I m malking test ATM on a ESP32-2432S028R (ESP32-WROOM-32 4M flash) , so have selected a Denky32 board (RAM: [=== ] 27.7% (used 90720 bytes from 327680 bytes))

The json take 40 ko

On the assistant I have

• This document suffers from the "JSON in JSON" syndrome, so you may need to call deserializeJson() multiple times. The ArduinoJson Assistant doesn't support settings scenario.
• This is too big to fit in the RAM. See How to deserialize a very large document?

With using a filter

But something strange the memory used don't seem so big ....

Edit: Was with String, I m testing with Stream ATM.

Edit2: All are fine "theoricaly" with Stream.

Need to make a try with version 7 (I m stil on 6)

[bblanchon]

This is too big to fit in the RAM

I noticed that this alert sometimes stays on screen when it shouldn't. It's a bug I introduced recently. I'll fix this soon.

[bblanchon]

This document suffers from the "JSON in JSON" syndrome

Do you think it's a false positive?

[Smanar]

Do you think it's a false positive?

Honnestly I think, this json is not complex enought to be considered for "json in json"> https://www.domoticz.com/wiki/Domoticz_API/JSON_URL%27s#Get_status_of_all_devices.2C_including_hidden_ones

I m trying with V7, and for the moment I can't sucess, Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled

        //DynamicJsonDocument doc(buffer); //V6
        JsonDocument doc; //V7
        DeserializationError err;
       err = deserializeJson(doc, client.getStream());
        *JS = doc["result"].as<JsonArray>();

        if (!*JS)
        {
            Serial.println("Json not available\n");
            return false;
        }

    char buffer[4096];
    serializeJsonPretty(JS, buffer);
    Serial.println(buffer);

The buffer can be small (It's not the same json) but even for (auto i : JS) // Scan the array (only 1) make a Guru meditation.

[bblanchon]

Honnestly I think, this json is not complex enought to be considered for "json in json"

It doesn't have to be complex. Please share the JSON document in an attached file.

*JS = doc["result"].as<JsonArray>();

I cannot think of any way this can be correct. Please replace with:

JsonArray result = doc["result"];

You can use step 3 of the assistant, it will write most of the extraction code for you.

[Smanar]

You have the json here https://pastebin.com/NFbB3uSb The complete code is here https://github.com/Smanar/CYD-Domoticz-Remote/blob/test/src/core/ip_engine.cpp#L26

I can make you a minimal code (like here https://github.com/bblanchon/ArduinoJson/issues/2061#issuecomment-1962882784 have make your replacement) but atm the code give me a Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled on version 7 but work without problem on version 6.

It crash on serializeJsonPretty(JS, buffer); or for (auto i : JS) (but it's more far away in the code) and I don't use the debugger (IDK how it work with esp32 device)

Edit: The calling fonction is here https://github.com/Smanar/CYD-Domoticz-Remote/blob/test/src/core/data_setup.cpp#L247

But now after reflexion, I think I have found something.

[bblanchon]

"Color" : "{\"b\":109,\"cw\":0,\"g\":255,\"m\":3,\"r\":0,\"t\":0,\"ww\":0}",

That's, indeed, JSON in JSON :wink:

You cannot use a JsonArray once the JsonDocument is destructed: https://github.com/Smanar/CYD-Domoticz-Remote/blob/test/src/core/data_setup.cpp#L256

That's because JsonArray is a reference, which is one of ArduinoJson's fundamentals. Please read the documentation, the tutorials, or the book. ArduinoJson 6 had the same restriction; you were lucky that the code didn't crash.

[Smanar]

That's, indeed, JSON in JSON

Ha yes, for me was just a String.

BTW thx a lot. I thought of this just after posting, but as it have worked with V6, it didn't shock me. So have updated the code, and make tries with V6, and :

• No more crash
• It work fine, even with the bigger json I can generate.

I will send the code to an other user that have issue, but I m almost sure It will works fine.

Thx again for help.

[bblanchon]

You're welcome, @Smanar! Thank you for using ArduinoJson. Don't forget to cast a star to support the project :wink: