ArduinoJson prefer Safari? Maybe this is not a bug!

[pin010]

Describe the bug
I discovered a small problem when forwarding a json document larger than about 160 bytes from a Chrome or Edge browser to a server located in an ESP8266 with bblanchon/ArduinoJson@^7.0.3 libraries

Troubleshooter report
Here is the report generated by the ArduinoJson Troubleshooter:

1. The program uses ArduinoJson 7
2. The issue happens at run time
3. The issue concerns deserialization
4. deserializeJson() returns InvalidInput
5. Input comes from HTTP
6. Program uses an unknown HTTP library

Environment
Here is the environment that I used:

• Microcontroller: ESP8266
• Core/runtime: ESP8266 core for Arduino
• IDE: VSCode / PlatformIO

Reproduction
Please see https://github.com/pin010/ArdJsonVsChrom

    server.on("/update_settings", HTTP_POST, [](AsyncWebServerRequest * request){}, NULL,
            [](AsyncWebServerRequest * request, uint8_t *data, size_t len, size_t index, size_t total) {
        long ms = millis();
        Serial.printf("\nA %ld update_settings len=%d", ms, len);
        JsonDocument doc;
        DeserializationError error = deserializeJson(doc, data);
        ms = millis();
        Serial.printf("\nB %ld update_settings index=%d", ms, index);
        if (error) {
            ms = millis();
            Serial.printf("\nC %ld update_settings Error decoding JSON: ", ms);
            Serial.println(error.c_str());
            request->send(400);
            return;
        } else if(doc.containsKey("settingsID")){
            uint16_t id = (doc["settingsID"]);
            Serial.printf("\nid=%d  ",id);
            String strdoc;
            serializeJsonPretty(doc, strdoc);
            Serial.println(strdoc);
        }
        doc.clear();
        request->send(200);
    });

Compiler output
If relevant, include the complete compiler output (i.e. not just the line that contains the error.)

Program output
If relevant, include the repro program output.

Expected output:

Correct if browser is SAFARI:
A 74611 update_settings len=94 
B 74612 update_settings index=0
id=1  {
  "settingsID": 1,
  "timeout_cavitazione": 3000, 
  "timeout_isolato": 4000,     
  "timeout_sovraccarico": 1000 
}

A 75099 update_settings len=188
B 75100 update_settings index=0
id=2  {
  "settingsID": 2,
  "timeout_cavitazione": 3000, 
  "timeout_isolato": 4000,     
  "timeout_sovraccarico": 1000,
  "timeout_dopo_cavitazione": 5000,
  "timeout_dopo_isolato": 5000,
  "timeout_dopo_sovraccarico": 60000
}

A 75437 update_settings len=336
B 75438 update_settings index=0
id=3  {
  "settingsID": 3,
  "timeout_cavitazione": 3000,
  "timeout_isolato": 4000,
  "timeout_sovraccarico": 1000,
  "timeout_dopo_cavitazione": 5000,
  "timeout_dopo_isolato": 5000,
  "timeout_dopo_sovraccarico": 60000,
  "timeout_attesa_read_ma": 50,
  "ma_overload": 2400,
  "ma_min_cavitazione": 350,
  "ma_max_cavitazione": 880,
  "ma_isolato": 50,
  "k_global": 45,
  "k_global_remote": 42
}

Actual output:

Wrong if browser is CHROME or EDGE
A 31084 update_settings len=94 
B 31085 update_settings index=0
id=1  {
  "settingsID": 1,
  "timeout_cavitazione": 3000, 
  "timeout_isolato": 4000,    
  "timeout_sovraccarico": 1000
}

A 31495 update_settings len=124
B 31496 update_settings index=0
C 31496 update_settings Error decoding JSON: IncompleteInput

A 31500 update_settings len=64
B 31500 update_settings index=124
C 31501 update_settings Error decoding JSON: InvalidInput

A 31893 update_settings len=124
B 31894 update_settings index=0
C 31894 update_settings Error decoding JSON: InvalidInput

A 31897 update_settings len=212
B 31897 update_settings index=124

[bblanchon]

Hi @pin010,

Expected output update_settings len=188

Actual output update_settings len=124

IncompleteInput seems perfectly appropriate since the end of the document is missing. I don't think the problem comes from ArduinoJson but from AsyncWebServer.

Please try to reproduce the issue without ArduinoJson and open an issue at AsyncWebServer. It could be related to chunked transfer encoding.

Best regards, Benoit

[pin010]

Hi Benoit, you gave me excellent advice, I made the suggested change, the result is indisputable. ArduinoJson performs great. I have to look on ESPAsyncWebServer for the problem. Best regards, Pin010

result with Chrome: fragmented update_settings mSec=112343 len=124 index=0 total=336 update_settings mSec=112351 len=212 index=124 total=336

result with Safari: whole update_settings mSec=155982 len=336 index=0 total=336

Thank you

[bblanchon]

Cool. Let me know how you solve the issue.

[pin010]

Hi, this way, I don't know if it's the best way, but it works.

server.on(
    "/update_settings", HTTP_POST, [](AsyncWebServerRequest *request) {}, NULL,
    [](AsyncWebServerRequest *request, uint8_t *data, size_t len, size_t index, size_t total)
    {
        if (total > BUFFER_SIZE)
        {
            request->send(400, "total Oversize");
        }
        else
        {
            for (size_t i = 0; i < len; i++)
            {
                    buffer[bufferIndex++] = data[i];
            }
            if (bufferIndex >= total)
            {
                bufferIndex = 0;
                JsonDocument doc;
                DeserializationError error = deserializeJson(doc, buffer);
                if (error)
                {
                    Serial.println(error.c_str());
                    request->send(400, error.c_str());
                    return;
                }
                else if (doc.containsKey("settingsID"))
                {
                    // do something
                }
                doc.clear();
            }
            request->send(200);
        }
    });

Best regards, Pin010

[bblanchon]

So, if I understand correctly, you accumulate the data in a global buffer and call deserializeJson() when the request is complete.

[pin010]

That's exactly how it is! I've been poking around espasyncwebserver and I found this solution which all in all works.... Greetings