search

bblanchon / ArduinoJson

📟 JSON library for Arduino and embedded C++. Simple and efficient.
https://arduinojson.org
MIT License
6.72k stars 1.12k forks source link

Empty JSON doc most of the time #969

Closed jonmacd closed 5 years ago

jonmacd commented 5 years ago

I'm trying to parse some JSON data. I've been successful using other sets of data. But something seems to be going wrong with this particular one. It appears to work intermittently. From time to time it does successfully print the correct piece of data. None of the GET requests themselves are failing. They're all passing with code 200.

I'm using a NodeMCU (ESP8266) with the latest version of PlatformIO and the ArduinoJson library. I used arduinojson.org/v6/assistant to calculate the dynamic document capacity.

Code:

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ArduinoJson.h>

const char* ssid     = "xxxxx";
const char* password = "xxxxx";

// NHL URL
const char* host = "statsapi.web.nhl.com";
const char* url = "/api/v1/schedule?teamId=10";

int requestNum = 1;

void setup() {
  // Initialize Serial port
  Serial.begin(9600);
  while (!Serial) continue;

  Serial.println();
  Serial.println();
  Serial.print("Connecting to Wifi");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");  
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  Serial.print("Netmask: ");
  Serial.println(WiFi.subnetMask());
  Serial.print("Gateway: ");
  Serial.println(WiFi.gatewayIP());
}

void loop() {
  Serial.print("GET request ");
  Serial.print(requestNum);
  Serial.println(":");
  requestNum++;
  Serial.println("-------------------------------------");
  WiFiClient client;
  if (!client.connect(host, 80)) {
    Serial.println(F("Connection failed"));
    return;
  }
  Serial.println(F("Connected!"));
  client.println(String("GET ") + url + " HTTP/1.1");
  client.print("Host: ");
  client.println(host);
  client.println("User-Agent: ESP");
  client.println();
  delay(200);
  if (client.println() == 0) {
    Serial.println(F("Failed to send request"));
    return;
  }

  // Check HTTP status
  char status[32] = {0};
  client.readBytesUntil('\r', status, sizeof(status));
  if (strcmp(status, "HTTP/1.1 200 OK") != 0) {
    Serial.print(F("Unexpected response: "));
    Serial.println(status);
    return;
  }
  Serial.print("Get request status: ");
  Serial.println(status);

  // Skip HTTP headers
  char endOfHeaders[] = "\r\n\r\n";
  if (!client.find(endOfHeaders)) {
    Serial.println(F("Invalid response"));
    return;
  }

  // Allocate the JSON document
  // Use arduinojson.org/v6/assistant to compute the capacity.
  const size_t capacity = 2*JSON_ARRAY_SIZE(0) + 2*JSON_ARRAY_SIZE(1) + JSON_OBJECT_SIZE(1) + JSON_OBJECT_SIZE(2) + 7*JSON_OBJECT_SIZE(3) + JSON_OBJECT_SIZE(5) + JSON_OBJECT_SIZE(7) + JSON_OBJECT_SIZE(8) + JSON_OBJECT_SIZE(9) + 1707;
  DynamicJsonDocument doc(capacity);

  // Parse JSON object
  DeserializationError error = deserializeJson(doc, client);
  if (error) {
    Serial.print(F("deserializeJson() failed: "));
    Serial.println(error.c_str());
    return;
  }

  // Print data
  const char* copyright = doc["copyright"];
  Serial.println(copyright);
  Serial.println("Finished printing data.");
  Serial.println("Waiting a bit to get data again!");
  Serial.println("-------------------------------------");

  // Disconnect
  client.stop();
  delay(10000);
}

And here's a serial monitor output example of the code above to show you what I mean by inconsistant. Sometimes it will correctly grab the copyright item and sometimes it's empty. If it is successful it will always work the second time around as well.

Serial Monitor Example:

WiFi connected
IP address: 
192.168.1.61
Netmask: 255.255.255.0
Gateway: 192.168.1.1
GET request 1:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK
NHL and the NHL Shield are registered trademarks of the National Hockey League. NHL and NHL team marks are the property of the NHL and its teams. © NHL 2019. All Rights Reserved.
Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 2:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK
NHL and the NHL Shield are registered trademarks of the National Hockey League. NHL and NHL team marks are the property of the NHL and its teams. © NHL 2019. All Rights Reserved.
Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 3:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 4:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 5:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 6:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 7:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK
NHL and the NHL Shield are registered trademarks of the National Hockey League. NHL and NHL team marks are the property of the NHL and its teams. © NHL 2019. All Rights Reserved.
Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 8:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK
NHL and the NHL Shield are registered trademarks of the National Hockey League. NHL and NHL team marks are the property of the NHL and its teams. © NHL 2019. All Rights Reserved.
Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 9:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 10:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 11:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 12:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 13:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK
NHL and the NHL Shield are registered trademarks of the National Hockey League. NHL and NHL team marks are the property of the NHL and its teams. © NHL 2019. All Rights Reserved.
Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 14:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK
NHL and the NHL Shield are registered trademarks of the National Hockey League. NHL and NHL team marks are the property of the NHL and its teams. © NHL 2019. All Rights Reserved.
Finished printing data.
Waiting a bit to get data again!
-------------------------------------
GET request 15:
-------------------------------------
Connected!
Get request status: HTTP/1.1 200 OK

Finished printing data.
Waiting a bit to get data again!
-------------------------------------
bblanchon commented 5 years ago

Hi,

Thank you very much for this clear and complete report.

This issue was the perfect opportunity to try my new Arduino library: StreamUtils. I used the ReadLoggingStream to see what came back from nhl.com. Here is what I saw:

HTTP/1.1 200 OK

Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,accept,Origin,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Type,X-Requested-With,accept,Origin,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization
Cache-Control: max-age=20, public
Date: Sat, 20 Apr 2019 10:00:34 GMT
Server: Apache-Coyote/1.1
Vary: Accept-Encoding
Age: 3
X-Cache: Hit from cloudfront
Via: 1.1 ac987789ab8e4a7dbf75086d523e8589.cloudfront.net (CloudFront)
X-Amz-Cf-Id: -pXqZSRaoQGFBr51YrlerJx_j-Zr7NcK7LliVtk6CQnIuoIVpZuMQg==

6b9

That's a very common problem: the response uses chunked transfer encoding and 6b9 is the size of the chunk. We don't see the rest of the response because ArduinoJson stopped reading.

So: good news and bad news.

Good news for you: you can fix your program by replacing just one character in the source code. Simply change HTTP/1.1 to HTTP/1.0 in the request.

Bad news for me: ArduinoJson should return an InvalidInput instead of Ok when it reads 6b9. I'll fix that ASAP.

Regards, Benoit

bblanchon commented 5 years ago

After patching the code, I received the error IncompleteInput, I believe this is a race condition in WifiClient, because if I keep ReadLoggingStream in the way (ie if we read slowly), everything works fine. We saw something similar in #951. BTW, I'm using the latest core for ESP8266: 2.5.0.

Hopefully, you'll not have the same problem. If you do, maybe I can create a new class in StreamUtils to slow down the read part.

jonmacd commented 5 years ago

Hi Benoit, Thanks for looking into this. I'm getting deserializeJson() failed: IncompleteInput now.

How did you use your StreamUtils library to slow down the reading exactly?

bblanchon commented 5 years ago

Hu @jonmacd,

I spent a lot of time trying to find a reliable solution and my conclusion is: there is definitely something wrong with WiFiClient.

What I tried:

I performed all my tests with ESP8266 core for Arduino 2.5.0.

It's not related to ArduinoJson; simply try to read the whole content with Stream::read() and you'll see that the stream is truncated.

Here are my recommendations: try another version of the core or change the microcontroller as Bill did. Anyways, please keep me informed with your results.

Regards, Benoit

jonmacd commented 5 years ago

I do own an ESP32, however I currently have it in use somewhere else. There are quite a few shortcomings of the ESP8266 that I'm not a fan of (lack of WPA-Enterprise for example). I might pickup some more ESP32s and switch over to using them for all my WiFi based projects. The ESP32 seems to be much better supported these days.

Thanks again for your help Benoit!