esp8266 - deserializing example from HTTP stream does not work

[MartijnvdB]

Hi,

Thanks for the awesome library. I bought the 'Mastering' book and I compliment you on your thoroughness.

Having said that, I cannot get the 'Reddit streaming' example from 7.3 to work: the output of

  client.find("\"children\":");
  client.find("[");

is zero.

• ArduinoJson 6.11.0
• ESP8266 (NodeMCU)
• Arduino IDE, 1.8.9

Entire sketch:

#define ARDUINOJSON_DECODE_UNICODE 1
#include <ArduinoJson.h>
#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

#define SSID <hidden>
#define PASSWORD <hidden>

void setup() {
  Serial.begin(115200);
  wifi_connect(); // connect to the local WiFi
}

// Connect to WiFi.
void wifi_connect() {
  WiFi.begin(SSID, PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(200);
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
} // wifi_connect

void loop() {

  WiFiClientSecure client;
  client.connect("www.reddit.com", 443);

  // Send the request
  client.println("GET /r/arduino.json HTTP/1.0");
  client.println("Host: www.reddit.com");
  client.println("User-Agent: Arduino");
  client.println("Connection: close");
  client.println();
  Serial.println("got data");

  // Jump to the "children" array
  client.find("\"children\":");
  client.find("[");
  // Alloc a huge JsonDocument
  DynamicJsonDocument doc(16 * 1024);
  do {
    // Deserialize the next post
    deserializeJson(doc, client);
    // Print score and title
    Serial.print(doc["data"]["score"].as<int>());
    Serial.print('\t');
    Serial.println(doc["data"]["title"].as<char*>());
  } while (client.findUntil(",", "]"));

  delay(20000);
}

[UPDATE] I figured perhaps I ran into a timeout while parsing, so I increased the timeout to 5000 ms but that did not solve the issue.

  client.setTimeout(5000);
  // Jump to the "children" array
  bool found = client.find("\"children\":");
  Serial.print("Found: ");
  Serial.println(found);

The result for found is still 0 (zero).

[bblanchon]

Hi @MartijnvdB,

Thank you very much for purchasing the book, I hope you like it.

I tried with core 2.5.2 and client.connect("www.reddit.com", 443) fails (returns 0). I don't know why it fails, but reverting the core to 2.4.2 solved the issue.

I cannot investigate further for now. Please keep me informed of your progress.

Best Regards, Benoit

[JasonMeredith1]

I am having the same issue... is there a solution other than reverting the core back to last years release?

`#define ARDUINOJSON_DECODE_UNICODE 1

include

include

include

define SSID "XXXXXXXXXXXX"

define PASSWORD "XXXXXXXXXXXX"

void setup() { Serial.begin(115200); wifi_connect(); // connect to the local WiFi }

// Connect to WiFi. void wifi_connect() { WiFi.begin(SSID, PASSWORD); while (WiFi.status() != WL_CONNECTED) { Serial.print("."); delay(200); }

Serial.println(""); Serial.println("WiFi connected"); Serial.print("IP address: "); Serial.println(WiFi.localIP()); } // wifi_connect

void loop() {

WiFiClientSecure client; client.connect("api.weather.com", 443);

// Send the request client.println("GET /v3/wx/forecast/daily/5day?postalKey=78249:US&units=e&language=en-US&format=json&apiKey=XXXXXXXXXXXXXXXXXXXXXX HTTP/1.0"); client.println("Host: api.weather.com"); client.println("User-Agent: Arduino"); client.println("Connection: close"); client.println(); Serial.println("got data");

// Jump to the "children" array client.find("\"children\":"); client.find("["); // Alloc a huge JsonDocument DynamicJsonDocument doc(16 1024); do { // Deserialize the next post deserializeJson(doc, client); // Print score and title Serial.print(doc["data"]["score"].as()); Serial.print('\t'); Serial.println(doc["data"]["title"].as<char>()); } while (client.findUntil(",", "]"));

delay(20000); }`

Any ideas?

-Jason

[bblanchon]

I don't know Jason; all I know is that it has nothing to do with ArduinoJson.

[MartijnvdB]

Didn't have much time to look into it either but I think it may have to do with TLS/certificates etc., based on some deep googling.

I made a similar project - seeking in a stream from another website over HTTP and that works. On the basis of that please feel free to close this issue as not a bug.

[bblanchon]

@MartijnvdB, please tell us if you find a solution.

[MartijnvdB]

The site I connect to doesn't use TLS, so I don't have to use WiFiSecure and I am not going to look into that.

The below code works for me (I hope I did not cut out too much of the code that isn't interesting for this case). It gets bus/tram travel information from a public REST API.

You'll have to pick a pick a stopareacode (from http://v0.ovapi.nl/stopareacode) and place that in 'XXXX' and you'll probably have to change the string 'HTM_' as well, to match the data that is returned.

Note that the site's JSON is not pretty - I cannot use a findUntil() because it's basically a whole bunch of objects lumped together, in stead of inside an aray. I parse until client.available() doesn't return anymore (and ignore the ensuing parser error ;) ).

#include <ArduinoJson.h>
#include <ESP8266WiFi.h>

#define DEBUG 1

const char* ovapi = "v0.ovapi.nl";
const char* stopareacode = "/stopareacode";
const char* stopareacodeid = "XXXX";
const char* resource = "/departures";
const char* useragent = "Mozilla/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"; // as suggested to use by ovapi to prevent blocking
const uint16_t request_interval_millis = 13000; // request interval, milliseconds
uint32_t previous_request;

void setup() {
  if (DEBUG) {
    Serial.begin(115200);
  }
  wifi_connect(); // connect to the local WiFi
}

void loop() {

  // HTTP GET request at interval in a non-blocking loop
  if ( (millis() - previous_request) > request_interval_millis ) {    // time to make a new request
    if (WiFi.status() == WL_CONNECTED) { // Check WiFi connection status
      // Connect to website
      WiFiClient client;
      client.setTimeout(2000);  // default is 1000 ms

      if (client.connect(ovapi, 80) ) {
        DEBUG && Serial.println("Connected to server");

        // Send request to the web server
        client.print(String("GET ") + stopareacode + "/" + stopareacodeid + "/" + resource + " HTTP/1.0\r\n" +
                     "Host: " + ovapi + "\r\n" +
                     "User-Agent: " + useragent + "\r\n" +
                     "Connection: close\r\n\r\n");

        // wait till response becomes available
        while (!client.available()) {
          delay(1);
        }

        // parse the response
        while (client.available()) {
          client.find("\"HTM_");  // passes start with HTM_
          client.find(":");

          // Allocate a big enough JsonDocument
          DynamicJsonDocument doc(1600);
          DeserializationError err = deserializeJson(doc, client);

          if (err && DEBUG) {
            //            Serial.print(F("deserializeJson() failed with code "));
            //            Serial.println(err.c_str());
          }
          else {
            uint32_t timingPointCode = doc["TimingPointCode"].as<int>();        // 320xxxxx
            char* destination = (char*)doc["DestinationName50"].as<char*>();    // is const char
            char* eta = (char*)doc["ExpectedArrivalTime"].as<char*>();
            char* tripstatus = (char*)doc["TripStopStatus"].as<char*>();       // PLANNED, DRIVING, PASSED

          } // no parse error

        } // client available

        client.stop();  // when done
        DEBUG && Serial.println("Disconnected from server");

        previous_request = millis();

      } // connected to server
      else {
        DEBUG && Serial.println("Could not connect to website");
      }
    } // WiFi connected
    else {
      DEBUG && Serial.println("Error in WiFi connection");
      wifi_connect();
    }
  } // new request

  // do other stuff here

} // loop

// Connect to WiFi.
void wifi_connect() {
  WiFi.begin(SSID, PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    DEBUG && Serial.print(".");
    delay(200);
  }

  if (DEBUG) {
    Serial.println("");
    Serial.println("WiFi connected");
    Serial.print("IP address: ");
    Serial.println(WiFi.localIP());
  }
} // wifi_connect

Hope this helps.