BIG array problem with version 5

Tried your assistant for the following API https://api.binance.com/api/v3/ticker/price on an ESP8266.

Assistant gives the size as 34962 bytes with:

const size_t bufferSize = JSON_ARRAY_SIZE(419) + 419*JSON_OBJECT_SIZE(2) + 14630;

But from the assistant the entries are coming back as empty for:

const char* root_0_symbol = root_[0]["symbol"]; // "ETHBTC"

When I try:

Serial.println(root_0_symbol);

The ESP8266 is not crashing and I get the same results if I just set the json manually rather than via an API call i.e.

const char* json = "[{\"symbol\":\"ETHBTC\",..........................................\"}]";

Any idea why it's not parsing the json array and not giving any errors?

Hi @pieman64,

The JSON document is too big to fit in memory; that's why the parsing fails. You can confirm this by calling JsonArray::success().

You can solve this problem by parsing each object independently. Please read Can I parse a JSON input that is too big to fit in memory?.

Regards, Benoit

@bblanchon thanks Benoît, just bought your book with the 20% GitHub discount code.

@bblanchon I have used your Wunderground Weather forecast example from your book and I can now get 50 tickers as shown below:

Return code is: 200
1   ETHBTC : 0.02599300
2   LTCBTC : 0.00697800
3   BNBBTC : 0.00144290
4   NEOBTC : 0.00172500
5   QTUMETH : 0.01849800
6   EOSETH : 0.02089400
7   SNTETH : 0.00016350
8   BNTETH : 0.00571000
9   BCCBTC : 0.07908100
10  GASBTC : 0.00049800
11  BNBETH : 0.05544500
12  BTCUSDT : 3372.94000000
13  ETHUSDT : 87.71000000
14  HSRBTC : 0.00041400
15  OAXETH : 0.00084540
16  DNTETH : 0.00015070
17  MCOETH : 0.02156600
18  ICNETH : 0.00166300
19  MCOBTC : 0.00056000
20  WTCBTC : 0.00026560
21  WTCETH : 0.01023800
22  LRCBTC : 0.00001059
23  LRCETH : 0.00040506
24  QTUMBTC : 0.00048100
25  YOYOBTC : 0.00000374
26  OMGBTC : 0.00037300
27  OMGETH : 0.01438100
28  ZRXBTC : 0.00008823
29  ZRXETH : 0.00340290
30  STRATBTC : 0.00018680
31  STRATETH : 0.00717000
32  SNGLSBTC : 0.00000264
33  SNGLSETH : 0.00010131
34  BQXBTC : 0.00002594
35  BQXETH : 0.00100260
36  KNCBTC : 0.00003832
37  KNCETH : 0.00147590
38  FUNBTC : 0.00000113
39  FUNETH : 0.00004314
40  SNMBTC : 0.00000568
41  SNMETH : 0.00022000
42  NEOETH : 0.06620000
43  IOTABTC : 0.00006554
44  IOTAETH : 0.00251905
45  LINKBTC : 0.00006511
46  LINKETH : 0.00250041
47  XVGBTC : 0.00000167
48  XVGETH : 0.00006432
49  SALTBTC : 0.00006120
50  SALTETH : 0.00236400
Free RAM: 45312
n equals: 50

I have: DynamicJsonBuffer jb(80);

As suggested by your Assistant (for version 5) of the largest json pair of {"symbol":"BTCUSDTABCDE","price":"3369.23000000"}

But the API returns 418 pairs and the ESP8266 crashes when I go much above 50 pairs even though that shows a healthy 45kb of free memory.

The site uses SSL but http will redirect to https. I am using the following notation to make the http request and it seems fine up to 50 pairs. http.begin("https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT", "41 82 D2 BA 64 E3 36 F1 3C 5E 49 05 2A A0 AA CB D0 F7 2B B7");

I think it might be more of a problem with the https request that the parsing of the json object. Any thoughts?

I think it might be more of a problem with the https request that the parsing of the json object.

I think so.

Please use the EspExeptionDecoder to decode the backtrace. As an alternative, you can use ArduinoTrace to find exactly which line causes the crash.

BTW, thank you very much for purchasing the book. I hope you'll like it.

@bblanchon good books thanks.

I have moved to ESP8266 core 2.5.0-beta1 that was released a few hours ago. I believe it includes the new BearSSL implementation. It also has a proper https stream facility. Below is the StreamHttpsClient example modified to work with the Binance API. It seems to be rock sold and line 81 print the 17,000+ characters to Serial. A constant 46,176 free memory after each 5 second loop.

/*
   StreamHTTPClientBinance.ino core 2.5.0-beta1 12/12/18
*/

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>

ESP8266WiFiMulti WiFiMulti;

void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("SSID", "PASSWORD");

}

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) 
  {
    std::unique_ptr<BearSSL::WiFiClientSecure> client(new BearSSL::WiFiClientSecure);
    bool mfln = client->probeMaxFragmentLength("api.binance.com", 443, 1024);
    Serial.printf("\nConnecting to Binance\n");
    Serial.printf("Maximum fragment Length negotiation supported: %s\n", mfln ? "yes" : "no");
    if (mfln) {
      client->setBufferSizes(1024, 1024);
    }

    Serial.print("[HTTPS] begin...\n");

    // configure server and url
    const uint8_t fingerprint[20] = {0x41, 0x82, 0xD2, 0xBA, 0x64, 0xE3, 0x36, 0xF1, 0x3C, 0x5E, 0x49, 0x05, 0x2A, 0xA0, 0xAA, 0xCB, 0xD0, 0xF7, 0x2B, 0xB7};      

    client->setFingerprint(fingerprint);

    HTTPClient https;

    if (https.begin(*client, "https://api.binance.com/api/v3/ticker/price")) 
    {
      Serial.print("[HTTPS] GET...\n");
      // start connection and send HTTP header
      int httpCode = https.GET();
      if (httpCode > 0) {
        // HTTP header has been send and Server response header has been handled
        Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

        // file found at server
        if (httpCode == HTTP_CODE_OK) {

          // get lenght of document (is -1 when Server sends no Content-Length header)
          int len = https.getSize();

          // create buffer for read
          static uint8_t buff[128] = { 0 };

          // read all data from server
          while (https.connected() && (len > 0 || len == -1)) {
            // get available data size
            size_t size = client->available();

            if (size) {
              // read up to 128 byte
              int c = client->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));

              // write it to Serial
              Serial.write(buff, c);

              if (len > 0) {
                len -= c;
              }
            }
            delay(1);
          }

          Serial.println();
          Serial.print("[HTTPS] connection closed or file end.\n");

        }
      } else {
        Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
      }

      https.end();
    } else {
      Serial.printf("Unable to connect\n");
    }
  }
  Serial.printf("Free RAM: %d\n", ESP.getFreeHeap());
  Serial.println("Wait 5s before the next round...");
  delay(5000);
}

I have tried merging the code above with the modified "Wunderground Weather" sketch but I am struggling to find the suitable place to create the stream.

Binance doesn't appear to support MFLN (Maximum Fragment Length Negotiation) so instead of a buffersize of 1024 bytes it looks to be working with 128 bytes.

The stack exception decoder is not really throwing up any clues. Will try your version when I have time.

Any idea where I should enter the following line of code in the loop to create the stream?

Stream &response = https.getStream();

OK I found where to insert the stream and it is fine for a quantity of 50 tickers but crashes beyond this level.

Stack trace suggests a problem with available memory as per umm_malloc.c at line 1759 which is covered in this section of code below:

void umm_free( void *ptr ) {

  ptr = GET_UNPOISONED(ptr);

  /* check poison of each blocks, if poisoning is enabled */
  if (!CHECK_POISON_ALL_BLOCKS()) {
    return;
  }

  /* check full integrity of the heap, if this check is enabled */
  if (!INTEGRITY_CHECK()) {
    return;
  }

  _umm_free( ptr );
}

I will see if I can work out what's going wrong here.

Thanks for the update @pieman64. Is there any way I can help?

@bblanchon I have dropped the API key and query from the function you have in Weather Underground example as they are hard coded in my sketch.

You only had a quantity of 10 for number of days forecast whereas I have from 40 to 420 for the tickers I require. Also you were parsing about 25kb whereas I have about 34kb.

I am slowly making some progress though. I can manually parse the stream and Serial Monitor shows the following:

411:    {"symbol":"ETHPAX","price":"85.28000000"}
412:    {"symbol":"XRPPAX","price":"0.29360000"}
413:    {"symbol":"EOSPAX","price":"1.84430000"}
414:    {"symbol":"XLMPAX","price":"0.10214000"}
415:    {"symbol":"RENBTC","price":"0.00000592"}
416:    {"symbol":"RENBNB","price":"0.00422000"}
417:    {"symbol":"XRPTUSD","price":"0.29315000"}
418:    {"symbol":"EOSTUSD","price":"1.83760000"}
419:    {"symbol":"XLMTUSD","price":"0.10202000"}]
[HTTPS] connection closed or file end.
https stream closed
Free RAM: 39608
Successful API calls: 687
Running time in seconds: 2399
Loop time in ms: 4286
Calls > 5s: 58
[HTTPS] begin...
[HTTPS] GET...
[HTTPS] GET... code: 200
HTTPS GET size -1
1:  [{"symbol":"ETHBTC","price":"0.02606600"}
2:  {"symbol":"LTCBTC","price":"0.00710800"}
3:  {"symbol":"BNBBTC","price":"0.00140540"}
4:  {"symbol":"NEOBTC","price":"0.00170400"}
5:  {"symbol":"QTUMETH","price":"0.02043700"}
6:  {"symbol":"EOSETH","price":"0.02151400"}
7:  {"symbol":"SNTETH","price":"0.00016112"}

So this shows the full 419 tickers on a separate line, followed by some debugging and then the start of the next API call.

The 40kb of free RAM is misleading as it drops below 17Kb during the manual parsing.

With the current code 10% of the time it fails to parse the stream correctly and breaks around ticker number 332. I can reduce the 10% to below 1% if I tweak the code (i.e. don't display the data in Serial Monitor etc). But over time I will be adding more code, for writing data to disk etc so 10% might still fail.

The reason for the failure and partly why the ESP8266 crashes is because it takes more than 8 seconds to exit the https stream loop when it encounters a problem. Even with software WDT disabled it still comes up with the hardware WDT that kicks in after 6 seconds.

There are two sequences that the code doesn't handle well. In addition to the normal json string the API receives "2 nibbles". One before and one after the json string. The nibble before the json is not a problem but the nibble after is.

As shown in the log below this terminating nibble is being picked up in Serial Monitor whilst it is still parsing the json string.

328:    {"symbol":"CVCETH","price":"0.00058701"}
329:    {"symbol":"CVCBNB","price":"0.01090000"}
330:    {"symbol":"THETABTC","price":"0.00001271"}
331:    {"symbol":"THETAETH","price":"0.00048671"}
332:    {"symbol":"THETABNB","price":"0.00895000"}
333:    {"sym
e19
bol":"XRPBNB","price":"0.06393000"}
334:    {"symbol":"TUSDUSDT","price":"1.01230000"}
335:    {"symbol":"IOTAUSDT","price":"0.21400000"}
336:    {"symbol":"XLMUSDT","price":"0.10301000"}
337:    {"symbol":"IOTXBTC","price":"0.00000208"}
338:    {"symbol":"IOTXETH","price":"0.00007992"}
339:    {"symbol":"QKCBTC","price":"0.00001246"}

It looks to be a fixed nibble of e19 and I might be able to code around it. In the log above and most of the time the code just runs on without a problem. But 10% of the time it halts for more than 8 seconds without going to the end of the stream, as shown below.

324:    {"symbol":"SKYBNB","price":"0.19400000"}
325:    {"symbol":"EOSUSDT","price":"1.84410000"}
326:    {"symbol":"EOSBNB","price":"0.39980000"}
327:    {"symbol":"CVCBTC","price":"0.00001521"}
328:    {"symbol":"CVCETH","price":"0.00058755"}
329:    {"symbol":"CVCBNB","price":"0.01090000"}
330:    {"symbol":"THETABTC","price":"0.00001270"}
331:    {"symbol":"THETAETH","price":"0.00048495"}
332:    {"symbol":"THETABNB","price":"0.00895000"}

[HTTPS] connection closed or file end.
https stream closed
Free RAM: 17960
Successful API calls: 62
Running time in seconds: 320
Loop time in ms: 8677
Calls > 5s: 6
[HTTPS] begin...
[HTTPS] GET...
[HTTPS] GET... code: 200
HTTPS GET size -1
1:  [{"symbol":"ETHBTC","price":"0.02604600"}
2:  {"symbol":"LTCBTC","price":"0.00711400"}
3:  {"symbol":"BNBBTC","price":"0.00140090"}
4:  {"symbol":"NEOBTC","price":"0.00170500"}
5:  {"symbol":"QTUMETH","price":"0.02040300"}

As per the log it took 8677 ms before quitting the https stream function. I have tried running a timer to exit the loop if it takes more than 5000 ms but it's not working.

These are just some of the reasons I am struggling with your parser but hopefully I will get there.

OK i have bumped up the SM baud rate from 115200 to 250000 and overclocked the ESP8266 to 160Mhz and no problems after more than 270 iterations.

412:    {"symbol":"XRPPAX","price":"0.29396000"}
413:    {"symbol":"EOSPAX","price":"1.84110000"}
414:    {"symbol":"XLMPAX","price":"0.10243000"}
415:    {"symbol":"RENBTC","price":"0.00000597"}
416:    {"symbol":"RENBNB","price":"0.00416000"}
417:    {"symbol":"XRPTUSD","price":"0.29345000"}
418:    {"symbol":"EOSTUSD","price":"1.84080000"}
419:    {"symbol":"XLMTUSD","price":"0.10244000"}]
[HTTPS] connection closed or file end.
https stream closed
Free RAM: 39608
Successful API calls: 272
Running time in seconds: 819
Loop time in ms: 2822
Calls > 5s: 0
[HTTPS] begin...
[HTTPS] GET...
[HTTPS] GET... code: 200
HTTPS GET size -1
1:  [{"symbol":"ETHBTC","price":"0.02606400"}
2:  {"symbol":"LTCBTC","price":"0.00712000"}
3:  {"symbol":"BNBBTC","price":"0.00141970"}
4:  {"symbol":"NEOBTC","price":"0.00170100"}
5:  {"symbol":"QTUMETH","price":"0.02040000"}
6:  {"symbol":"EOSETH","price":"0.02155100"}

This is progress.

From my experience, random failures like that are very often caused by memory fragmentation. Tips: avoid using String objects in the loop, and use a StaticJsonBuffer.

Also, the e19 looks like a chunk size from the chunked Transfer Encoding. I'm surprised that the HTTP library does not handle it. Most of the time, you can get rid of the chunked transfer encoding by using HTTP 1.0 instead of 1.1.

@bblanchon I am now back on trying to parse with your library. I have dropped the ticker count down to 10 to match your WU example. Free RAM for your example, even during the stream parsing is 46K whereas I am down to 25K.

I am trying to copy line for line and I have a simpler structure than you, just 2 Strings where you have 2 Strings and 2 Integers. I know I have 35kb to parse but I only have a buffer of 80 compared with your 2048 because I have a flat json array rather than your nested array.

When I take the tickers up to 55 the RAM doesn't drop too much, down from 25kb to 22kb. But it's enough to cause problems. Plus I then have to watch for the WDT kicking in as 55 takes quite a long time to process.

The only difference I can find is you are using an http address and I am using https with fingerprint. WU accepts http (and https) but not mine. I looked up the fingerprint for WU and got two different ones but couldn't find a format that would connect to WU via https.

@bblanchon I was using a dynamic buffer of just 80 bytes as the WU example is dynamic. 80 bytes seems enough for each ticker pair like: {"symbol":"LTCABCDBTC","price":"99990.00711400"}

With a Static buffer do you want it to be a huge 35kb with:

  const size_t bufferSize = JSON_ARRAY_SIZE(419) + 419*JSON_OBJECT_SIZE(2) + 14630; 
  StaticJsonBuffer<bufferSize> jb;

Dynamic buffer of 80 bytes seems to be performing much better than any value of Static buffer. The only String's I have in the code are the two in the Structure for ticker symbol and price. Are these OK as String?

Your library is parsing ok for 50 of the 419 ticker pairs. Memory and reliability for up to 50 tickers is shown below:

LINKBTC : 0.00006287
LINKETH : 0.00241625
XVGBTC : 0.00000185
XVGETH : 0.00007093
SALTBTC : 0.00005890
SALTETH : 0.00226000
Free RAM: 46616
Successful API calls: 209
Running time in seconds: 376
Loop time in ms: 1590
Calls > 5s: 0
Maximum tickers is 50
Free RAM: 26800
ETHBTC : 0.02600200
LTCBTC : 0.00709600
BNBBTC : 0.00140550
NEOBTC : 0.00173000
QTUMETH : 0.02030000
EOSETH : 0.02110700
SNTETH : 0.00016065
BNTETH : 0.00555000

It's now over 300 iterations with no API calls taking over 5s and most are done in 1.7s. Almost 47kb memory outside the function and normally 25K to 27K inside the function but I notice it sometimes shows as much as 32kb free.

As soon as I push the tickers up to 55 it is lucky if it can do 3 or 4 iterations without the ESP8266 crashing. It's like the WU API offering a 365 day forecast and only being able to parse 44 days.

@bblanchon I finally cracked it.

There are now 426 tickers available. I removed the struct and functions so the https.begin() loop simply contains the following code:

      // Get a reference to the apijson
      Stream &apijson = https.getStream();
      if(!apijson.find("[")) // found start of ticker array
      {
        Serial.write("Parse failed\n");
        return; 
      } 
      for (unsigned int i = 0; i < maxtickqty; i++)
      {
            DynamicJsonBuffer jb(80);
            JsonObject &obj = jb.parseObject(apijson);
            if (obj.success())
            {   
              Serial.write(obj["symbol"].as<char *>()); // can use const char *tickname etc but not needed
              Serial.write (":");
              Serial.write(obj["price"].as<char *>());
            }   
            else
            { 
              Serial.printf("End: Free RAM: %d\t", ESP.getFreeHeap()); // almost 20K before https function ends
              https.end();
              Serial.printf("%d\n", ESP.getFreeHeap());  // almost 43K when https function ends
              countAPIcalls++;
              Serial.printf("Successful API calls: %i\n", countAPIcalls);
              Serial.printf("Running time in seconds: %lu\n", (millis() / 1000));
              Serial.printf("Loop time in ms: %lu\n", (millis() - looptime));
              if((millis() - looptime) > 5000)
              {
                countLongcalls++;  
              }
              Serial.printf("Calls > 5s: %i\n", countLongcalls);
              looptime = millis(); // reset looptime - approx 2968 ms for 426 tickers
              return; 
            }
            Serial.write ("\n");
            apijson.findUntil(",", "]");  // move to next ticker until we get to the end
      }

It takes a little under 3s to collect the 35K and parse out the 426 tickers. As I need to use https, rather than http, memory is down to 20K during the routine and 43K when it finishes.

XRPUSDC:0.28503000
EOSUSDC:1.93250000
XLMUSDC:0.09641000
USDCUSDT:1.00910000
End: Free RAM: 19992    42776
Successful API calls: 571
Running time in seconds: 1759
Loop time in ms: 2968
Calls > 5s: 1

Connecting
GET... code: 200
ETHBTC:0.02650200
LTCBTC:0.00807000
BNBBTC:0.00143730
NEOBTC:0.00180500

Hi @pieman64,

If I understand correctly, it works now, right? To me, the problem seems related to the allocation of the Strings.

You can easily avoid the String in the ticker struct by using a char[]:

struct Ticker {
    char symbol[8];
    double price;
};

And then extract the values as shown in JsonConfigFile.ino:

ticker.price= obj["price"];
strlcpy(ticker.symbol, root["symbol"] | "???", sizeof(ticker.hostname));

Notice the use of strlcpy() to avoid the buffer overrun and the "or" operator to stop the propagation of nullptr.

Regards, Benoit

Yes @bblanchon your parser is handling the 426 tickers ok now. When I have time I will switch to char[] in the earlier versions of the code and strlcpy() to see if that works ok. Thanks for your assistance.

@bblanchon I take it that your line of code: strlcpy(ticker.symbol, root["symbol"] | "???", sizeof(ticker.hostname)); Should actually be: strlcpy(ticker.symbol, root["symbol"] | "???", sizeof(ticker.symbol));

With "???" being a default ticker name like "BTCUSDT".

@changing to char still doesn't allow me to go beyond 45 tickers with the "Weather Underground" sketch format.

I will post the 3 files and maybe you can take a look to see if there are any obvious issues. Tickers.h just has the structure and the function definition. symbol is set to a size of 12. Most tickers are 8 or 9 characters but there are a few with 10 or 11 characters.

#pragma once

struct BinPricesStruct 
{
    char symbol[12];
    double price;
};

int fetchBinPricesStruct(BinPricesStruct *thetickers, unsigned int maxTickers);

This is Tickers.cpp

#include <ArduinoJson.h>
#include <ESP8266HTTPClient.h>     // needed for BearSSL::WiFiClientSecure
#include <ESP8266WiFi.h>           // BearSSL is in general ESP8266WiFi.h now
#include "Tickers.h"

// Skip all bytes until we receive the start of the json string from Binance
static bool jumpToJsonStart(Stream &stream) 
{
  return stream.find("[");    // required for array of tickers like https://api.binance.com/api/v3/ticker/price
}

// Skip all bytes until we found a comma or a closing bracket
static bool jumpToNextElement(Stream &stream) 
{
  return stream.findUntil(",", "]");    // array of tickers
}

static bool deserializeBinPricesStruct(Stream &response, BinPricesStruct &t) 
{  
  //const size_t bufferSize = JSON_ARRAY_SIZE(419) + 419*JSON_OBJECT_SIZE(2) + 14630; 
  //const size_t bufferSize = 4800;
  //StaticJsonBuffer<bufferSize> jb;

  //DynamicJsonBuffer jb(2048);  // bigger might be better - takes about 2.6s less 1s delay = 1.6s for 10 tickers
  DynamicJsonBuffer jb(80);  // was 2048 for Weather Underground but Binance only needs about 80 for each ticker as it's a flat array
                             // Assistant shows {"symbol":"BTCABCUSDT","price":"3369.23000000"} as needing 78 bytes
                             //takes about 2.5s less 1s delay = 1.5s for 10 tickers, so probably a bit better than 2048 buffer

  JsonObject &obj = jb.parseObject(response);

  if (!obj.success())  // for multiple ticker
  {
    Serial.println("Failed to parse object");
    return false;
  }

  strlcpy(t.symbol, obj["symbol"] | "BTCUSDT", sizeof(t.symbol));
  t.price= obj["price"];  

  return true;
}

int fetchBinPricesStruct(BinPricesStruct *thetickers, unsigned int maxTickers)
{
  //ESP.wdtDisable();  // disable Software WDT but hardware WDT will still be active (max 6 seconds)

  std::unique_ptr<BearSSL::WiFiClientSecure> client(new BearSSL::WiFiClientSecure);

  // configure server and url    
  const uint8_t fingerprint[20] = {0x41, 0x82, 0xD2, 0xBA, 0x64, 0xE3, 0x36, 0xF1, 0x3C, 0x5E, 0x49, 0x05, 0x2A, 0xA0, 0xAA, 0xCB, 0xD0, 0xF7, 0x2B, 0xB7};      
  client->setFingerprint(fingerprint);

  HTTPClient https;
  if (https.begin(*client, "https://api.binance.com/api/v3/ticker/price")) 
  {  
    if (https.GET() != 200) {
      Serial.println(F("HTTP request failed"));
      return 0;
    }
    Stream &response = https.getStream(); // Get a reference to the response
    Serial.printf("Free RAM: %d\n", ESP.getFreeHeap());
    if (!jumpToJsonStart(response)) 
    {
      Serial.println(F("Opening [ is missing from response"));     // OK for array of tickers
      return 0;
    }  

    int n = 0;
    while (n < maxTickers) 
    {
      // We are now in the array, we can read the objects one after the other
      if (!deserializeBinPricesStruct(response, thetickers[n++])) 
      {
        Serial.printf("Free RAM: %d\n", ESP.getFreeHeap());
        Serial.println(F("Failed to parse data"));
        break;
      }

      // After reading a price object, the next character is either a
      // comma (,) or the closing bracket (])    
      if (!jumpToNextElement(response))
      {
        Serial.println(F("Failed to get next item"));
        break;
      }  
    } 
    https.end(); 
    return n;
  }
}

And this is AllTickersV11.ino set to 45 tickers and Serial Monitor at 250000 baud.

/* AllTickerV11.ino API: https://api.binance.com/api/v3/ticker/price
* typical API call is 35kb so has to be processed a symbol at a time
* this works ok for about 45 tickers but crashes at 55 or more.

* V11 revised struct for char and double  with 45 tickers memory available in the loop is 19544, down to 18032 for 50 tickers
*/
#include <ESP8266WiFi.h>
#include "Tickers.h"

const char *ssid = "xxxx";
const char *password = "xxxxx";
const unsigned int maxTickers  =  45;       // maximum number of tickers on Binance - currently 426 required but unstable over 50
unsigned int countAPIcalls = 0;
unsigned long looptime = 0;
unsigned int countLongcalls = 0;

void setup() {
  //Serial.begin(115200); // Initialize Serial Port
  Serial.begin(250000); // Initialize Serial Port at 250000 baud
  delay(10);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print("Connecting to WiFi");
    delay(500);
  } 
  Serial.println(); 
  looptime = millis(); // start looptime
}

void getTickers()
{
  // Process ticker prices   
  Serial.printf("Maximum tickers is %u\n", maxTickers); 
  BinPricesStruct thetickers[maxTickers];  
  int n = fetchBinPricesStruct(thetickers, maxTickers); 

  // Loop through each ticker
  for (int i = 0; i < n; i++) {    
    BinPricesStruct &t = thetickers[i];  // Get a reference to the symbol (not a copy)  
    Serial.print(t.symbol);
    Serial.print(" : ");
    Serial.println(String(t.price,8));
  }

  Serial.printf("Free RAM: %d\n", ESP.getFreeHeap());
  countAPIcalls++;
  Serial.printf("Successful API calls: %u\n", countAPIcalls);
  Serial.printf("Running time in seconds: %lu\n", (millis() / 1000));
  Serial.printf("Loop time in ms: %lu\n", (millis() - looptime));
  if((millis() - looptime) > 5000)
  {
    countLongcalls++;  
  }
  Serial.printf("Calls > 5s: %u\n", countLongcalls);
  // reset looptime - approx 1700 ms with print on and buff size 80, 47K memory free when function finishes and 26K to 32K inside the function
  looptime = millis(); 
}

void loop() 
{
  getTickers();
  delay(1);
}

Hi @pieman64,

There is something you need to know about the ESP8266 core for Arduino: it limits the stack size to 4KB and raises an exception if you use more. It means that (sadly) you cannot put big variables in the stack.

In your case, you need to move thetickers either to the heap or to the globals. I believe that's the reason for the crash.

After making this change, switch to a StaticJsonBuffer to avoid the dynamic allocation in the loop; it should slightly increase the speed.

Regards, Benoit

@bblanchon fortunately with the ESP8266 you can disable exceptions and it saves another 4kb of memory. In a PHP script I have I changed from serialize to json_encode and when I paste the data into your assistant I notice the size is "only" 14,786 bytes (for 429 tickers) and not the 35,000 bytes I thought it was.

With the latest version of the code your parser is working fine "99%" of the time. You will need to scroll right in the Serial Monitor output below for the full details. The code is set to find one particular ticker (Bitcoin) from the 429 currently available but it is parsing all of them.

Parsing normally takes about 3.1 seconds but 1 in 100 times it fails as shown at item 211. It only parsed 379 tickers and took 18.4 seconds. 18 seconds appears to be an HTTP timeout. I think the 1% failures might be down to what you described as chunked encoding and HTTP 1.1. I can use HTTP 1.0 when doing a POST with the WiFiClientSecure library but I haven't found a syntax that accepts HTTP 1.0 with the HTTPClient library I am currently using.

203 BTCUSDT:3738.04000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3116   > 5s: 2 0.99%
204 BTCUSDT:3738.04000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3145   > 5s: 2 0.98%
205 BTCUSDT:3738.94000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3179   > 5s: 2 0.98%
206 BTCUSDT:3739.98000000   Tickers: 429     Free: 15224    Parsed OK   Free: 44064 Loop ms: 3329   > 5s: 2 0.97%
207 BTCUSDT:3738.73000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3205   > 5s: 2 0.97%
208 BTCUSDT:3739.95000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3410   > 5s: 2 0.96%
209 BTCUSDT:3738.57000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3254   > 5s: 2 0.96%
210 BTCUSDT:3739.00000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3463   > 5s: 2 0.95%
211 BTCUSDT:3739.02000000   Tickers: 379     Free: 15712    Parse FAILED    Free: 44064 Loop ms: 18399  > 5s: 3 1.42%
212 BTCUSDT:3746.00000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 3373   > 5s: 3 1.42%
213 BTCUSDT:3746.79000000   Tickers: 429     Free: 14960    Parsed OK   Free: 43792 Loop ms: 3204   > 5s: 3 1.41%
214 BTCUSDT:3747.27000000   Tickers: 429     Free: 15040    Parsed OK   Free: 43872 Loop ms: 3265   > 5s: 3 1.40%
215 BTCUSDT:3747.37000000   Tickers: 429     Free: 15032    Parsed OK   Free: 43872 Loop ms: 3299   > 5s: 3 1.40%
216 BTCUSDT:3750.31000000   Tickers: 429     Free: 14960    Parsed OK   Free: 43792 Loop ms: 3231   > 5s: 3 1.39%
217 BTCUSDT:3752.00000000   Tickers: 429     Free: 15040    Parsed OK   Free: 43872 Loop ms: 3104   > 5s: 3 1.38%

Actually when I check the ESP8266HTTPClient.h I see that the timeout is 5000 ms so I need to see why the 1% failures are > 18,000 ms.

In an attempt to use HTTP 1.0 in the library I have changed: #define HTTPCLIENT_1_1_COMPATIBLE 1 to: #define HTTPCLIENT_1_1_COMPATIBLE 0

Setting HTTPCLIENT_1_1_COMPATIBLE to 0 in the ESP8266HTTPClient library AND adding my own timeout in the while loop seems to have had an improvement in performance. No more 18 second total cycle times and average cycle is down 2.7 s (from 3.2s). Percentage failures of about 0.3% is misleading as the code is for cycles that take > 5000 ms and my manual timeout is stopping most of these even though it sometimes stops parsing at ticker 228 or 298. Maybe ticker 228 & 298 are sometimes malformed?

185 BTCUSDT:3986.22000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2662   > 5s: 0 0.00%
186 BTCUSDT:3986.89000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2712   > 5s: 0 0.00%
187 BTCUSDT:3985.26000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2680   > 5s: 0 0.00%
188 BTCUSDT:3986.11000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2576   > 5s: 0 0.00%
189 BTCUSDT:3986.88000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2640   > 5s: 0 0.00%
190 BTCUSDT:3985.27000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2663   > 5s: 0 0.00%
191 BTCUSDT:3985.59000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2660   > 5s: 0 0.00%
192 BTCUSDT:3985.22000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2671   > 5s: 0 0.00%
193 BTCUSDT:3985.22000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2577   > 5s: 0 0.00%
194 BTCUSDT:3985.22000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2656   > 5s: 0 0.00%
195 BTCUSDT:3985.22000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2665   > 5s: 0 0.00%
196 BTCUSDT:3985.03000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2587   > 5s: 0 0.00%
197 BTCUSDT:3984.99000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2615   > 5s: 0 0.00%
198 BTCUSDT:3983.27000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2748   > 5s: 0 0.00%
199 BTCUSDT:3983.76000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2752   > 5s: 0 0.00%
200 BTCUSDT:3983.83000000   Tickers: 429     Free: 14448    Parsed OK   Free: 43392 Loop ms: 2623   > 5s: 0 0.00%
201 BTCUSDT:3984.71000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2650   > 5s: 0 0.00%
202 BTCUSDT:3983.82000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2560   > 5s: 0 0.00%
203 BTCUSDT:3983.82000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2614   > 5s: 0 0.00%
204 BTCUSDT:3983.00000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2569   > 5s: 0 0.00%
205 BTCUSDT:3982.96000000   Tickers: 429     Free: 15096    Parsed OK   Free: 44064 Loop ms: 2608   > 5s: 0 0.00%
206 BTCUSDT:3981.00000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2646   > 5s: 0 0.00%
207 BTCUSDT:3980.00000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2851   > 5s: 0 0.00%
208 BTCUSDT:3977.45000000   Tickers: 228     Free: 15112    Parse FAILED    Free: 44064 Loop ms: 2780   > 5s: 0 0.00%
209 BTCUSDT:3978.89000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2717   > 5s: 0 0.00%
210 BTCUSDT:3977.01000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2789   > 5s: 0 0.00%
211 BTCUSDT:3974.68000000   Tickers: 429     Free: 15080    Parsed OK   Free: 44064 Loop ms: 2704   > 5s: 0 0.00%
212 BTCUSDT:3974.07000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2841   > 5s: 0 0.00%
213 BTCUSDT:3972.03000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2653   > 5s: 0 0.00%
214 BTCUSDT:3971.30000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2676   > 5s: 0 0.00%
215 BTCUSDT:3971.00000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2774   > 5s: 0 0.00%
216 BTCUSDT:3970.00000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2817   > 5s: 0 0.00%
217 BTCUSDT:3970.97000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2715   > 5s: 0 0.00%
218 BTCUSDT:3970.00000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2723   > 5s: 0 0.00%
219 BTCUSDT:3970.01000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2841   > 5s: 0 0.00%
220 BTCUSDT:3965.70000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2652   > 5s: 0 0.00%
221 BTCUSDT:3968.61000000   Tickers: 429     Free: 14432    Parsed OK   Free: 44064 Loop ms: 2777   > 5s: 0 0.00%
222 BTCUSDT:3968.21000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2683   > 5s: 0 0.00%
223 BTCUSDT:3964.74000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2758   > 5s: 0 0.00%
224 BTCUSDT:3965.11000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2902   > 5s: 0 0.00%
225 BTCUSDT:3966.65000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2729   > 5s: 0 0.00%
226 BTCUSDT:3967.76000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2779   > 5s: 0 0.00%
227 BTCUSDT:3967.81000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2829   > 5s: 0 0.00%
228 BTCUSDT:3970.08000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2824   > 5s: 0 0.00%
229 BTCUSDT:3970.01000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2952   > 5s: 0 0.00%
230 BTCUSDT:3971.70000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2677   > 5s: 0 0.00%
231 BTCUSDT:3972.06000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2577   > 5s: 0 0.00%
232 BTCUSDT:3974.14000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2570   > 5s: 0 0.00%
233 BTCUSDT:3974.12000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2655   > 5s: 0 0.00%
234 BTCUSDT:3971.53000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2811   > 5s: 0 0.00%
235 BTCUSDT:3974.02000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2672   > 5s: 0 0.00%
236 BTCUSDT:3972.22000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2892   > 5s: 0 0.00%
237 BTCUSDT:3975.21000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2675   > 5s: 0 0.00%
238 BTCUSDT:3974.42000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2586   > 5s: 0 0.00%
239 BTCUSDT:3976.01000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2619   > 5s: 0 0.00%
240 BTCUSDT:3975.18000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2743   > 5s: 0 0.00%
241 BTCUSDT:3974.55000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2706   > 5s: 0 0.00%
242 BTCUSDT:3974.72000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2704   > 5s: 0 0.00%
243 BTCUSDT:3974.75000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2602   > 5s: 0 0.00%
244 BTCUSDT:3974.62000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2708   > 5s: 0 0.00%
245 BTCUSDT:3974.65000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2637   > 5s: 0 0.00%
246 BTCUSDT:3974.65000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2618   > 5s: 0 0.00%
247 BTCUSDT:3974.56000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2722   > 5s: 0 0.00%
248 BTCUSDT:3976.23000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2593   > 5s: 0 0.00%
249 BTCUSDT:3977.40000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2740   > 5s: 0 0.00%
250 BTCUSDT:3978.38000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2725   > 5s: 0 0.00%
251 BTCUSDT:3980.17000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 3393   > 5s: 0 0.00%
252 BTCUSDT:3980.21000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2784   > 5s: 0 0.00%
253 BTCUSDT:3980.16000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2665   > 5s: 0 0.00%
254 BTCUSDT:3978.77000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2798   > 5s: 0 0.00%
255 BTCUSDT:3976.24000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2659   > 5s: 0 0.00%
256 BTCUSDT:3978.40000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2802   > 5s: 0 0.00%
257 BTCUSDT:3974.90000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2779   > 5s: 0 0.00%
258 BTCUSDT:3978.45000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2781   > 5s: 0 0.00%
259 BTCUSDT:3976.47000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2735   > 5s: 0 0.00%
260 BTCUSDT:3976.43000000   Tickers: 429     Free: 14432    Parsed OK   Free: 44064 Loop ms: 2728   > 5s: 0 0.00%
261 BTCUSDT:3976.42000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2848   > 5s: 0 0.00%
262 BTCUSDT:3976.37000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2898   > 5s: 0 0.00%
263 BTCUSDT:3977.84000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 5398   > 5s: 1 0.38%
264 BTCUSDT:3976.57000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2789   > 5s: 1 0.38%
265 BTCUSDT:3976.96000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2813   > 5s: 1 0.38%
266 BTCUSDT:3978.82000000   Tickers: 429     Free: 14416    Parsed OK   Free: 44064 Loop ms: 2735   > 5s: 1 0.38%
267 BTCUSDT:3978.24000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2723   > 5s: 1 0.37%
268 BTCUSDT:3978.24000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2799   > 5s: 1 0.37%
269 BTCUSDT:3978.12000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 3147   > 5s: 1 0.37%
270 BTCUSDT:3978.15000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2766   > 5s: 1 0.37%
271 BTCUSDT:3978.15000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2776   > 5s: 1 0.37%
272 BTCUSDT:3978.14000000   Tickers: 429     Free: 15232    Parsed OK   Free: 44064 Loop ms: 2706   > 5s: 1 0.37%
273 BTCUSDT:3979.88000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2628   > 5s: 1 0.37%
274 BTCUSDT:3978.16000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2556   > 5s: 1 0.36%
275 BTCUSDT:3978.65000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2580   > 5s: 1 0.36%
276 BTCUSDT:3979.84000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2590   > 5s: 1 0.36%
277 BTCUSDT:3979.85000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2630   > 5s: 1 0.36%
278 BTCUSDT:3979.86000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2619   > 5s: 1 0.36%
279 BTCUSDT:3981.35000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2633   > 5s: 1 0.36%
280 BTCUSDT:3982.98000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2602   > 5s: 1 0.36%
281 BTCUSDT:3982.98000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2768   > 5s: 1 0.36%
282 BTCUSDT:3982.36000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2631   > 5s: 1 0.35%
283 BTCUSDT:3982.98000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2679   > 5s: 1 0.35%
284 BTCUSDT:3983.80000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2595   > 5s: 1 0.35%
285 BTCUSDT:3984.78000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2669   > 5s: 1 0.35%
286 BTCUSDT:3979.74000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2589   > 5s: 1 0.35%
287 BTCUSDT:3979.79000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 3070   > 5s: 1 0.35%
288 BTCUSDT:3982.84000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2612   > 5s: 1 0.35%
289 BTCUSDT:3982.51000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2604   > 5s: 1 0.35%
290 BTCUSDT:3981.51000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2707   > 5s: 1 0.34%
291 BTCUSDT:3981.41000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2667   > 5s: 1 0.34%
292 BTCUSDT:3981.37000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2678   > 5s: 1 0.34%
293 BTCUSDT:3981.25000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2558   > 5s: 1 0.34%
294 BTCUSDT:3981.19000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2646   > 5s: 1 0.34%
295 BTCUSDT:3981.03000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2666   > 5s: 1 0.34%
296 BTCUSDT:3980.88000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2734   > 5s: 1 0.34%
297 BTCUSDT:3978.41000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2701   > 5s: 1 0.34%
298 BTCUSDT:3979.66000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2604   > 5s: 1 0.34%
299 BTCUSDT:3978.42000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2580   > 5s: 1 0.33%
300 BTCUSDT:3978.41000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2596   > 5s: 1 0.33%
301 BTCUSDT:3980.87000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2590   > 5s: 1 0.33%
302 BTCUSDT:3978.44000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2738   > 5s: 1 0.33%
303 BTCUSDT:3979.16000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2650   > 5s: 1 0.33%
304 BTCUSDT:3980.78000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2716   > 5s: 1 0.33%
305 BTCUSDT:3979.09000000   Tickers: 429     Free: 14336    Parsed OK   Free: 44064 Loop ms: 2703   > 5s: 1 0.33%
306 BTCUSDT:3980.21000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2572   > 5s: 1 0.33%
307 BTCUSDT:3980.24000000   Tickers: 228     Free: 15112    Parse FAILED    Free: 44064 Loop ms: 2676   > 5s: 1 0.33%
308 BTCUSDT:3981.32000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2596   > 5s: 1 0.32%
309 BTCUSDT:3980.96000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2688   > 5s: 1 0.32%
310 BTCUSDT:3982.98000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2567   > 5s: 1 0.32%
311 BTCUSDT:3984.30000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 3329   > 5s: 1 0.32%
312 BTCUSDT:3981.96000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2620   > 5s: 1 0.32%
313 BTCUSDT:3984.24000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2541   > 5s: 1 0.32%
314 BTCUSDT:3983.55000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2633   > 5s: 1 0.32%
315 BTCUSDT:3981.64000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2660   > 5s: 1 0.32%
316 BTCUSDT:3984.24000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2602   > 5s: 1 0.32%
317 BTCUSDT:3984.25000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2780   > 5s: 1 0.32%
318 BTCUSDT:3982.00000000   Tickers: 429     Free: 15112    Parsed OK   Free: 44064 Loop ms: 2602   > 5s: 1 0.31%
319 BTCUSDT:3982.01000000   Tickers: 429     Free: 14416    Parsed OK   Free: 44064 Loop ms: 2734   > 5s: 1 0.31%
320 BTCUSDT:3983.16000000   Tickers: 228     Free: 15112    Parse FAILED    Free: 44064 Loop ms: 2490   > 5s: 1 0.31%
321 BTCUSDT:3983.16000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2642   > 5s: 1 0.31%
322 BTCUSDT:3982.21000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2659   > 5s: 1 0.31%
323 BTCUSDT:3984.60000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2636   > 5s: 1 0.31%
324 BTCUSDT:3982.58000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2642   > 5s: 1 0.31%
325 BTCUSDT:3982.58000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 3426   > 5s: 1 0.31%
326 BTCUSDT:3984.48000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2806   > 5s: 1 0.31%
327 BTCUSDT:3983.79000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2606   > 5s: 1 0.31%
328 BTCUSDT:3982.58000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2579   > 5s: 1 0.30%
329 BTCUSDT:3982.59000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2560   > 5s: 1 0.30%
330 BTCUSDT:3980.15000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2671   > 5s: 1 0.30%
331 BTCUSDT:3980.01000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2635   > 5s: 1 0.30%
332 BTCUSDT:3982.15000000   Tickers: 429     Free: 15120    Parsed OK   Free: 44064 Loop ms: 2653   > 5s: 1 0.30%
333 BTCUSDT:3981.99000000   Tickers: 429     Free: 14448    Parsed OK   Free: 44064 Loop ms: 2623   > 5s: 1 0.30%

OK I let it ran for 1286 cycles and it failed 44 times (3.42% failure). A failure now is > 10,000 ms total cycle time (none) or failed to parse all the tickers.

Of the 44 failures the point of failure was at the following tickers:

 18 -  4 times
228 - 13 times
298 - 25 times
404 -  2 times

I am going to try and trap the data when the ticker is 297 to 299 to see if I can identify if 298 has random corruptions etc.

@bblanchon do you have any timeouts / throttling in your library?

When I ask the ESP8266 to print out tickers 297 to 299 the failure rate drops to 0.3% (1 failure in 787 cycles). When I ask the ESP8266 to print out just ticker 298 the failure rate drops to 0%. No fails from more than 500 cycles. I will leave it running to see when the first fail occurs.

495 BTCUSDT:3988.32000000   GRSETH:0.00231100   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2496   Fails: 0    0.00%
496 BTCUSDT:3987.00000000   GRSETH:0.00231100   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 3226   Fails: 0    0.00%
497 BTCUSDT:3990.00000000   GRSETH:0.00231100   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2587   Fails: 0    0.00%
498 BTCUSDT:3990.10000000   GRSETH:0.00231100   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2726   Fails: 0    0.00%
499 BTCUSDT:3991.58000000   GRSETH:0.00231100   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2561   Fails: 0    0.00%
500 BTCUSDT:3991.58000000   GRSETH:0.00231100   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2510   Fails: 0    0.00%
501 BTCUSDT:3995.92000000   GRSETH:0.00231100   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2605   Fails: 0    0.00%
502 BTCUSDT:3995.88000000   GRSETH:0.00231100   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2512   Fails: 0    0.00%
503 BTCUSDT:3993.33000000   GRSETH:0.00231100   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2542   Fails: 0    0.00%
504 BTCUSDT:3995.75000000   GRSETH:0.00231100   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2516   Fails: 0    0.00%
505 BTCUSDT:3995.76000000   GRSETH:0.00231100   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2535   Fails: 0    0.00%

Strangely when I add extra code in the full cycle the cycle time is dropping. I am now down to about 2.5s per cycle. Actually I did remove a delay(1) from the while loop as I noticed the ESP8266 library has it's own delay of 1ms. With over 400 tickers this will have reduced the cycle time by over 400ms.

Maybe a few lines of code in the full cycle are just enough to make your parser work at the optimum speed i.e. if it runs too fast it fails 1% of the time. Could just be internet problems. What do you think?

First fail didn't occur until cycle 899 and then it was at ticker 18, not 298. Although it was quickly followed by a fail of ticker 298 at cycle 959. As shown below with over 1000 cycles the success rate is 99.8% which ain't too shabby.

890 BTCUSDT:4005.99000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2491   Fails: 0    0.00%
891 BTCUSDT:4002.12000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2523   Fails: 0    0.00%
892 BTCUSDT:4002.12000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2549   Fails: 0    0.00%
893 BTCUSDT:4003.85000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2525   Fails: 0    0.00%
894 BTCUSDT:4003.79000000   GRSETH:0.00231677   Tickers: 429     Free: 14992    Parsed OK   Free: 44056 Loop ms: 2483   Fails: 0    0.00%
895 BTCUSDT:4002.09000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2551   Fails: 0    0.00%
896 BTCUSDT:4001.38000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2478   Fails: 0    0.00%
897 BTCUSDT:4000.01000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2558   Fails: 0    0.00%
898 BTCUSDT:4001.09000000   GRSETH:0.00231677   Tickers: 429     Free: 15224    Parsed OK   Free: 44056 Loop ms: 2522   Fails: 0    0.00%
899 BTCUSDT:3997.14000000   Tickers: 18  Free: 14784    Parse FAILED    Free: 44056 Loop ms: 2249   Fails: 1    0.11%
900 BTCUSDT:3995.83000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2516   Fails: 1    0.11%
901 BTCUSDT:3995.72000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2617   Fails: 1    0.11%
902 BTCUSDT:3995.02000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2561   Fails: 1    0.11%
903 BTCUSDT:3996.58000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2611   Fails: 1    0.11%
904 BTCUSDT:3994.36000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2534   Fails: 1    0.11%
905 BTCUSDT:3995.12000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2478   Fails: 1    0.11%
906 BTCUSDT:3994.33000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2619   Fails: 1    0.11%
907 BTCUSDT:3994.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2460   Fails: 1    0.11%
908 BTCUSDT:3995.15000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2464   Fails: 1    0.11%
909 BTCUSDT:3994.39000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2642   Fails: 1    0.11%
910 BTCUSDT:3995.15000000   GRSETH:0.00231677   Tickers: 429     Free: 14320    Parsed OK   Free: 44056 Loop ms: 2464   Fails: 1    0.11%
911 BTCUSDT:3992.49000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2515   Fails: 1    0.11%
912 BTCUSDT:3993.91000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2570   Fails: 1    0.11%
913 BTCUSDT:3996.57000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2609   Fails: 1    0.11%
914 BTCUSDT:3997.33000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2463   Fails: 1    0.11%
915 BTCUSDT:3999.01000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2645   Fails: 1    0.11%
916 BTCUSDT:3997.33000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2510   Fails: 1    0.11%
917 BTCUSDT:3999.02000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2497   Fails: 1    0.11%
918 BTCUSDT:4001.44000000   GRSETH:0.00231677   Tickers: 429     Free: 14208    Parsed OK   Free: 43832 Loop ms: 2513   Fails: 1    0.11%
919 BTCUSDT:4003.80000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2515   Fails: 1    0.11%
920 BTCUSDT:4003.77000000   GRSETH:0.00231677   Tickers: 429     Free: 14208    Parsed OK   Free: 44056 Loop ms: 2515   Fails: 1    0.11%
921 BTCUSDT:4003.75000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2469   Fails: 1    0.11%
922 BTCUSDT:4002.49000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2483   Fails: 1    0.11%
923 BTCUSDT:4001.50000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2497   Fails: 1    0.11%
924 BTCUSDT:4003.69000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2454   Fails: 1    0.11%
925 BTCUSDT:3999.01000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2575   Fails: 1    0.11%
926 BTCUSDT:4000.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2503   Fails: 1    0.11%
927 BTCUSDT:3997.33000000   GRSETH:0.00231677   Tickers: 429     Free: 14320    Parsed OK   Free: 43944 Loop ms: 2513   Fails: 1    0.11%
928 BTCUSDT:3999.72000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2537   Fails: 1    0.11%
929 BTCUSDT:3999.52000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2560   Fails: 1    0.11%
930 BTCUSDT:3998.43000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2684   Fails: 1    0.11%
931 BTCUSDT:3996.24000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2583   Fails: 1    0.11%
932 BTCUSDT:3996.27000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2537   Fails: 1    0.11%
933 BTCUSDT:3995.04000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2672   Fails: 1    0.11%
934 BTCUSDT:3995.20000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2481   Fails: 1    0.11%
935 BTCUSDT:3995.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2619   Fails: 1    0.11%
936 BTCUSDT:3996.92000000   GRSETH:0.00231677   Tickers: 429     Free: 14320    Parsed OK   Free: 44056 Loop ms: 2491   Fails: 1    0.11%
937 BTCUSDT:3997.23000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2573   Fails: 1    0.11%
938 BTCUSDT:3996.45000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2425   Fails: 1    0.11%
939 BTCUSDT:3995.73000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2559   Fails: 1    0.11%
940 BTCUSDT:3995.11000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2512   Fails: 1    0.11%
941 BTCUSDT:3995.17000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2455   Fails: 1    0.11%
942 BTCUSDT:3996.42000000   GRSETH:0.00231677   Tickers: 429     Free: 14352    Parsed OK   Free: 43976 Loop ms: 2472   Fails: 1    0.11%
943 BTCUSDT:3996.38000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2537   Fails: 1    0.11%
944 BTCUSDT:3996.01000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2513   Fails: 1    0.11%
945 BTCUSDT:3995.50000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2484   Fails: 1    0.11%
946 BTCUSDT:3996.05000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2641   Fails: 1    0.11%
947 BTCUSDT:3994.86000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2542   Fails: 1    0.11%
948 BTCUSDT:3992.13000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2441   Fails: 1    0.11%
949 BTCUSDT:3992.70000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2520   Fails: 1    0.11%
950 BTCUSDT:3990.67000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2526   Fails: 1    0.11%
951 BTCUSDT:3987.91000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2472   Fails: 1    0.11%
952 BTCUSDT:3986.30000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2467   Fails: 1    0.11%
953 BTCUSDT:3986.09000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2511   Fails: 1    0.10%
954 BTCUSDT:3981.10000000   GRSETH:0.00231677   Tickers: 429     Free: 14424    Parsed OK   Free: 44056 Loop ms: 2610   Fails: 1    0.10%
955 BTCUSDT:3984.96000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2605   Fails: 1    0.10%
956 BTCUSDT:3978.13000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2531   Fails: 1    0.10%
957 BTCUSDT:3977.98000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2600   Fails: 1    0.10%
958 BTCUSDT:3976.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2488   Fails: 1    0.10%
959 BTCUSDT:3977.95000000   Tickers: 298     Free: 15024    Parse FAILED    Free: 43976 Loop ms: 2428   Fails: 2    0.21%
960 BTCUSDT:3975.77000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2602   Fails: 2    0.21%
961 BTCUSDT:3975.76000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2572   Fails: 2    0.21%
962 BTCUSDT:3979.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2562   Fails: 2    0.21%
963 BTCUSDT:3980.98000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2863   Fails: 2    0.21%
964 BTCUSDT:3984.91000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2451   Fails: 2    0.21%
965 BTCUSDT:3984.52000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2541   Fails: 2    0.21%
966 BTCUSDT:3982.79000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2482   Fails: 2    0.21%
967 BTCUSDT:3984.60000000   GRSETH:0.00231677   Tickers: 429     Free: 14360    Parsed OK   Free: 43304 Loop ms: 2593   Fails: 2    0.21%
968 BTCUSDT:3980.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15088    Parsed OK   Free: 44056 Loop ms: 2538   Fails: 2    0.21%
969 BTCUSDT:3980.01000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2543   Fails: 2    0.21%
970 BTCUSDT:3980.61000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2545   Fails: 2    0.21%
971 BTCUSDT:3980.68000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2489   Fails: 2    0.21%
972 BTCUSDT:3980.73000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2455   Fails: 2    0.21%
973 BTCUSDT:3980.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2500   Fails: 2    0.21%
974 BTCUSDT:3977.96000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2523   Fails: 2    0.21%
975 BTCUSDT:3979.77000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2549   Fails: 2    0.21%
976 BTCUSDT:3977.31000000   GRSETH:0.00231677   Tickers: 429     Free: 15024    Parsed OK   Free: 43976 Loop ms: 2493   Fails: 2    0.20%
977 BTCUSDT:3977.03000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2512   Fails: 2    0.20%
978 BTCUSDT:3977.03000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2690   Fails: 2    0.20%
979 BTCUSDT:3977.03000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2612   Fails: 2    0.20%
980 BTCUSDT:3977.22000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2520   Fails: 2    0.20%
981 BTCUSDT:3978.19000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2468   Fails: 2    0.20%
982 BTCUSDT:3975.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2546   Fails: 2    0.20%
983 BTCUSDT:3975.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2497   Fails: 2    0.20%
984 BTCUSDT:3975.72000000   GRSETH:0.00231677   Tickers: 429     Free: 15224    Parsed OK   Free: 44056 Loop ms: 2523   Fails: 2    0.20%
985 BTCUSDT:3973.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2551   Fails: 2    0.20%
986 BTCUSDT:3973.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2507   Fails: 2    0.20%
987 BTCUSDT:3973.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2507   Fails: 2    0.20%
988 BTCUSDT:3974.55000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2531   Fails: 2    0.20%
989 BTCUSDT:3973.33000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2564   Fails: 2    0.20%
990 BTCUSDT:3972.21000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2505   Fails: 2    0.20%
991 BTCUSDT:3972.08000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2535   Fails: 2    0.20%
992 BTCUSDT:3972.02000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2518   Fails: 2    0.20%
993 BTCUSDT:3972.01000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2639   Fails: 2    0.20%
994 BTCUSDT:3971.01000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2542   Fails: 2    0.20%
995 BTCUSDT:3968.85000000   GRSETH:0.00231677   Tickers: 429     Free: 15216    Parsed OK   Free: 44056 Loop ms: 2464   Fails: 2    0.20%
996 BTCUSDT:3968.40000000   GRSETH:0.00231677   Tickers: 429     Free: 15224    Parsed OK   Free: 44056 Loop ms: 2567   Fails: 2    0.20%
997 BTCUSDT:3965.20000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2593   Fails: 2    0.20%
998 BTCUSDT:3965.19000000   GRSETH:0.00231677   Tickers: 429     Free: 15224    Parsed OK   Free: 44056 Loop ms: 2526   Fails: 2    0.20%
999 BTCUSDT:3961.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2550   Fails: 2    0.20%
1000    BTCUSDT:3958.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2520   Fails: 2    0.20%
1001    BTCUSDT:3958.00000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2522   Fails: 2    0.20%
1002    BTCUSDT:3958.00000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2659   Fails: 2    0.20%
1003    BTCUSDT:3960.54000000   GRSETH:0.00231677   Tickers: 429     Free: 14432    Parsed OK   Free: 44056 Loop ms: 2516   Fails: 2    0.20%
1004    BTCUSDT:3959.98000000   GRSETH:0.00231677   Tickers: 429     Free: 15104    Parsed OK   Free: 44056 Loop ms: 2549   Fails: 2    0.20%
1005    BTCUSDT:3965.19000000   GRSETH:0.00231677   Tickers: 429     Free: 14440    Parsed OK   Free: 44056 Loop ms: 2510   Fails: 2    0.20%

@bblanchon just to add failure rate is 50% to 100% with a static buffer. As 80 bytes is fine for a dynamic buffer (0.2% failure rate) should it be the same size for a static buffer? I know you advocate static for my project but it performs considerably better with the small dynamic buffer.

Hi @pieman64,

No, ArduinoJson doesn't contain any throttling or timeout feature. However, the WifiClient has a timeout that you can change with setTimeout().

You say that adding delays increases the success rate, so it looks like a concurrency issue, but I don't see any reason, except a bug in the ESP8266 core. I don't think that the reason, it may simply be a network error.

There is something really strange about the StaticJsonBuffer: it would work! Please check the value of JsonBuffer::size() to see if the prediction of the Assistant is true.

if (jb.size() >= 80) {
  Serial.print("JsonBuffer size = ");
  Serial.println(jb.size());
}

If you want to see what the input looks like, you can read each ticker in a temporary buffer with Stream::readBytesUntil(), instead of letting ArduinoJson pull the bytes from the stream.

Regards, Benoit

@bblanchon when I add the jb.size() check of the StaticJsonBuffer over 480 cycles it never finds more than 80 bytes. 80 should be enough with most tickers being about 74 and a few up to 78. As shown below Static fails 56% of the time. I will try your suggestion of readBytesUntil() and see if that finds anything interesting.

467 ICXUSDT:0.23920000  Tickers: 429     Free: 14408    Parsed OK   Free: 44024 Loop ms: 3601   Fails: 262  56.10%
468 ICXUSDT:0.23920000  Tickers: 429     Free: 14408    Parsed OK   Free: 44024 Loop ms: 3552   Fails: 262  55.98%
469 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3361   Fails: 263  56.08%
470 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3369   Fails: 264  56.17%
471 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3386   Fails: 265  56.26%
472 Tickers: 333     Free: 14992    Parse FAILED    Free: 43944 Loop ms: 3187   Fails: 266  56.36%
473 ICXUSDT:0.23810000  Tickers: 429     Free: 14408    Parsed OK   Free: 44024 Loop ms: 3656   Fails: 266  56.24%
474 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3333   Fails: 267  56.33%
475 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3435   Fails: 268  56.42%
476 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3332   Fails: 269  56.51%
477 ICXUSDT:0.23810000  Tickers: 429     Free: 14336    Parsed OK   Free: 43944 Loop ms: 4045   Fails: 269  56.39%
478 Tickers: 333     Free: 15072    Parse FAILED    Free: 44024 Loop ms: 3366   Fails: 270  56.49%

@bblanchon readBytesUntil() didn't identify anything and on reflection I am not surprised as the API call is provided by a multi billion dollar company that have an enormous number of trading bots using the data.

With Dynamic buffer the log below now shows 100% parsing success for over 1700 cycles!!!

1695    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2753   Fails: 0    0.00%
1696    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2606   Fails: 0    0.00%
1697    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2826   Fails: 0    0.00%
1698    Tickers: 429     Free: 14408    Parsed OK   Free: 44024 Loop ms: 2748   Fails: 0    0.00%
1699    Tickers: 429     Free: 14376    Parsed OK   Free: 44024 Loop ms: 2753   Fails: 0    0.00%
1700    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2859   Fails: 0    0.00%
1701    Tickers: 429     Free: 13840    Parsed OK   Free: 42680 Loop ms: 2846   Fails: 0    0.00%
1702    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2690   Fails: 0    0.00%
1703    Tickers: 429     Free: 13840    Parsed OK   Free: 42680 Loop ms: 2686   Fails: 0    0.00%
1704    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2754   Fails: 0    0.00%
1705    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2723   Fails: 0    0.00%

Are you familiar with Arduino's SimpleTimer library? I have a couple of timers in my project and they use modified version of the SimpleTimer library. Effectively the timers should be disabled most of the time but when they are enabled they trigger at 100ms and 1300ms intervals. I think the 1300ms timer might have been kicking in during the parsing and causing the earlier problems. I have now totally disabled the timers and all looks fine.

I will switch to Static buffer now and see how that performs with the timers disabled. From your understanding of the way static and dynamic buffers work do you know of any reason why a timer "interrupt" would affect static more than a dynamic buffer?

Over 1000 cycles with a Static buffer and 100% parsing success. Cycle time seems a bit less with Static buffer too. Just going to enable the timer again to be sure that was the problem.

1046    Tickers: 429     Free: 14408    Parsed OK   Free: 44024 Loop ms: 2515   Fails: 0    0.00%
1047    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2656   Fails: 0    0.00%
1048    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2544   Fails: 0    0.00%
1049    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2671   Fails: 0    0.00%
1050    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2566   Fails: 0    0.00%
1051    Tickers: 429     Free: 13840    Parsed OK   Free: 43352 Loop ms: 2659   Fails: 0    0.00%

I let the code run overnight and it has now made over 10,000 API calls with 100% success rate (and Static Json Buffer).

10087   Tickers: 429     Free: 14960    Parsed OK   Free: 43792 Loop ms: 2914   Fails: 0    0.00%
10088   Tickers: 429     Free: 14176    Parsed OK   Free: 43792 Loop ms: 2953   Fails: 0    0.00%
10089   Tickers: 429     Free: 14176    Parsed OK   Free: 43792 Loop ms: 2836   Fails: 0    0.00%
10090   Tickers: 429     Free: 14160    Parsed OK   Free: 43792 Loop ms: 2841   Fails: 0    0.00%
10091   Tickers: 429     Free: 13608    Parsed OK   Free: 43120 Loop ms: 2834   Fails: 0    0.00%
10092   Tickers: 429     Free: 14176    Parsed OK   Free: 43792 Loop ms: 2926   Fails: 0    0.00%

However this is with the timers enabled so that was not the problem.

I think the problem was that within some of the parsing functions I was actually sending logging data to Serial Monitor.

I don't believe the timer library could interfere with ArduinoJson. However, it could interfere with the Wifi stack.

bblanchon / ArduinoJson

BIG array problem with version 5 #866