Large JSON files

[fellinga]

Hi,

I started using your library a few days ago and I don't completely understand how to use your library with large files.

My ESP8266 project (with Arduino IDE) should add/remove objects from/to a json file (sd card). I am currently using a static buffer with a size of 2000. I am able to remove and add cocktails as long as the total number of cocktails is smaller than ~8.

This is the (small) data file: {"cocktails":[{"name":"Mai Tai","category":"Rum-Cocktail","ingredients":[{"name":"Ananassaft","cl":5},{"name":"Rum","cl":6},{"name":"Zitronensaft","cl":3}]},{"name":"Screwdriver","category":"Klassiker","ingredients":[{"name":"Orangensaft","cl":10},{"name":"Wodka","cl":4}]},{"name":"Dry Martini","category":"Klassiker","ingredients":[{"name":"Gin","cl":6},{"name":"Wermut","cl":3}]}]}

This is the remove cocktail method (the add method is similar):

bool removeCocktail(String cocktailName) {
  String json = readData(cocktailDataPath); // read from sd card
  StaticJsonBuffer<2000> jsonBuffer;
  JsonObject& root = jsonBuffer.parseObject(json);

  if (root.success()) {
    JsonArray& cocktails = root["cocktails"];
    for (int i=0; i<cocktails.size();i++) {
      JsonObject& aObject = cocktails[i];
      String objectName = aObject["name"];
      if (objectName.equals(cocktailName)) {
        cocktails.remove(i);
        json = "";
        root.printTo(json);
        storeData(json, cocktailDataPath);
        return true;
      }
    }
  }
  return false;
}

If I add more cocktails parsing fails. And increasing the buffer size causes my device to reboot... I want to store about 40 to 50 cocktails in the list - is this somehow possible?

Thanks!

[devyte]

Memory on the ESP is, of course, limited. Some tips: -Create a json by hand with the full form that you want, then check it in the online assistant. -you are reading the file into a String on the stack, then parsing the string into a json object. This duplicates the contents in memory. Instead, open the file and pass the File directly to the parser (stream it). -your jsonbuffer is on the stack. There is a max stack size limit, which I believe is around 5K. Use dynamicbuffer instead, or create the staticbuffer dynamically with new (preferably straight into a std::unique_ptr). -Instead of having a single json with an array of elements, consider splitting into several json files, where each file contains an element of that array. Don't forget there are limitations on the SPIFFS filenames, though.

If non of the above help you, you may need to switch to a streaming json lib instead of this one.

On Jul 26, 2017 9:48 AM, "fellinga" notifications@github.com wrote:

Hi,

I started using your library a few days ago and I don't completely understand how to use your library with large files.

My ESP8266 project (with Arduino IDE) should add or remove objects into a json file (sd card). I am currently using a static buffer with a size of

2000. I am able to remove and add cocktails as long as the total number of cocktails is smaller than ~8.

This is the (small) data file: {"cocktails":[{"name":"Mai Tai","category":"Rum-Cocktail" ,"ingredients":[{"name":"Ananassaft","cl":5},{"name":" Rum","cl":6},{"name":"Zitronensaft","cl":3}]},{" name":"Screwdriver","category":"Klassiker","ingredients":[{" name":"Orangensaft","cl":10},{"name":"Wodka","cl":4}]},{"name":"Dry Martini","category":"Klassiker","ingredients":[{" name":"Gin","cl":6},{"name":"Wermut","cl":3}]}]}

This is the remove cocktail method (the add method is similar): `bool removeCocktail(String cocktailName) { String json = readData(cocktailDataPath); // read from sd card StaticJsonBuffer<2000> jsonBuffer; JsonObject& root = jsonBuffer.parseObject(json);

if (root.success()) { JsonArray& cocktails = root["cocktails"]; for (int i=0; i<cocktails.size();i++) { JsonObject& aObject = cocktails[i]; String objectName = aObject["name"]; if (objectName.equals(cocktailName)) { cocktails.remove(i); json = ""; root.printTo(json); storeData(json, cocktailDataPath); return true; } } } return false; }`

If I add more cocktails parsing fails. And increasing the buffer size causes my device to reboot... I want to store about 40 to 50 cocktails in the list - is this somehow possible?

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bblanchon/ArduinoJson/issues/562, or mute the thread https://github.com/notifications/unsubscribe-auth/AQC6Brd2bJ8KCI6UMEWp2q3MFh2bNz0Lks5sR0PCgaJpZM4Oj8p5 .

[bblanchon]

Hi @fellinga,

The ESP8266 stack is limited to 4KB, so you should use a DynamicJsonBuffer instead of a StaticJsonBuffer, otherwise the program crashes. As @devyte stated, use the ArduinoJson Assistant to compute the required size (which you must pass to the DynamicJsonBuffer constructor), and try to read directly from the Stream or File.

See also:

• F.A.Q: What are the differences between StaticJsonBuffer and DynamicJsonBuffer?
• F.A.Q: How to determine the buffer size?
• F.A.Q: Why does my device crash or reboot?
• F.A.Q: Can I parse a JSON input that is too big to fit in memory?
• F.A.Q: How to reduce memory usage?
• F.A.Q: Can I parse data from a stream?

Regards, Benoit

[ewaldc]

I have successfully parsed JSON strings over 20K on ESP8266/12E. If the string has deep JSON nesting think about increasing nestingLimit. Took me a half day to figure this out, plus wasted Benoit's time as I logged an issue for it (fortunately some good came out of it...) One more thing to watch out for when using large JSON string's on e.g. ESP8266: make sure you allocate the printBuffer on the heap and not the stack. So not

size_t len = obj.measureLength() + 1;
char printBuffer[len];
obj.printTo(printBuffer, len);

which allocates the local variable printBuffer on the stack which if bigger than 4K generates an exception. Instead use either of

String printBuffer;
char *printBuffer = (char *) malloc(len);

Ewald

[fellinga]

Hi,

thanks all. I just changed the StaticJsonBuffer to DynamicJsonBuffer(2000) and it is now possible to store at least 50 objects. When I started using your library I read your documentation and I am pretty sure I that I read that if someone uses an ESP he should also use a StaticJsonBuffer, which was the reason I didn't even try the dynamic one.

I am now going to adopt the code with all the other recommendations.

Thanks again!

[bblanchon]

Hi @fellinga,

Thanks for the confirmation :+1:

To get the best performance, you should replace 2000 by the value given by the Assistant.

There are probably many mistakes in the documentation :bug: Do you think you can find the page which got you in the wrong direction?

Regards, Benoit

[fellinga]

Hi @bblanchon,

I tried to find the site but I didn't.. sorry :/

I also wanted to say that I changed:

String json = readData(cocktailDataPath);
StaticJsonBuffer<2000> jsonBuffer;
JsonObject& root = jsonBuffer.parseObject(json);

to

DynamicJsonBuffer jsonBuffer(bufSize);  
File myFile = SD.open(cocktailDataPath);
JsonObject& root = jsonBuffer.parseObject(myFile);

and it really works well!

Do you also plan to support something like: root.printTo(myFile);

[bblanchon]

root.printTo(myFile);

This should already work.

[DedeHai]

First off, thanks for the gread library! I am working on an RFID access controller that reads the user database from a local server. Currently the database contains about 150 entries. So this is a similar issue: not the whole thing can be parsed at once. But then I don't really need to, I could parse the array piece by piece. The stream from the server is structured like this: [{"tid":1,"uid":"0856486896","owner":"Some User","start":"2017-05-31T22:00:00.000Z","end":"2017-12-31T23:00:00.000Z"}, ... ,{"tid":2,"uid":"0079830412","owner":"Admin","start":"2017-06-19T22:00:00.000Z","end":"2017-12-31T23:00:00.000Z"}] Is it currently possible to read this stream piece-by-piece and parse it? if not: how would I do it? I am doing this on an ESP8266 btw. and I am not quite sure how it even handles a large stream from an api meaning how it splits it into dataframes and builds the stream.

[bblanchon]

@DedeHai, please see Can I parse a JSON input that is too big to fit in memory?

[DedeHai]

That's what I suspected from what I understand how it works (did not check the full source code though). Since I do get a an array in the form stated in the post above (i.e. known structure) I just manually divide the sting into the individual JSON objects and parse each string. Works perfectly.

[mauriciojost]

Hi,

thanks all. I just changed the StaticJsonBuffer to DynamicJsonBuffer(2000) and it is now possible to store at least 50 objects. When I started using your library I read your documentation and I am pretty sure I that I read that if someone uses an ESP he should also use a StaticJsonBuffer, which was the reason I didn't even try the dynamic one.

I am now going to adopt the code with all the other recommendations.

Thanks again!

For the reference, I am pretty sure that @fellinga is talking about memory fragmentation when malloc and free are heavily used: https://github.com/esp8266/Arduino/issues/3597

Mauricio