squix78 / json-streaming-parser

Arduino library for parsing potentially huge json streams on devices with scarce memory
MIT License
205 stars 88 forks source link

ESP8266 Community board upgrade to >2.5.0 breaks the parser #32

Closed essenemari closed 1 year ago

essenemari commented 1 year ago

Just heads up. At least for me, this cool parser stops work well once ESP8266 board is upgraded from 2.3.0 to 2.5 or higher. I am using this to parse Home Assistant data. Once board is upgraded, parser does return mismatch values. On Esp8266 2.3.0 still works well :) 👍

mrcodetastic commented 1 year ago

Try with:

https://github.com/mrfaptastic/json-streaming-parser2

Because this library isn't maintained anymore it seems .

khairullaizatt commented 1 year ago

how do i parse from this

https://data-live.flightradar24.com/zones/fcgi/feed.js?faa=1&bounds=3.462%2C2.438%2C101.156%2C102.504&satellite=1&mlat=1&flarm=1&adsb=1&gnd=1&air=1&vehicles=1&estimated=1&maxage=14400&gliders=1&stats=1&enc=aUZxeUj-cudNzTMJ_LNTMhchOm2J_TlYbC5HPIDKD_0

if any vidoe toturial please give link.

mrcodetastic commented 1 year ago

I suggest using the ArduinoJson library. Much easier and lots of documentation.

Try using ChatGPT to get example code.

essenemari commented 1 year ago

Try with:

https://github.com/mrfaptastic/json-streaming-parser2

Because this library isn't maintained anymore it seems .

So, cool! Many thanks! I will start my testing soon. Looks like your parser is based on squix78 which means easier jump to yours parser.

mrcodetastic commented 1 year ago

So, cool! Many thanks! I will start my testing soon. Looks like your parser is based on squix78 which means easier jump to yours parser.

Yep. If you're already using squix78's then you should be able to migrate across to parser2 with minimal effort.

essenemari commented 1 year ago

Yep. If you're already using squix78's then you should be able to migrate across to parser2 with minimal effort.

Indeed, it was quite simple.

Many thanks for your work on parser2 (anto to squix78 for parser1)!

I am parsing data from Home Assistant over parser2. Now testing and all good.

Actually, thanks to opportunity on working on implementing parser2, I noticed that issue I initially hit with parser1 /squix78/ (subject of this thread) also occurs on parser2, so may be caused not by parsers but something different than parsers or ESP board updates.

I think issue was around the way how data was pulled from server, means networking issues.

For parser1, for each entity I pulled data from server by following way. Parser2, time to time, returned wrong data. Same as parser1.

    while (client.connected()) {
      while ((size = client.available()) > 0) {
        c = client.read();
        //Serial.println(c);
        if (c == '{' || c == '[') {
          isBody = true;
        }
        if (isBody) {
          parser.parse(c);
        }

Probably in a way above, in case of weak WiFi signal, reading source XML byte by byte could simply miss some characters. Then, if missed was "{", parser probably could take values from other entities or it took something from "cache". As a results, let's say state of switch1 had state of switch2. Or state has parsed value from temp. sensor which couldn't happen in expectations. It happen not so often so my systems worked well enough to live with it (always trying to have good wifi coverage).

The way you proposed in parser2 is to pull data is based on http library, so probably over TCP: (http.writeToStream(&parser)); Looks like this way, ensures pulling "confirmed" XML from server over http or nothing will be parsed. Thanks to that, parsed data is 100% accurate or there is no data at all, which is great approach.

I confirmed this in initial testing - 23k queries and 100% accurate!

Many thanks for your help here! :)