It looks very much like finish() is not working

berkut0 commented 5 months ago

I have this code on the ESP32-C3 board (CP 9.1-beta)

async def get_data():
    pool = socketpool.SocketPool(wifi.radio)
    session = requests.Session(pool, ssl.create_default_context())
    while True:
        SCORE_URL = "https://api.coincap.io/v2/assets"
        resp = session.get(SCORE_URL)
        json_data = json_stream.load(resp.iter_content(99))
        for c in json_data['data']:
            print(c.finish())
        resp.close()
        await asyncio.sleep(60)

I expect to output the data contained in json_data['data']. Generally it's a list of "coins". But I get:

None
None
None
...

It looks like you can't "sparse data" via stream_json. I came to this after several attempts using filter() and map(), and in a normal console I get normal results (but without json_stream), but using json_stream on a microcontroller I get errors. Explicit fetching via keys works.

In general, I suspect that TransientObject is not behaving correctly.

berkut0 commented 5 months ago

This code gives an error:

coins = ['BTC']
coin_data = {}

async def get_data():
    pool = socketpool.SocketPool(wifi.radio)
    session = requests.Session(pool, ssl.create_default_context())
    while True:
        SCORE_URL = "https://api.coincap.io/v2/assets"
        resp = session.get(SCORE_URL)
        json_data = json_stream.load(resp.iter_content(99))
        for coin in json_data['data']:
            if coin['symbol'] in coins:
                coin_data[coin['symbol']]=coin.finish()
        print(coin_data)    
        resp.close()
        await asyncio.sleep(60)

code.py output:
Traceback (most recent call last):
  File "code.py", line 79, in <module>
  File "asyncio/core.py", line 317, in run
  File "asyncio/core.py", line 276, in run_until_complete
  File "asyncio/core.py", line 261, in run_until_complete
  File "code.py", line 77, in main
  File "asyncio/funcs.py", line 162, in gather
  File "asyncio/core.py", line 261, in run_until_complete
  File "code.py", line 57, in get_data
  File "adafruit_json_stream.py", line 164, in __getitem__
KeyError: 

Code done running.

update: Most likely there is a problem getting the "id" key. Is it my fault or is it a bug?

tannewt commented 5 months ago

Can you post a sample of the JSON? Your requests may not match the order the data comes in.

berkut0 commented 5 months ago

You can always look at this Json via this link, it's an open API.

full example of the json data

berkut0 commented 5 months ago

I did the following:

SCORE_URL = "https://api.coincap.io/v2/assets"
        resp = session.get(SCORE_URL)
        json_data = json_stream.load(resp.iter_content(99))
        for coin in json_data['data']:
            symbol = coin['symbol']
            if symbol in coins:
                keys=['priceUsd', 'changePercent24Hr', 'vwap24Hr']
                coin_data[symbol]= {key: coin[key] for key in keys}
        print(coin_data)    
        resp.close()

It seems to be the only way to get the job done. Finish() doesn't work, accessing a key twice seems to "consume" that value without the possibility to access it again.

tannewt commented 5 months ago

Finish() doesn't work, accessing a key twice seems to "consume" that value without the possibility to access it again.

finish() isn't meant to return the object. It just consumes all of the object's characters and implicitly returns None.

All accesses will consume the stream and you cannot go backwards. That's the only way you can load without taking a bunch of memory.

Seems like finish() is working as intended but not as you expected.

adafruit / Adafruit_CircuitPython_JSON_Stream

It looks very much like finish() is not working #4