Deceiving exception handling messages when available memory is low (below ~41k)

[jake1164]

I have run into issues with using the Requests library on a Pico W while getting JSON responses from an https url. The exception messages will vary based on the amount of free memory. I have created a quick sample to test and show the behavior.

The sample just loads images into a list and prints out the JSON returned until the board runs low on memory.

Below ~41k the exceptions make it look like a certification error and has the message 'Sending request failed' Below ~27k it just fails and gives no exception message at all.

The same result happens with requests.get(url) and requests.get(url, stream=True).

Crudely tested on: CircuitPython 8.2.0, 8.1.0, 8.0.5, 8.0.2 Requests 1.14.1, 1.13.4, 1.12.9

before 13488
appended images 44160
appended image 44000
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 44000
free memory after 6224
{'correlationId': '606fcf6', 'status': 400, 'instance': 'https://api.weather.gov/requests/606fcf6', 'detail': 'Bad Request', 'type': 'https://api.weather.gov/problems/BadRequest', 'title': 'Bad Request', 'parameterErrors': [{'parameter': 'query.area[0]', 'message': 'Does not have a value in the enumeration ["AM","AN","GM","LC","LE","LH","LM","LO","LS","PH","PK","PM","PS","PZ","SL"]'}, {'parameter': 'query.area[0]', 'message': 'Does not have a value in the enumeration ["AL","AK","AS","AR","AZ","CA","CO","CT","DE","DC","FL","GA","GU","HI","ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI","MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC","ND","OH","OK","OR","PA","PR","RI","SC","SD","TN","TX","UT","VT","VI","VA","WA","WV","WI","WY","MP","PW","FM","MH"]'}, {'parameter': 'query.area[0]', 'message': 'Failed to match exactly one schema'}]}
before 11184
appended images 41856
appended image 41696
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 41696
x509-crt-bundle:PK verify failed with error FFFFBD70
x509-crt-bundle:Failed to verify certificate
requests.get(url): Sending request failed
{}
...
before 32640
appended images 30336
appended image 30176
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 30176
requests.get(url): Sending request failed
{}
before 30336
appended images 28032
appended image 27872
Getting json
getting url: https://api.weather.gov/alerts/active?area={state}
free memory 27872
requests.get(url): 
{}

Any secrets to getting this to work on low memory situations (like a pico driving a Matrix LED)? Can we get better exception messages to indicate what is happening?

[jake1164]

Is 41k really the minimum memory requirement for SSL to work?

[tannewt]

SSL does take a good amount of memory and network requests generally do. You may be able to save memory with the json_stream library: https://github.com/adafruit/Adafruit_CircuitPython_JSON_Stream/ It'll pick values out in a stream rather than creating a bunch of objects holding all of the json data.

[jake1164]

SSL does take a good amount of memory and network requests generally do. You may be able to save memory with the json_stream library: https://github.com/adafruit/Adafruit_CircuitPython_JSON_Stream/ It'll pick values out in a stream rather than creating a bunch of objects holding all of the json data.

This is interesting, but does not really address the problem since the exception is thrown at the requests.get(url) which is the equivilant of your resp = session.get(SCORE_URL) which is before the JSON is read into memory.

On a side note, your library crashes at resp.close() with a ValueError: invalid syntax for integer with base 16 exception thrown from the requests library. Specifically at line 339 of requests.close()

                    chunk_header = bytes(self._readto(b"\r\n")).split(b";", 1)[0]