adafruit / Adafruit_CircuitPython_Requests

Requests-like interface for web interfacing
MIT License
51 stars 36 forks source link

raise error indicating gzipped data #100

Closed FoamyGuy closed 2 years ago

FoamyGuy commented 2 years ago

This change presents a short term improvement of the error messages that arise from fetching GZipped data from an API. The message has been changed to state the exact cause of the problem and let the user know the way to access the raw data.

Tested successfully on Feather TFT ESP32-S2 and CPython on Linux using this code (with slightly modified initialization on the Feather for the circuitpython way of setup):

https = requests.Session(socket, ssl.create_default_context())

JSON_GET_URL = "https://analytics.usa.gov/data/live/realtime.json"

print("Fetching JSON data from %s" % JSON_GET_URL)
response = https.get(JSON_GET_URL)
print("-" * 40)

print("JSON Response: ", response.text)
print("-" * 40)

# print("JSON Response: ", response.json())
# print("-" * 40)

current version output (on CPython, it's very similar in CircuitPython but the error type is more general ValueError):

Fetching JSON data from https://analytics.usa.gov/data/live/realtime.json
----------------------------------------
Traceback (most recent call last):
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/examples/gov_requests_https_cpython.py", line 19, in <module>
    print("JSON Response: ", response.text)
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/adafruit_requests.py", line 437, in text
    self._cached = str(self.content, self.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

new output:

Fetching JSON data from https://analytics.usa.gov/data/live/realtime.json
----------------------------------------
Traceback (most recent call last):
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/examples/gov_requests_https_cpython.py", line 19, in <module>
    print("JSON Response: ", response.text)
  File "/home/timc/repos/circuitpython/Adafruit_CircuitPython_Requests/adafruit_requests.py", line 439, in text
    raise ValueError(
ValueError: Content-encoding is gzip, data cannot be accessed as json or text. Use content property to access raw bytes.

I also verified that catching the exception does allow user code to successfully access the raw data:

try:
    print("JSON Response: ", response.json())
    print("-" * 40)
except ValueError:
    stream = io.BytesIO(response.content)
    field = pyflate.RBitfield(stream)
    magic = field.readbits(16)
    if magic == 0x1f8b: # GZip
        out = pyflate.gzip_main(field)

    json_data = json.loads(out)
    print(json_data)
    print("Active Visitors {}".format(json_data["data"][0]["active_visitors"]))