espidf memory error with 9.0 alpha.5

[jerryneedell]

As requested, here is a report of a memory error with CP 9.0 the works under 8

The code runs OK with CP 8.2.8 but gives the following error with 9.0 Alpha.5

Adafruit CircuitPython 9.0.0-alpha.5 on 2023-11-15; Adafruit QT Py ESP32S2 with ESP32S2
>>> 
>>> 
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
ip 10.0.0.53
Traceback (most recent call last):
  File "adafruit_requests.py", line 515, in _get_socket
espidf.MemoryError: 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "code.py", line 31, in <module>
  File "adafruit_io/adafruit_io.py", line 758, in create_and_get_feed
  File "adafruit_io/adafruit_io.py", line 722, in get_feed
  File "adafruit_io/adafruit_io.py", line 565, in _get
  File "adafruit_requests.py", line 711, in get
  File "adafruit_requests.py", line 650, in request
  File "adafruit_requests.py", line 496, in _get_socket
RuntimeError: Sending request failed

Code done running.

Here is the code

import alarm
import ipaddress
import wifi
import socketpool
import time
import adafruit_requests
import ssl
import espidf
from adafruit_io.adafruit_io import IO_HTTP, AdafruitIO_RequestError
import adafruit_ahtx0
import board
import busio
from secrets import secrets
import microcontroller

print("ip", wifi.radio.ipv4_address)

pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

# Set your Adafruit IO Username and Key in secrets.py
# (visit io.adafruit.com if you need to create an account,
# or if you need your Adafruit IO key.)
aio_username = secrets["aio_username"]
aio_key = secrets["aio_key"]

# Initialize an Adafruit IO HTTP API object
io = IO_HTTP(aio_username, aio_key, requests)

# Get the 'temperature' feed from Adafruit IO
temperature_feed = io.create_and_get_feed("qtpys2-aht20-temperature")
humidity_feed = io.create_and_get_feed("qtpys2-aht20-humidity")

i2c = busio.I2C(board.SCL1,board.SDA1)

# Create sensor object, communicating over the board's default I2C bus

sensor = adafruit_ahtx0.AHTx0(i2c)

while True:
    try:
        temperature = sensor.temperature
        temperature = temperature * (9./5.) + 32. # convert to F
        humidity = sensor.relative_humidity
        # set temperature value to two precision points
        temperature = "%0.2f" % (temperature)
        humidity = "%0.2f" % (humidity)

        print("Current Temperature: {0}*F".format(temperature))
        print("Sending to Adafruit IO...")
        io.send_data(temperature_feed["key"], temperature)
        print("Current Humidity: {0}%".format(humidity))
        print("Sending to Adafruit IO...")
        io.send_data(humidity_feed["key"], humidity)

        time.sleep(300)
    except KeyboardInterrupt:
        print("Keyboard Interrupt")
    except Exception as e:
        print("Exception occured: reset in 5 seconds")
        time.sleep(5)
        microcontroller.reset()

[anecdata]

I'm not sure if there's an issue already for spurious espidf.MemoryError:. I get these on Requests sock.connect somewhat frequently on 8.2.8, when espidf memory is plentiful, even early after a reset. Several resets later, it will just automatically work. I can only guess there's some non-deterministic espidf activities that trigger this (early after startup, and after running for some time). May not be related to espidf memory at all. May be unrelated to this particular issue.

Your could print out espidf memory right before this happens:

import espidf
print(f"{espidf.get_total_psram()=}")
print(f"{espidf.get_reserved_psram()=}")
print(f"{espidf.heap_caps_get_total_size()=}")
print(f"{espidf.heap_caps_get_free_size()=}")
print(f"{espidf.heap_caps_get_largest_free_block()=}")

edit: I had a 6.x issue: https://github.com/adafruit/circuitpython/issues/3562 But closed it since the codebase for it was so old.

[jerryneedell]

Here is what I get:

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
ip 10.0.0.53
espidf.get_total_psram()=2097152
espidf.heap_caps_get_total_size()=2272023
espidf.heap_caps_get_free_size()=1981999
espidf.heap_caps_get_largest_free_block()=1900544
Traceback (most recent call last):
  File "adafruit_requests.py", line 515, in _get_socket
espidf.MemoryError: 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "code.py", line 36, in <module>
  File "adafruit_io/adafruit_io.py", line 758, in create_and_get_feed
  File "adafruit_io/adafruit_io.py", line 722, in get_feed
  File "adafruit_io/adafruit_io.py", line 565, in _get
  File "adafruit_requests.py", line 711, in get
  File "adafruit_requests.py", line 650, in request
  File "adafruit_requests.py", line 496, in _get_socket
RuntimeError: Sending request failed

Code done running.

Note, I had to remove one line due to AttributeError: 'module' object has no attribute 'get_reserved_psram'

I placed the test code directly before the line crating the error:

import alarm
import ipaddress
import wifi
import socketpool
import time
import adafruit_requests
import ssl
import espidf
from adafruit_io.adafruit_io import IO_HTTP, AdafruitIO_RequestError
import adafruit_ahtx0
import board
import busio
from secrets import secrets
import microcontroller

print("ip", wifi.radio.ipv4_address)

pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

# Set your Adafruit IO Username and Key in secrets.py
# (visit io.adafruit.com if you need to create an account,
# or if you need your Adafruit IO key.)
aio_username = secrets["aio_username"]
aio_key = secrets["aio_key"]

# Initialize an Adafruit IO HTTP API object
io = IO_HTTP(aio_username, aio_key, requests)

print(f"{espidf.get_total_psram()=}")
#print(f"{espidf.get_reserved_psram()=}")
print(f"{espidf.heap_caps_get_total_size()=}")
print(f"{espidf.heap_caps_get_free_size()=}")
print(f"{espidf.heap_caps_get_largest_free_block()=}")
# Get the 'temperature' feed from Adafruit IO
temperature_feed = io.create_and_get_feed("qtpys2-aht20-temperature")

FYI with 8.2.8 I get

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
ip 10.0.0.53
espidf.get_total_psram()=2097152
espidf.heap_caps_get_total_size()=144216
espidf.heap_caps_get_free_size()=53428
espidf.heap_caps_get_largest_free_block()=34816
Current Temperature: 70.72*F
Sending to Adafruit IO...
Current Humidity: 39.74%
Sending to Adafruit IO...

[FoamyGuy]

I am seeing these same errors consistently on a FunHouse with 9.0.0 alpha 5 (and each alpha all the way back to 2, also with the absolute newest UF2 at the top of the list at the time of this writing 1f5e692.uf2):

Traceback (most recent call last):
  File "adafruit_requests.py", line 515, in _get_socket
espidf.MemoryError: 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "code.py", line 36, in <module>
  File "adafruit_io/adafruit_io.py", line 758, in create_and_get_feed
  File "adafruit_io/adafruit_io.py", line 722, in get_feed
  File "adafruit_io/adafruit_io.py", line 565, in _get
  File "adafruit_requests.py", line 711, in get
  File "adafruit_requests.py", line 650, in request
  File "adafruit_requests.py", line 496, in _get_socket
RuntimeError: Sending request failed

Using the https circuitpython simpletest from the requests library: https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/main/examples/requests_https_circuitpython.py

The same example code runs and compeletes successfully with 8.2.8

[FoamyGuy]

I te-tested this today on a QTPy 9.0.0.alpha.6 with the above linked simpletest script from requests. I still get the same results as with alpha.5, consistently have the MemoryError when it runs. I also tested 8.2.9 and confirmed that it consistently succeeds.

[tannewt]

Looks like this is due to the limited internal memory on the S2. With the new split heap we allocated to internal memory for small heap splits and then PSRAM after. The old code only used SPIRAM for CP. The fix is to allocate to SPIRAM first before trying internal memory.

[veloyage]

I am still getting an identical error when trying a basic request (getting time from adafruit IO) on 9.0 beta.0 on an ESP32-S2. Same code worked before on 8.2.9, still works on an S3 right now.

I think this is the same issue, so seems like it's not fixed.

code.py output: Traceback (most recent call last): File "adafruit_requests.py", line 515, in _get_socket espidf.MemoryError:

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "code.py", line 132, in File "adafruit_requests.py", line 732, in get File "adafruit_requests.py", line 671, in request File "adafruit_requests.py", line 496, in _get_socket RuntimeError: Sending request failed

Code done running.

[tannewt]

I am still getting an identical error when trying a basic request (getting time from adafruit IO) on 9.0 beta.0 on an ESP32-S2. Same code worked before on 8.2.9, still works on an S3 right now.

I think this is the same issue, so seems like it's not fixed.

code.py output: Traceback (most recent call last): File "adafruit_requests.py", line 515, in _get_socket espidf.MemoryError:

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "code.py", line 132, in File "adafruit_requests.py", line 732, in get File "adafruit_requests.py", line 671, in request File "adafruit_requests.py", line 496, in _get_socket RuntimeError: Sending request failed

Code done running.

What board are you using? Does it have external PSRAM?

[veloyage]

Its a Feather S2 reverse, article 5345, 2 MB PSRAM. I could test with the same thing in S3, if you think that may be relevant.

The espidf commands immediately before the requests.get() read:

code.py output: espidf.get_total_psram()=2097152 espidf.heap_caps_get_total_size()=2271279 espidf.heap_caps_get_free_size()=1975079 espidf.heap_caps_get_largest_free_block()=1867776 Traceback (most recent call last): File "adafruit_requests.py", line 515, in _get_socket espidf.MemoryError:

[tannewt]

Please post your code and I'll reopen this to take another look.

[veloyage]

Here is a reduced example which triggers the error:

import os import espidf import ssl import wifi import socketpool import adafruit_requests

pool = socketpool.SocketPool(wifi.radio) requests = adafruit_requests.Session(pool, ssl.create_default_context())

aio_username = os.getenv("ADAFRUIT_IO_USERNAME") aio_key = os.getenv("ADAFRUIT_IO_KEY")

location = "Europe/Zurich" TIME_URL = "https://io.adafruit.com/api/v2/%s/integrations/time/struct?x-aio-key=%s&tz=%s" % (aio_username, aio_key, location)

print(f"{espidf.get_total_psram()=}") print(f"{espidf.heap_caps_get_total_size()=}") print(f"{espidf.heap_caps_get_free_size()=}") print(f"{espidf.heap_caps_get_largest_free_block()=}")

response = requests.get(TIME_URL)

[veloyage]

Just tested on an ESP32-S3 Reverse TFT Feather with the same flash/psram setup, and it works fine. So, S2, specific? Because of the somewhat smaller SRAM?

[veloyage]

Went and also ran it with alpha5 on the S3, also works fine.

[tannewt]

Ya, S2 has less internal SRAM. Trying to replicate it now on a plain ESP32S2 feather with the same flash and ram. I don't have one with a display.

[tannewt]

Thanks for the code @veloyage. I've reproduced it here. It looks like the S2 has more free internal memory with 9.0 but still runs out when setting up mbedtls. I'll keep poking at this. Thanks!

[veloyage]

Thanks for your efforts. I tested the current build after the commit and can confirm that the test code above works. Unfortunately, my full code with display and all still crashes on the S2, and works fine on the S3 (alpha5). When running the first time it gets as far as the MQTT connection and fails to connect. After a soft reset it crashes in the same place as before, during the first get request (espidf.MemoryError).