Script hangs after 12 hours with 1-second refresh

[Bodobolero]

I am running the following script on Arduino RP2040 Connect with Arduino IoT Cloud. The script reads temperature and humidity from DHT20 over I2C using https://files.seeedstudio.com/wiki/Grove-Temperature-Humidity-Sensor/Pico-micropython-master.zip dht20.py.

The script runs fine for about 12 hours (about 40.000 loop iterations) and then gets stuck. I suspect this is due to heap fragmentation in ArduinoIoTCloud python module or other module it uses.

It would be helpful to add a watchdog like in the C implementation that reboots the board in case of becoming unresponsive.

Here is my script to repro the problem

from machine import I2C, Pin
from dht20 import DHT20
import time
import network
import logging
from arduino_iot_cloud import ArduinoCloudClient

from secrets import WIFI_SSID
from secrets import WIFI_PASSWORD
from secrets import DEVICE_ID
from secrets import CLOUD_PASSWORD

led = Pin("LED", Pin.OUT)  # Configure the desired LED pin as an output.
i2c = I2C(0, scl=Pin(13), sda=Pin(12))
dht20 = DHT20(i2c)

def on_switch_changed(client, value):
    # Toggles the hardware LED on or off.
    led.value(value)

    # Sets the value of the cloud variable "led" to the current state of the LED
    # and thus mirrors the hardware state in the cloud.
    client["led"] = value

def read_temperature(client):
    global dht20
    temperature = dht20.dht20_temperature()
    logging.info(f"DHT20: {temperature} C")
    return temperature

def read_humidity(client):
    global dht20
    humidity = dht20.dht20_humidity()
    logging.info(f"DHT20: {humidity} %")
    return humidity

def wifi_connect():
    if not WIFI_SSID or not WIFI_PASSWORD:
        raise (
            Exception("Network is not configured. Set SSID and passwords in secrets.py"))
    wlan = network.WLAN(network.STA_IF)
    wlan.active(True)
    wlan.connect(WIFI_SSID, WIFI_PASSWORD)
    while not wlan.isconnected():
        logging.info("Trying to connect. Note this may take a while...")
        time.sleep_ms(500)
    logging.info(f"WiFi Connected {wlan.ifconfig()}")

if __name__ == "__main__":

    # Configure the logger.
    # All message equal or higher to the logger level are printed.
    # To see more debugging messages, set level=logging.DEBUG.
    logging.basicConfig(
        datefmt="%H:%M:%S",
        format="%(asctime)s.%(msecs)03d %(message)s",
        level=logging.INFO,
    )

    logging.info(f"Connect to WiFi")

    # NOTE: Add networking code here or in boot.py
    wifi_connect()

    # Create a client object to connect to the Arduino IoT cloud.
    # For MicroPython, the key and cert files must be stored in DER format on the filesystem.
    # Alternatively, a username and password can be used to authenticate:
    client = ArduinoCloudClient(
        device_id=DEVICE_ID, username=DEVICE_ID, password=CLOUD_PASSWORD)

    # Register cloud objects.
    # Note: The following objects must be created first in the dashboard and linked to the device.
    # This cloud object is initialized with its last known value from the cloud. When this object is updated
    # from the dashboard, the on_switch_changed function is called with the client object and the new value.
    client.register("ledSwitch", value=None,
                    on_write=on_switch_changed, interval=0.250)

    # This cloud object is updated manually in the switch's on_write_change callback to update the LED state in the cloud.
    client.register("led", value=None)

    # read only variables temperature and humidity
    client.register("temperature", value=None, on_read=read_temperature, interval=1.0)
    client.register("humidity", value=None, on_read=read_humidity, interval=1.0)

    logging.info(f"starting IoT client loop")

    # Start the Arduino IoT cloud client.
    client.start()

Maybe the problem is in my script and not in the arduino-iot-cloud-py implementation - however so far I did not succeed in deploying a Arduino IoT cloud project in production in Micropython while I have several Arduino IoT cloud "things" successfully deployed already using the Arduino Cloud C/C++ libraries.

[iabdalkader]

@Bodobolero I don't think so, no, let me test that firmware and make sure it's working, will get back to you later today.

[iabdalkader]

I saved your script as main.py, and loaded the firmware attached here, https://github.com/arduino/arduino-iot-cloud-py/issues/72#issuecomment-1787270297 Seems to work normally, I don't see that error you posted. Note I removed the DHT part and used the internal sensor like you did. I also made sure UART REPL is working. Maybe you have a boot.py in that's getting in the way ? You should remove it if you do.

[Bodobolero]

I ordered a USB to serial bridge which I expect to be delivered on Nov, 2nd. Then I can start debugging.

@iabdalkader I have received and tested my USB-2-TTL Adapter and have connected it to the microcontroller. I verified that I can open a terminal (minicom on macOS) and see the output, for example:

socket_ioctl(56, 3)
17:38:20.000 DHT20: 21.78382 C
17:38:20.000 Update: temperature value: 21.78382 ts: 1698946700
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
socket_ioctl(56, 3)
17:38:21.000 Pushing records to Arduino IoT cloud:
17:38:21.000   ==> record: temperature value: 21.78382...
socket_send(56, 31)
socket_polling_rw(56, 5000, 2)

I am now waiting for the failure scenario to debug it.

[Bodobolero]

@iabdalkader so far no outage with the new debug firmware. Maybe the timing is different due to more debug output - or the problem is not deterministic. I will keep running the microcontroller until it fails.

[iabdalkader]

If you'd like to test another firmware with less debugging output just let me know.

[Bodobolero]

@iabdalkader it is still running. Now I wonder if the debug firmware 'MicroPython v1.22.0-preview.69.gc146017f8.dirty on 2023-10-31;' already contained some fixes that are not yet released in the upstream Micropython firmware I used before and also not in the prior firmware version 'MicroPython v1.22.0-preview.51.g91a3f1839.dirty on 2023-10-23; Arduino Nano RP2040 Connect with RP2040'?

[iabdalkader]

No all of the firmware images attached here have the same fixes. I don't know which upstream firmware you used before, but the nightly/preview builds (Not the stable one) should have the fixes by now (and more fixes will follow).

If you want to retest an image with no debugging output, this one should do:

https://micropython.org/resources/firmware/ARDUINO_NANO_RP2040_CONNECT-20231108-v1.22.0-preview.133.gdff293840.uf2

This one also doesn't have debugging output enabled, but enables REPL over UART.

firmware.uf2.zip

[Bodobolero]

This one also doesn't have debugging output enabled, but enables REPL over UART.

firmware.uf2.zip

@iabdalkader the firmware with debugging output and REPL over UART never failed and kept running. I have now deployed this version with debugging output disabled but REPL over UART to see if it still fails without debugging output enabled.

[Bodobolero]

@iabdalkader Since I cannot reproduce the problem with the current script and the latest firmware you provided I close this issue for now, until I have a reliable repro. Thanks for your support.

[iabdalkader]

That's great! Feel free to reopen this issue if it ever gets stuck again.