Writing to the NVM on the RP2350 crashes to Safe Mode, CP-9.2.0

[b-blake]

CircuitPython version

Adafruit CircuitPython 9.2.0 on 2024-10-28; Adafruit Feather RP2350 with rp2350a
Board ID:adafruit_feather_rp2350
UID:A4E22C9B6217AF8D

Code/REPL

# in boot.py
go_dev_mode = 0xff # change to crash

if go_dev_mode:
    if microcontroller.nvm[0:1] == b"\x00":
        microcontroller.nvm[0:1] = b"\xff"
else:
    if microcontroller.nvm[0:1] == b"\xff":
        microcontroller.nvm[0:1] = b"\x00"

Behavior

In boot.py...

If you change go_dev_mode to 0xff from 0x00 or 0x00 from 0xff the RP2350 MCU crashes to Safe Mode when the RESET button is pushed. Also if you remove and restore power.

Description

The crash happens the first time RESET is pushed after value is changed. The second time RESET is pushed the NVM byte has been changed and no crash happens. When changed back, crash happens on the first RESET, good on the second.

Additional information

No response

[jepler]

fwiw just referring back to my private notes I tested microcontroller.nvm on rp2350 back on September 17 and didn't see this. However, I didn't record my exact testing steps. I was most likely testing on the Pico 2 and probably wasn't resetting the board during my testing procedure, just immediately reading the value back.

[jepler]

I tried reproducing this on Adafruit CircuitPython 9.2.0 on 2024-10-28; Raspberry Pi Pico 2 with rp2350a.

full content of boot.py:

# in boot.py
import microcontroller
go_dev_mode = 0xff # change to crash

if go_dev_mode:
    if microcontroller.nvm[0:1] == b"\x00":
        microcontroller.nvm[0:1] = b"\xff"
else:
    if microcontroller.nvm[0:1] == b"\xff":
        microcontroller.nvm[0:1] = b"\x00"

Full content of code.py:

import microcontroller
import supervisor
import time
import sys

# wait for serial connection
if supervisor.runtime.serial_connected:
    print("serial already connected, continuing")
else:
    while not supervisor.runtime.serial_connected:
        pass
    time.sleep(.1)
    while supervisor.runtime.serial_bytes_available:
        sys.stdin.read(1)

    print("press any key", end='')
    while not supervisor.runtime.serial_bytes_available:
        pass
    sys.stdin.read(1)

print()

print(f"dev mode is {microcontroller.nvm[0]}")

Testing method: Modify boot.py, changing the value of go_dev_mode between0x0 and 0xff, then press RESET button.

Outcome: circuitpython boots normally. When I press enter, the expected value is printed. Safe mode does not occur.

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
serial already connected, continuing

dev mode is 255

Code done running.

Press any key to enter the REPL. Use CTRL-D to reload.

[tio 09:20:27] Disconnected
[tio 09:20:30] Connected
press any key
dev mode is 0

Code done running.
Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.

Press any key to enter the REPL. Use CTRL-D to reload.

This could mean that it's specific to Feather RP2350 (not affecting all RP2350 boards) or that there's another aspect to reproducing the problem that I haven't followed.

If you have any more information it might be helpful. For various reasons it's not convenient for me to test on a Feather RP2350 right now.

[dhalbert]

I also tried to reproduce, on a Feather RP2350, with PSRAM https://www.adafruit.com/product/4677 installed, running 9.2.0 final. I did not get a crash, either by reset or power cycle.

My boot.py was exactly as above. I tried 0x00 and 0xff as values for go_dev_mode. My code.py was just a print statement.

@b-blake What is in your code.py?

[b-blake]

My boot.py

import time
import board
import storage
import neopixel
import digitalio
import microcontroller

go_dev_mode = 0x12# change to crash

if go_dev_mode:
    if microcontroller.nvm[0:1] == b"\x00":
        microcontroller.nvm[0:1] = b"\xff"
elif:
    if microcontroller.nvm[0:1] == b"\xff":
        microcontroller.nvm[0:1] = b"\x00"
else:
    microcontroller.nvm[0:1] = b"\x00"

[b-blake]

# The first 100 lines of a 700+ line program.

# Set the DS3231 time from GPS time.
# Eventually adjust DS3231 for more accurate time keeping.

import  gc
import  os
import sys
import  time
import  board
import  busio
import  random
import  binascii
import  neopixel
import  displayio
import  traceback
import  digitalio
import  terminalio
import  supervisor
import  microcontroller

import  adafruit_gps
import  adafruit_ds3231
import  adafruit_displayio_ssd1306
from    adafruit_display_text import label

# jepler's code
# wait for serial connection
if supervisor.runtime.serial_connected:
    print("serial already connected, continuing")
else:
    while not supervisor.runtime.serial_connected:
        pass
    time.sleep(.1)
    while supervisor.runtime.serial_bytes_available:
        sys.stdin.read(1)

    print("press any key", end='')
    while not supervisor.runtime.serial_bytes_available:
        pass
    sys.stdin.read(1)

print()

print(f"dev mode is {hex(microcontroller.nvm[0]}"))

set_alarms  = not True
set_DS3231  = not True
calibration = not True
CalValue    = -16 # +num = Slower clock, -num = Faster clock
Flip_Flop   = 0
OLED        = 0x3C
EEPROM      = 0x57
RTC3231     = 0x68
srsc = supervisor.runtime.serial_connected
sruc = supervisor.runtime.usb_connected
#>>> dir (supervisor)
#['__class__', '__name__', '__dict__', 
#'RunReason', 'SafeModeReason', 'get_previous_traceback', 'reload', 'reset_terminal', 'runtime', 'set_next_code_file', 'set_usb_identification', 'status_bar', 'ticks_ms']
#
#>>> dir (supervisor.runtime)
#['__class__', 
#'autoreload', 'ble_workflow', 'rgb_status_brightness', 'run_reason', 'safe_mode_reason', 'serial_bytes_available', 'serial_connected', 'usb_connected']
#>>> 
if srsc: print('\t*********************')
if srsc: print('\t*** Program Start ***')
_Delay = const(5)
for _ in range(_Delay):
    if srsc: print('\t\t', _Delay - _, end=' \r')
    time.sleep(1) # 1 second
if srsc: print('\t*********************')
if board.board_id != 'adafruit_feather_rp2350':
    if srsc: print(board.board_id, '~ Wrong CircuitPython OS!\a°')
    while True: pass
if srsc: print('\f', end='')

i = 63 # 127 63 31 15 7 3 1
I = i*2+1
Neo_White   = ((i, i, I))
Neo_Black   = ((0, 0, 0))
Neo_Red     = ((i, 0, 0))
Neo_Green   = ((0, i, 0))
Neo_Blue    = ((0, 0, I))
Neo_Yellow  = ((i, i, 0))
Neo_Cyan    = ((0, i, I))
Neo_Magenta = ((i, 0, I))

RAINBOW = [Neo_White, Neo_Black, Neo_Red, Neo_Green, Neo_Blue, Neo_Yellow, Neo_Cyan, Neo_Magenta] # 8
DoW     = ('Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday', 'Beerday')
Month   = ('Null', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December')
FREQ    = ["secondly", "minutely", "hourly", "daily", "weekly", "monthly"]
NMEA    = ('$GPRMC', '$GPVTG', '$GPGGA', '$GPGSA', '$GPGSV', '$GPGLL', '$GPTXT')

# board.D12, board.IO4
GPIO4 = digitalio.DigitalInOut(board.D12)
GPIO4.direction = digitalio.Direction.OUTPUT
GPIO4.value = True

[b-blake]

When I change the value: 1) I save the boot.py. The code.py restarts just fine. Sometimes it waits for a key in Jeff's code. 2) I push the RESET button on the Feather RP2350. The NeoPixel starts flashing Yellow. I don't believe code.py is started. 3) I open COM13 and REPL says the MCU is in Safe Mode. 4) I push RESET again and all is fine and code.py runs and waits for a key stroke.

[b-blake]

I don't know if this plays in this issue or not; but If I close REPL and edit and save code.py, Jeff's code is skipped and the rest of code.py starts and runs. Only if I change the byte written in boot.py does Jeff's code execute on the second RESET push.

P.S. I am on a Windows 10 machine.

[dhalbert]

This boot.py has invalid syntax:

if go_dev_mode:
    if microcontroller.nvm[0:1] == b"\x00":
        microcontroller.nvm[0:1] = b"\xff"
elif:
    if microcontroller.nvm[0:1] == b"\xff":
        microcontroller.nvm[0:1] = b"\x00"
else:
    microcontroller.nvm[0:1] = b"\x00"

You can't have elif: -- that's a syntax error. If that's really what you have in boot.py, you'll see "Syntax Error: invalid syntax" in boot_out.txt, and the safe mode would be due to something else.

If that's a red-herring typo, could you show the actual boot.py and come up with a minimal code.py that causes the safe mode? Thanks.

[b-blake]

Yup,

Let me change it back and retest.

[b-blake]

Changed back. I'll have to go look for your smarter code.

import time
import board
import storage
import neopixel
import digitalio
import microcontroller

go_dev_mode = 0xff # change to crash

if go_dev_mode == 0xff:
    if microcontroller.nvm[0:1] == b"\x00":
        microcontroller.nvm[0:1] = b"\xff"
else:
    if microcontroller.nvm[0:1] == b"\xff":
        microcontroller.nvm[0:1] = b"\x00"

[b-blake]

Dan,

I had to change your 'smaller' code example as below. go_dev_mode now has to be True or False. The line starting with mode (#2) results in 'mode' being set to True or False. So in the 'if' line go_dev_mode must be True or False. I could not get it to work when go_dev_mode was set to 0x00 or 0xff.

go_dev_mode = True
mode = (microcontroller.nvm[0] == 0xff)
if mode != go_dev_mode:
    microcontroller.nvm[0] = 0xff if go_dev_mode else 0x00

[dhalbert]

I am just looking for a small piece of code that causes the safe mode problem, regardless of whether or not it's really doing the mode logic correctly.

[b-blake]

I added extra code to my boot.py file. The NeoPixel flashes as appropriate before the crash. It runs fully. It looks like after the byte in NVM is changed the crash happens during the boot.py hand-off to code.py. I commented out line 47.

Here is a .zip of my boot file.

boot.zip

[dhalbert]

@b-blake I am unable to cause safe mode with the boot.py you gave. Here is a much simpler test, which always writes microcontroller.nvm[0]:

boot.py

import microcontroller

microcontroller.nvm[0] = (microcontroller.nvm[0] + 1) % 256

code.py

import microcontroller
import time

while True:
    time.sleep(1)
    print(f"{microcontroller.nvm[0]=}")

You should see micrcontroller.nvm[0] increment by 1 after each reset or power cycle.

Does this cause safe mode for you? It does not for me. I am running this on 9.2.0 on a Feather RP2350.

[b-blake]

I have found that the problem is adafruit_foo.py {sometimes}. Below is my boot.pt and a very cut down code.py, all of it. If you run as it is, it runs fine, just like Dan's boot.py & code.py. If you uncomment any one of the ignored adafruit_ libraries, it crashes. Why it likes and doesn't like is a mystery to me. FYI I have unzipped all the afafruit libraries in the /lib directory

boot.py

import microcontroller
microcontroller.nvm[1] = 1 - microcontroller.nvm[1]
print(microcontroller.nvm[1])

code.py

#Set the DS3231 time from GPS time.
#Eventually adjust DS3231 for more accurate time keeping.

import  gc
import  os
import  sys
import  time
import  board
import  busio
import  random
import  binascii
import  neopixel
import  displayio
import  traceback
import  digitalio
import  terminalio
import  supervisor
import  microcontroller

#import  adafruit_gps
#import  adafruit_ds3231
import  adafruit_displayio_ssd1306
#from    adafruit_display_text import label
#from    adafruit_onewire.bus import OneWireBus
#from    adafruit_ds18x20 import DS18X20

[dhalbert]

If you remove your cut-down boot.py, does safe mode not happen?

Could you erase CIRCUITPY and reinstall those libraries? It's possible one or more of them is corrupted in some way that's causing a hard crash.

[b-blake]

No boot.py, no crash.

restored boot.py deleted all files in /lib. Filled it with /lib/bundle 9.mpy dated 20241024 No change [crash]

copied all files to desktop folder Used rp2350_flash_nuke.uf2 loaded adafruit-circuitpython-adafruit_feather_rp2350-en_US-9.2.0.uf2 copied all files back No change [crash]

EDIT by @dhalbert per comment below.

[dhalbert]

Thanks, that's straightforward; I'll attempt to reproduce.

[b-blake]

No change is crash. I was not clear

[dhalbert]

I've reproduced this and am working on a fix.

[dhalbert]

@b-blake #9783 has a prospective fix. There are builds to try in the "Artifacts" listing here if you would like to test: https://github.com/adafruit/circuitpython/actions/runs/11637328418?pr=9783

[b-blake]

@dhalbert @jepler

Your fix works great on my Feather RP2350!

Still curious why including the adafruit_ libraries caused the crash. The relationship eludes me.

[dhalbert]

Still curious why including the adafruit_ libraries caused the crash. The relationship eludes me.

Thanks for testing. I don't think it's a specific library, but it may have to do with the size of the library. I got it down to importing adafruit_display_text, which is fairly large. As I trimmed things out, it stopped crashing.