Hard crash on ESP32S2 using 2 displays

[darianbjohnson]

CircuitPython 8.0.0-beta.6 Board - Adafruit Feather ESP32s2

Error message:

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
NLR jump failed. Likely memory corruption.
Please file an issue with the contents of your CIRCUITPY drive at 
https://github.com/adafruit/circuitpython/issues

Code:

import board
import displayio
import vectorio
import terminalio
from adafruit_simplemath import map_range
from adafruit_display_text import bitmap_label, wrap_text_to_lines, label #remove label later
from adafruit_st7789 import ST7789
import adafruit_imageload
from adafruit_bitmap_font import bitmap_font
import time
import gc
import math
import random
import rtc

import hexTileFunctions as hexFuncs

PANTHER_FONT_16 = bitmap_font.load_font("/fonts/Panther-16.bdf")
PANTHER_FONT_25 = bitmap_font.load_font("/fonts/Panther-25.bdf")
PANTHER_FONT_20 = bitmap_font.load_font("/fonts/Panther-20.bdf")
PANTHER_FONT_33 = bitmap_font.load_font("/fonts/Panther-33.bdf")
WEATHER_FONT = bitmap_font.load_font("/fonts/weather-31.bdf")

# Release any resources currently in use for the displays
displayio.release_displays()

# define main display
spi = board.SPI()
tft_cs = board.D9
tft_dc = board.D10

# define menu display
menu_tft_cs = board.D12
menu_tft_dc = board.D11

# set up menu display
menu_display_bus = displayio.FourWire(spi, command=menu_tft_dc, chip_select=menu_tft_cs)
menu_display = ST7789(
    menu_display_bus, rotation=0, width=135, height=240, rowstart=40, colstart=53
)

print(" menu bus", gc.mem_free())

# Make the display context
splash = displayio.Group()
menu_display.show(splash)

menuLabel = bitmap_label.Label(PANTHER_FONT_25, text="1abc\n2def\n3abc\n4def\n5abc\n6def\n7abc\n8def", color=0xFFFFFF, scale = 1,anchor_point = (0.0, 0.0), anchored_position= (0, 0))

splash.append(menuLabel)

# set up maindisplay
display_bus = displayio.FourWire(spi, command=tft_dc, chip_select=tft_cs)

display = ST7789(display_bus, width=240, height=320, rotation=0)

print(" display bus 2", gc.mem_free())

while True:
    pass

Background I compiled a new version of CPy for the ESP32s2 Feather, and I added a line to the mpconfig.h file to use two displays [#define CIRCUITPY_DISPLAY_LIMIT (2)]

The code compiles; and I am able to get the code to run on a first attempt. However, when I make a code change and save the file, I get the error listed above. The Mu editor looses the ability to write to the file and the board starts to run in safe mode.

I'm not exactly sure what I am doing wrong. Any thoughts?

Note - I am attempting to make a DEBUG version of the code for testing, but I'm running out of room on the board. Any ideas on what files to move so that I can create a DEBUG version?

[Breazile]

Adafruit CircuitPython 8.0.0-beta.6 on 2022-12-21; Adafruit Feather ESP32S3 4MB Flash 2MB PSRAM with ESP32S3 Board ID:adafruit_feather_esp32s3_4mbflash_2mbpsram UID:4F21AFA5124C

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Crash into the HardFault_Handler.
Please file an issue with the contents of your CIRCUITPY drive at 
https://github.com/adafruit/circuitpython/issues

I'm using an external 1.14" display with the ESP32S3 detailed here - https://github.com/Breazile/MandoPuter

It ran for almost 8 hours and then crashed hard. Second time this has happened. I found it crashed overnight and restarted it this morning.

Code is below and contents of the drive is here - https://github.com/Breazile/MandoPuter/blob/master/Releases/ESP32-S3Pre-Beskar.zip

"""
MandoPuter will display text in a Mandalorian font on a tiny LCD display

File   - code.py
Author - Jon Breazile

https://github.com/Breazile/MandoPuter

Font credits to ErikStormtrooper, the bitmap fonts were created from his TrueType font
http://www.erikstormtrooper.com/mandalorian.htm
"""
import gc
import time
import alarm
import board
import busio
import pwmio
import neopixel
import digitalio
import displayio
import adafruit_dotstar as dotstar
import adafruit_imageload
from analogio import AnalogIn
from adafruit_display_text import label
from adafruit_bitmap_font import bitmap_font
from adafruit_st7789 import ST7789
from adafruit_lc709203f import LC709203F
import audiocore
import audiobusio

"""
  ----------- User configurable items -----------

  This is where you can customize your display and hardware setup.
  Select either the Pre-Beskar (1.14" LCD) or Beskar (1.3" LCD) display.
  Some lines are commented out which means the line is not active.
  A commented line starts with a #

"""
# Set the display type
DISPLAY        = "Pre-Beskar"           # Adafruit 1.14" LCD display  https://www.adafruit.com/product/4383
#DISPLAY       = "Beskar"               # Adafruit 1.3" LCD display   https://www.adafruit.com/product/4313
DISP_BRIGHT    = 80                     # How bright to make the display - 0% to 100%

# Board being used
BOARD_TYPE     = "ESP32-S3"             # ESP32-S3                  https://www.adafruit.com/product/5477
#BOARD_TYPE    = "FeatherM4"            # Feather M4 Express        https://www.adafruit.com/product/3857
#BOARD_TYPE    = "ItsyBitsyM4"          # ItsyBitsy M4 Express      https://www.adafruit.com/product/3800
#BOARD_TYPE    = "ItsyBitsyRP2040"      # ItsyBitsy RP2040          https://www.adafruit.com/product/4888
#BOARD_TYPE    = "PiPicoRP2040"         # Raspberry Pi Pico RP2040  https://www.adafruit.com/product/4864

# Mandalorian charater sequence that is shown on the display
messages       = [ "MLM", "JBM", "SAS", "JAS", "JBM", "MLM", "SAS", "AJS", "SAS"]
# Time that each character group is shown 0.50 is 500 milliseconds, or 1/2 of a second
delays         = [  0.75,  0.75,  0.650,  0.75, 0.50,  0.84,  1.00,  0.35,  0.84]
TEXT_COLOR     = 0xFF0000               # Red on black (you can chose colors here - https://www.color-hex.com/)

# Name of the owner shown after startup and before the sequence starts
SHOW_NAME      = 1                      # Set to 1 to display the name, or 0 to not display a name
OWNER_NAME     = "Your Name Here"       # Name of the owner to be shown
NAME_COLOR     = 0x00FF00               # Green on black (you can chose colors here - https://www.color-hex.com/)
NAME_HOLD      = 3.0                    # How many seconds to display the name

# Banner graphic(s) shown after the owner's name and before the sequence starts
SHOW_IMG       = 2                      # How many images to show. 0 = no images, 1 = 1 image, 2 = 2 images
IMG1           = "TheMandalorian135.bmp"         # File name of the first 8 bit BMP graphic to be shown after each text sequence
IMG1_HOLD      = 5.00                   # How long the first image is displayed in seconds
IMG2           = "BabyYoda135.bmp"      # File name of the second 8 bit BMP graphic to be shown after the first image
IMG2_HOLD      = 5.00                   # How long the second image is displayed in seconds

# Other settings for debugging and battery monitoring
BATTERY_SZ     = 500                    # Size of battery in mAh (only for the ESP32-S3 board)
BATTERY_MON    = 0                      # Set to 1 to enable the battery monitor, 0 to disable it
LOW_BATT_LEVEL = 10                     # Show the low battery icon when the battery goes below this percentage
ENABLE_LEDS    = 0                      # Set to 1 to turn on LEDs for debugging, set to 0 to save battery
SPI_SPEED      = 48000000               # How fast the SPI bus to the LCD operates
"""
# ---------------------------------------------------------------------------------
"""

wave_file = open("Whistling-Birds-Fired2.wav", "rb")
wave = audiocore.WaveFile(wave_file)

# For Feather M0 Express, ItsyBitsy M0 Express, Metro M0 Express
audio = audiobusio.I2SOut(board.TX, board.D12, board.D11)

def DisplayName(name, hold, color, init_tm) :
    ownerfont = bitmap_font.load_font("Alef-Bold-18.bdf")  # 18 point bitmap font
    banner_text = label.Label(ownerfont, text=name, color=color)
    banner_text.x = int(((display.width - banner_text.bounding_box[2])/2)-1)
    banner_text.y = int(((display.height - banner_text.bounding_box[3])/2)+1)
    #banner_text.y = int((display.height / 2)-5)
    display.show(banner_text)
    display.refresh()
    if SHOW_IMG > 0 :
        time.sleep(hold)              # Display the name before the graphics
    else :
        if hold > init_tm :
            time.sleep(hold - init_tm)  # minus the initialization time if there are images
    # release memory
    del ownerfont
    del banner_text
    gc.collect()

def DisplayImage(img, hold, images, init_tm) :
    # Create the first image centered on the display
    try :
        bitmap, palette = adafruit_imageload.load(img, bitmap=displayio.Bitmap, palette=displayio.Palette)
    except:
        bitmap = displayio.OnDiskBitmap(img)
        palette = bitmap.pixel_shader
    x = int((display.width - bitmap.width) / 2)
    y = int((display.height - bitmap.height) / 2)
    if x < 0 : x = 0
    if y < 0 : y = 0
    tile_grid = displayio.TileGrid(bitmap, pixel_shader=palette, x=x, y=y)
    img = displayio.Group()
    img.append(tile_grid)
    display.show(img)
    display.refresh()
    if hold > init_tm :
        if images > 1 :
            time.sleep(hold)              # hold the first image before showing the second
        else :
            if hold > init_tm :
                time.sleep(hold - init_tm)  # minus the initialization time if there is 1 image
    img.pop()
    del bitmap
    del img
    del tile_grid
    gc.collect()

def GetBattPercent(batt_pin) :
    percent = 0
    # read the battery voltage (approximation, not exact levels with no fuel gauge to read)
    batt_volts = (batt_pin.value * 3.3) / 65536 * 2
    if batt_volts > 3.80 :
        percent = 86                    # 100% to 81% capacity
    elif batt_volts > 3.65 :
        percent = 50                    # 80%  to 31% capacity
    elif batt_volts > 3.40 :
        percent = 25                    # 30%  to 16% capacity
    else :
        percent = 5                     # 15%  to  0% capacity
    return percent

# Turn off the LCD Backlight
displayio.release_displays()

# Setup the bus, display object, and font for the display
if BOARD_TYPE == "PiPicoRP2040" :
    spi = busio.SPI(clock=board.GP10, MOSI=board.GP11, MISO=board.GP12)
else :
    spi = board.SPI()
while not spi.try_lock():
    pass
spi.configure(baudrate=SPI_SPEED)  # Configure SPI with the specified speed
spi.unlock()
if BOARD_TYPE == "FeatherM4" or BOARD_TYPE == "ESP32-S3" :
    tft_cs  = board.D6
    tft_dc  = board.D9
    lcd_rst = board.D5
    lcd_light = board.D10
elif BOARD_TYPE == "PiPicoRP2040" :
    tft_cs    = board.GP28
    tft_dc    = board.GP14
    lcd_rst   = board.GP27
    lcd_light = board.GP15
else:
    lcd_light = board.D10
    tft_cs = board.D2
    tft_dc  = board.D3
    lcd_rst = board.D4

display_bus = displayio.FourWire(spi, command=tft_dc, chip_select=tft_cs, baudrate=SPI_SPEED, reset=lcd_rst, polarity=0, phase=0)
if DISPLAY == "Pre-Beskar":
    display     = ST7789(display_bus, rotation=270, width=240, height=135, rowstart=40, colstart=53, auto_refresh=False, backlight_pin=lcd_light, brightness=0)
    font        = bitmap_font.load_font("mandalor135.bdf")  # 135 pixel tall bitmap font
    offset      = 12
elif DISPLAY == "Beskar":
    display     = ST7789(display_bus, rotation=0, width=240, height=240, rowstart=80, auto_refresh=False, backlight_pin=lcd_light, brightness=0)
    font        = bitmap_font.load_font("mandalor165.bdf")  # 165 pixel tall bitmap font
    offset      = 14
stage = displayio.Group()
display.show(stage)

# Disable WiFi power
if BOARD_TYPE == "ESP32-S3" :
    import wifi
    wifi.radio.enabled = 0

# Configure LEDs
if BOARD_TYPE == "ESP32-S3" :
    # setup the onboard neopixel LED power control
    led_pwr = digitalio.DigitalInOut(board.NEOPIXEL_POWER)
    led_pwr.direction = digitalio.Direction.OUTPUT

if BOARD_TYPE == "FeatherM4" or BOARD_TYPE == "ESP32-S3" or BOARD_TYPE == "ItsyBitsyRP2040" :
    led = neopixel.NeoPixel(board.NEOPIXEL, 1)
elif BOARD_TYPE == "ItsyBitsyM4" :
    led = dotstar.DotStar(board.APA102_SCK, board.APA102_MOSI, 1) # onboard dotstar
elif BOARD_TYPE == "PiPicoRP2040" :
    led = digitalio.DigitalInOut(board.LED)
    led.direction = digitalio.Direction.OUTPUT

if ENABLE_LEDS > 0 :
    if BOARD_TYPE != "PiPicoRP2040" :
        led.brightness = 0.05  # dim the LED to 5%
        led[0] = (255, 0, 255) # purple
    if BOARD_TYPE == "ESP32-S3" :
        led_pwr.value = True
else :
    if BOARD_TYPE != "PiPicoRP2040" :
        led.brightness = 0  # dim the LED to 0%
    if BOARD_TYPE == "ESP32-S3" :
        led_pwr.value = False

# Setup battery monitoring
if BATTERY_MON > 0 :
    lowbattX = 0
    lowbattY = 0
    vbat_voltage = 0
    if BOARD_TYPE == "ESP32-S3" :
        i2c = board.I2C()  # uses board.SCL and board.SDA
        sensor = LC709203F(i2c)
        sensor.PackSize = BATTERY_SZ
    elif BOARD_TYPE == "FeatherM4" :
        vbat_voltage = AnalogIn(board.VOLTAGE_MONITOR) # for measuring battery voltage
    elif BOARD_TYPE == "PiPicoRP2040" :
        # battery voltage measurements need a jumper from VSYS to ADC0 (pin 39 - 31)
        vbat_voltage = AnalogIn(board.GP26) # for measuring battery voltage
    elif BOARD_TYPE == "ItsyBitsyM4" or BOARD_TYPE == "ItsyBitsyRP2040" :
        # battery voltage measurements need a jumper from batt to A1
        vbat_voltage = AnalogIn(board.A1)

    # Create the second image centered on the display
    lowbattImg, lowbattPal = adafruit_imageload.load("LowBatt.bmp", bitmap=displayio.Bitmap, palette=displayio.Palette)
    x = int(display.width - lowbattImg.width)
    y = int(display.height - lowbattImg.height)
    if x < 0 : x = 0
    if y < 0 : y = 0
    lowbattX = x
    lowbattY = y
else :
    # Disable I2C port power
    if BOARD_TYPE == "ESP32-S3" :
        i2c_pwr = digitalio.DigitalInOut(board.I2C_POWER)
        i2c_pwr.direction = digitalio.Direction.OUTPUT
        i2c_pwr.value = False

# Turn on the Backlight
display.brightness = DISP_BRIGHT / 100
init_time = 4.5   # how long it takes to initialize the sequence

# Show the owner's name at startup
if SHOW_NAME > 0 :
    DisplayName(OWNER_NAME, NAME_HOLD, NAME_COLOR, init_time)

# Show the banner graphic(s)
if SHOW_IMG > 0 :
    DisplayImage(IMG1, IMG1_HOLD, SHOW_IMG, init_time)
    DisplayImage(IMG2, IMG2_HOLD, SHOW_IMG, init_time)

audio.play(wave)

gc.collect()
low_batt_icon = 0
batt_percent  = 0

# Prepare the Mandalorian characters
text   = label.Label(font, text=messages[0], color=TEXT_COLOR)
text.x = int(((display.width - text.width)/2)-1)
text.y = int((display.height / 2)+offset)
if text.x < 0:
    text.x = 0
if text.y < 0:
    text.y = 0
stage.append(text)

# Initialize the low battery icon
if BATTERY_MON > 0 :
    batt_tile = displayio.TileGrid(lowbattImg, pixel_shader=lowbattPal, x=lowbattX, y=lowbattY)

display.show(stage)
while True:
    index = 0
    for msg in messages:
        text.text = msg
        display.refresh()
        if BOARD_TYPE == "PiPicoRP2040" :
            time.sleep(delays[index]* 0.75)
        else :
            time.sleep(delays[index])
        index = index + 1

    if BATTERY_MON > 0 :
        if BOARD_TYPE == "ESP32-S3" :
            sensor = LC709203F(i2c)
            batt_percent = sensor.cell_percent
        else :
            batt_percent = GetBattPercent(vbat_voltage)
        if batt_percent < LOW_BATT_LEVEL :
            # Add low battery icon
            if low_batt_icon == 0 :
                stage.append(batt_tile)
                low_batt_icon = 1
        else :
            if low_batt_icon > 0 :
                # Remove low battery icon
                stage.pop()
                low_batt_icon = 0

    #import supervisor

    #display.brightness = 0
    #splash = display.root_group
    #supervisor.reset_terminal(display.width, display.height//2)
    #splash.y=display.height//2
    #time_alarm = alarm.time.TimeAlarm(monotonic_time=time.monotonic() + 20)
    #alarm.exit_and_deep_sleep_until_alarms(time_alarm, preserve_dios=[digitalio.DigitalInOut(board.D10)])

    #pin_alarm = alarm.pin.PinAlarm(pin=board.D11, value=False, pull=True)
    # Exit the program, and then deep sleep until the alarm wakes us.
    #alarm.exit_and_deep_sleep_until_alarms(pin_alarm)

[FoamyGuy]

@darianbjohnson you can disable things by setting 0 to some variables inside of mpconfigboard.mk.

You can see an example that disables some things here: https://github.com/adafruit/circuitpython/blob/9c066825a7c2127f69da90e80de5eb94e2d92a00/ports/atmel-samd/boards/matrixportal_m4/mpconfigboard.mk#L15-L19

doing a few of those that aren't used by your program should free up space to allow a debug build.

[FoamyGuy]

@Breazile that may be a different issue I think. Your device is ESP32S3 and seems to only use 1 display. But this issue is for an S2 device and using two displays.

It may be best to create a seperate issue for yours instead of posting here.

[FoamyGuy]

I have replicated this issue on a feather S2 TFT. I believe the same issue actually effects more ports as well, perhaps all of them. I'm thinking it may trace back to the displayio api change, but haven't done before/after testing yet to narrow it down.

I think @Neradoc saw this occur on Monster M4sk as well, and I've seen the same on that device.

It seems to boil down to right now the initial setup of 2 displays works fine, but then whenever the device resets it will hard fault, as described in the initial issue post.

I have collected this decoded backtrace from Feather S2 TFT:

❯ python tools/decode_backtrace.py adafruit_feather_esp32s2_tft
adafruit_feather_esp32s2_tft
? 0x400A16A1:0x3FFE59A0 0x400A23F2:0x3FFE59C0 0x400A1541:0x3FFE59E0 0x4008307D:0x3FFE5A00 0x4008E2E7:0x3FFE5A20 0x4008E418:0x3FFE5A40 0x400C7CF9:0x3FFE5A60 0x400C7FDF:0x3FFE5A80 0x400CAC5D:0x3FFE5AA0 0x400A0CF7:0x3FFE5AC0 0x400A10AD:0x3FFE5B10 0x400A14C5:0x3FFE5B70 0x400A18B3:0x3FFE5BA0 0x40176990:0x3FFE5BC0
0x400a16a1: reset_cpu at /home/timc/repos/circuitpython/circuitpython/ports/espressif/supervisor/port.c:425
0x400a23f2: reset_into_safe_mode at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../supervisor/shared/safe_mode.c:139
0x400a1541: nlr_jump_fail at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../main.c:1153
0x4008307d: nlr_jump at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../py/nlrsetjmp.c:35
0x4008e2e7: mp_raise_msg at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../py/runtime.c:1626
0x4008e418: mp_raise_ValueError at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../py/runtime.c:1674
0x400c7cf9: common_hal_displayio_display_set_root_group at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../shared-module/displayio/Display.c:410
0x400c7fdf: reset_display at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../shared-module/displayio/Display.c:440
0x400cac5d: reset_displays at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../shared-module/displayio/__init__.c:279
0x400a0cf7: cleanup_after_vm at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../main.c:323
0x400a10ad: run_code_py at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../main.c:475
0x400a14c5: main at /home/timc/repos/circuitpython/circuitpython/ports/espressif/../../main.c:1087
0x400a18b3: app_main at /home/timc/repos/circuitpython/circuitpython/ports/espressif/supervisor/port.c:544
0x40176990: main_task at /home/timc/repos/circuitpython/circuitpython/ports/espressif/esp-idf/components/freertos/port/port_common.c:141

I will try to narrow it further and see if I can figure out a fix.

[Neradoc]

Yep I get a hard crash when saving on the Monster M4sk, with the displays setup form the monster_mask library (which occasionally causes the file I'm editing to be corrupted).

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
NLR jump failed. Likely memory corruption.
Please file an issue with your program at https://github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.

[FoamyGuy]

I followed the backtrace a bit, and I think this actually is the same kind of issue as #7829 is addressing. The circuitpython_spash was being added after already being added to the first I think.

I'm not really sure why this one resulted in the hard crash though as the backtrace does flow through a line trying to raise the exception. It must be failing for some reason down the line.

I added a commit to that PR branch that addresses the issue. It's adding the same if statement previously added into the constructor into reset_display() function also.

Tested successfully with the Feather S2 TFT. Will try Monster M4sk next.

Worth noting something may still be wonky though because now I no longer hard crash but I do seem to see duplicated terminal outputs on the screen. First ctrl-C from running results in 3 duplicates stacked one on top of the other. The next ctrl-C to go to reply splits those 3 rows into 2 columns each to make 6 total copies visible.

[FoamyGuy]

I tried the latest version from that PR on a Monster M4sk and I do still get a hard fault. Slightly different wording I don't have "NLR jump failed. Likely memory corruption."but rather "fault detected by hardware"

I did ctrl-C instead of saving a file though, that could factor in.

I'm not sure if it's possible or how to get hard crash traces out of the samd51 / Monster M4sk but if we could it might help work toward the solution.

Later on the week I'll try to probe a bit further on the feather S2 TFT setup to see if I can figure out the duplicated terminal outputs, and hopefully manage to find whatever hard fault the Monster M4sk is still seeing to get it resolved too.

[FoamyGuy]

@darianbjohnson if you have a moment please try the latest build from the S3 link for your project that this issue was originally made for.

I think that the specific hard crash you observed should be resolved after #7829, but I'm also still trying to work through further potential issues. It'd be helpful to know how your setup behaves on the current version.

[darianbjohnson]

@FoamyGuy I’ll test it out… though I might need a few weeks to retest (heads down on another project that I need to resolve first)

[furbrain]

@FoamyGuy I wonder if issue #2204 (fixed by #7983) may be responsible for the hard faults here?

[dhalbert]

Probably fixed by #7983. Re-test.