Building precompiled module for itsybitsy esp32 gives incompatible .mpy arch

[graeme-winter]

CircuitPython version

Adafruit CircuitPython 9.1.1-5-g901dd228cb-dirty on 2024-07-28; Adafruit ItsyBitsy ESP32 with ESP32

Code/REPL

import mandelbrot

Behavior

>>> import mandelbrot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: incompatible .mpy arch

Description

Building a C-based extension module for circuitpython based on one for micropython, works fine one ARM e.g. itsybitsy M4, grand central M4, Teensy 4.1 but fails on ESP32 due to misalignment in the chain between the magic number to report the architecture and the circuitpython validation

Think the header is correct:

header: 43:06:29:1f

In Makefile I have ARCH = xtensawin which I think is correct?

Additional information

At this commit

commit 901dd228cbdb2069e66fae8c2108d44466ab1e7f (HEAD -> 9.1.x, origin/9.1.x)
Merge: 0b6b746779 a05584281e
Author: Scott Shawcroft <scott@adafruit.com>
Date:   Fri Jul 26 10:02:44 2024 -0700

    Merge pull request #9466 from jepler/issue9465-91x

    Only use pyi files for autoapi doc generation

• this delta

diff --git a/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.h b/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.h
index 9b20c5f73e..706bf07230 100644
--- a/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.h
+++ b/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.h
@@ -11,6 +11,8 @@
 #define MICROPY_HW_BOARD_NAME       "Adafruit ItsyBitsy ESP32"
 #define MICROPY_HW_MCU_NAME         "ESP32"

+#define MICROPY_PY_UCTYPES (1)
+
 #define MICROPY_HW_NEOPIXEL (&pin_GPIO0)
 #define CIRCUITPY_STATUS_LED_POWER (&pin_GPIO2)

diff --git a/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.mk b/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.mk
index a32c17b806..a485d6d017 100644
--- a/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.mk
+++ b/ports/espressif/boards/adafruit_itsybitsy_esp32/mpconfigboard.mk
@@ -10,3 +10,6 @@ CIRCUITPY_ESP_FLASH_SIZE = 8MB
 CIRCUITPY_ESP_PSRAM_SIZE = 2MB
 CIRCUITPY_ESP_PSRAM_MODE = qio
 CIRCUITPY_ESP_PSRAM_FREQ = 40m
+
+CIRCUITPY_ENABLE_MPY_NATIVE = 1
+MICROPY_ENABLE_DYNRUNTIME = 1

Tried debugging this myself but couldn't find where the logic was going sideways: the absence of printf or equivalent from the C layer & my inability to run this inside a debugger makes it hard to figure out why the failure is happening...

[graeme-winter]

To aid debug, added code to repo at

https://github.com/graeme-winter/circuitpython-mandelbrot-esp32

[graeme-winter]

I note that native extensions are unusual in the circuitpython ecosystem but I am sure it should work and this does work on ARM so felt the issue report was justified. Please let me know if there is anything I can do to help

[graeme-winter]

I note from discussion in https://github.com/adafruit/circuitpython/issues/6040 that this is somewhat unsupported but I have found in the past that it works following the upstream conventions (though I have not tested this with the 9.x series)

Adafruit CircuitPython 8.2.7-1-g817c486d6d-dirty on 2023-10-22; Adafruit Grand Central M4 Express with samd51p20
>>> import mandelbrot
>>> 

[graeme-winter]

Adding some debug info was useful: to persistentcode.h:

#define VALUE_TO_STRING(x) #x
#define VALUE(x) VALUE_TO_STRING(x)
#define VAR_NAME_VALUE(var) #var "="  VALUE(var)

#pragma message(VAR_NAME_VALUE(MPY_FEATURE_ARCH))

Emits at compile time:

In file included from ../../py/builtinimport.c:37:
../../py/persistentcode.h:86:9: note: '#pragma message: MPY_FEATURE_ARCH=(MP_NATIVE_ARCH_ARMV6M)'
   86 | #pragma message(VAR_NAME_VALUE(MPY_FEATURE_ARCH))
      |         ^~~~~~~

i.e. the incorrect architecture is being asserted in here

[graeme-winter]

My reading of this is it has only ever been tested with ARM Thumb instructions - but should be a relatively easy fix if I get a chance.

[tannewt]

We'd be happy to merge any changes you have to get this working. It isn't a high priority for us though.

You can print from C code using mp_printf(&mp_plat_print, <format string>, <args>);. Happy to help with debug tips in the #circuitpython-dev channel on Discord.

[graeme-winter]

Work in progress at https://github.com/graeme-winter/adafruit-circuitpython (detatched fork) now hitting

>>> import mandelbrot
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 271414342, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:396
ho 0 tail 12 room 4
load:0x40078000,len:13916
load:0x40080400,len:4
ho 8 tail 4 room 4
load:0x40080404,len:3156
entry 0x40080558
Serial console setup

Reached out for debugging help on discord as suggested

[graeme-winter]

Update on progress: the system locks up at mp_call_function_0(module_fun); in do_execute_raw_code - looking at the address of the function:

function at: 3f8077f0

and the memory map from the ESP32 technical reference manual:

I suspect that the issue is that the module is loaded into data memory not instruction => system throws exception when hits the function call. Tried adding

CONFIG_ESP_SYSTEM_MEMPROT_FEATURE = 0

to mpconfogport.mk but that didn't help...