Enable native modules

[tannewt]

Native modules are a feature of MicroPython that allow for code compiled to native assembly to be loaded dynamically into MP. This can be really helpful for adding fast code without having to recompile the main runtime.

There are some things we need to handle before we can enable it:

• [ ] Establish a file name suffix convention so it's clear what platforms a native file support.
• [ ] Also include this info in the file itself and error if the suffix or internal metadata don't match.

These two pieces are critical to explaining to folks when a native module can or cannot work with their board. They are the bare minimum.

Once we have that, we can explore:

• [ ] Providing linking hooks to public_hal apis that may or may not mirror common_hal apis.
• [ ] Provide CI tooling for auto generating native files for all supported architectures (assuming the code is agnostic.)

[s-ol]

For reference, here's the documentation of the corresponding upstream micropython feature (mpy / native modules).

... what platforms a native file support. Also include this info in the file itself and error if the suffix or internal metadata don't match.

@tannewt how do you intend "platform" in this case? Upstream .mpy encodes the ARCH which is of course the minimum prerequisite for executing machine code. I'm not sure if it would be necessary to build separately for different BOARD configurations by default, although it might make sense in some cases (small changes like pinouts could be resolved at runtime, but different boards of the same port might require e.g. vastly different buffer sizes via #defines etc).

[tannewt]

@tannewt how do you intend "platform" in this case? Upstream .mpy encodes the ARCH which is of course the minimum prerequisite for executing machine code.

Architecture is what I meant. Not board.

[s-ol]

Architecture is what I meant. Not board.

Then aside from establishing a filename convention (module.$ARCH.mpy ?), is there a need to deviate from the upstream native module implementation?

Last I tried to build the example native modules it was not an option because there was no rp2040 support in micropython, but this has been added in micropython v1.19, so after #7954 is completed I'd be curious to try that again.

[tannewt]

Then aside from establishing a filename convention (module.$ARCH.mpy ?), is there a need to deviate from the upstream native module implementation?

I don't think so.

Probably best to have a designator that it's for CircuitPython if there are any MP specifc things we won't have.

[s-ol]

True, choosing something other than .mpy also makes sense since the dynamic linking table will be different so there will not be cross-compatibility. From the upstream docs linked above:

Linker limitation: the native module is not linked against the symbol table of the full MicroPython firmware. Rather, it is linked against an explicit table of exported symbols found in mp_fun_table (in py/nativeglue.h), that is fixed at firmware build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system function, for example.

New symbols can be added to the end of the table and the firmware rebuilt. The symbols also need to be added to tools/mpy_ld.py’s fun_table dict in the same location. This allows mpy_ld.py to be able to pick the new symbols up and provide relocations for them when the mpy is imported. Finally, if the symbol is a function, a macro or stub should be added to py/dynruntime.h to make it easy to call the function.

I don't have enough insight into the complexity of this aspect, but from a module author perspective it sounds like ideally you would have access to:

• shared-bindings
• $PORT/bindings
• $PORT/common-hal
• mp_*
• port-specific SDK (e.g. the pico-sdk)

I guess the port-specific SDK can be linked statically? Is there a trade-off to consider when including more functions on this list (possibly firmware build size)?

[tannewt]

This is what I'd mark using the prefix public_hal_. The smaller it is, the easier to maintain everything.

[jonnor]

I am using native modules for ML algorithms in https://github.com/emlearn/emlearn-micropython Glad to see that this is being thought of for CircuitPython, as I would like to bring these modules there as well. Regarding naming convention, I have for now resorted to producing folders with the ARCH name and MPYABI (6.1 for MicroPython 1.20+). This means that the filename is standard and can be imported normally.

I believe that CircuitPython will also need some ABI version identifier, in addition to ARCH.

I also use CI to produce the .mpy files across the architectures. It's not the prettiest setup, but it works :)

[jimmo]

FYI, since this issue was raised we (MicroPython) have added a "sub version" along with the main "mpy version". (@jonnor refers to this above, i.e. the current version is 6.1).

So now both the runtime and .mpy files have a version and sub version. As long as the main version matches, a bytecode .mpy file will work. To load a .mpy with native code (i.e. it has an arch set in its header) then the sub version must also match. cc @jepler @dhalbert this might be relevant when you're merging v1.19 and v1.20. We tried to do this in a backwards compatible way by re-using a field that was unused as of v1.19 and was zero (hence v6.0 was implicit).

This allows us to change the native ABI more frequently than the bytecode ABI. Typically we update the native ABI because we need to add new entries to dynruntime.h.

MicroPython will also need to solve a similar naming problem if/when we start publishing packages that use native code (either via the native emitter or dynamic native modules) to micropython-lib. That said, we currently already handle this for the mpy version, but in the directory name, not the filename.

e.g. https://micropython.org/pi/v2/package/6/hmac/3.4.3.json gives you the metadata for the mpy-v6.x files for v3.4.3 of the hmac package.

[bluelasers]

CircuitPython will only pick one major and one minor version of MicroPython per major version of CircuitPython? This would be to enable consistency of native modules. Native modules must be promoted to CircuitPython major versions.

[bluelasers]

Symbols are an area which will require a top level choice and there will be consequences of that choice. This choice will likely involve some level of support by Adafruit. MicroPython currently is a fan of static symbols and management be provided by fork or firmware flavors. (Based on your quote.) This represents one choice. The RP2040 and SDK somewhat forces this.

This creates challenges for Adafruit. To avoid this Adafruit may decided to ban certain practices in native modules. This will limit performance in some cases. Note Adafruit may also force native modules being compiled into CircuitPython to support other features. Adafruit will likely discourage native modules for beginners. Adafruit may also attempt to limit access to their build system and attempt to only endorse some native modules.

Community native modules may face challenges of their own.

[tannewt]

CircuitPython will only pick one major and one minor version of MicroPython per major version of CircuitPython? This would be to enable consistency of native modules. Native modules must be promoted to CircuitPython major versions.

We don't have a specific plan to support native modules.

Until then, we welcome any native code additions as a new module in the core CP repo instead. That way it can be maintained by others and doesn't rely on the native module mechanics from MP.