Error when using native extensions built with maturin and pyo3

[alec-deason]

I'm trying to use a native extension built with maturin and pyo3 in a wasm binary built by componentize-py. The python package with the native extension works perfectly when built and installed in my native python environment. When it's built for wasm32-wasi the output looks sensible to me but when I try to run componentize-py I get this error:

AssertionError: failed to extract linking metadata from /0/rusty_python/rusty_python.cpython-312-wasm32-wasi.so

Caused by:
    unsupported export kind for memory: Memory

I can run the matrix-math example with numpy so I can see that this should be possible.

I suspect there's some sort of version mismatch but I'm not sure where it's coming from or if it's a problem with componentize-py or the way I'm using maturin.

The attached tar has a minimal setup including the template code from maturin new with the pyo3 option selected.

example.tgz

I have maturin 1.7.1 installed and componentize-py build from the latest source.

This is how I'm doing my test:

I build the rust extension with: PYO3_CROSS_LIB_DIR="/PATH/TO/COMPONENTIZE/REPO/cpython/builddir/wasi/" maturin build --target wasm32-wasi

Then I unpack the resulting .whl file in the top level directory with wheel unpack rusty_python_src/target/wheels/rusty_python-0.1.0-cp312-cp312-any.whl and move the inner rusty_python directory out to the top level.

Then I run componentize-py with: componentize-py -d /PATH/TO/COMPONENTIZE/REPO/wit -w wasi:cli/command@0.2.0 componentize example -o cli.wasm

which gives me:

Traceback (most recent call last):
  File "python_venv/bin/componentize-py", line 8, in <module>
    sys.exit(script())
             ^^^^^^^^
AssertionError: failed to extract linking metadata from /0/rusty_python/rusty_python.cpython-312-wasm32-wasi.so

Caused by:
    unsupported export kind for memory: Memory

[dicej]

Thanks for reporting this, @alec-deason.

What I think you're running into here is that crate-type = ["cdylib"] means something different for Rust on wasm32-* targets than it does for other targets. The resulting output (rusty_python.cpython-312-wasm32-wasi.so) is not a dynamic library as defined here, and thus it's not in a format wit-component (which is what componentize-py uses internally to link native extensions into the final component) can handle.

The important differences between a Wasm dynamic library and what Rust produces when you tell it crate-type = ["cdylib"] are that the former only imports a memory and function table (i.e. is intended to be linked with another module that provides those things), while the latter has its own memory and function table (i.e. is intended to be run on its own). The abuse of the cdylib term is very misleading IMHO, but it's too late to change that.

Anyway, there's currently no way to tell Rust to build a Wasm dynamic library directly. Instead, you have to tell it crate-type = ["staticlib"], which will produce a .a file, then use clang from a recent version of wasi-sdk to turn that into a .so file like we do here. Except it's not quite that easy (because computers); you first need to build your crate using -fPIC (meaning "generate position-independent code"), and build the Rust standard library that way, too. That means using a nightly build of Rust and the -Z build-std flag like so. The upcoming wasm32-wasip2 target will make that easier since it will generate -fPIC code by default (I think; @alexcrichton correct me if I'm wrong), but for now that's the drill.

Given that you're using maturin to build rather than cargo, you'll probably need to research how to forward all those flags through the various layers of tools. Assuming you can do that, everything should work.

Hope that helps. Let me know if anything is unclear.

[alec-deason]

Heh, ok. There are several things there I wasn't understanding then. Thanks for your detailed response. I was playing with wasm32-wasip2 but that produced the same results in my first test (and I didn't really understand what it changed anyway) so I didn't explore it further. I'll see if I can get something working based on the build script you linked.

[alec-deason]

Thanks so much for your help. Using your comment and the build script I was able to get this working.