error when compiling to wasm32-unknown-unknown or wasm32-unknown-emscripten

[astrale-sharp]

I'm building a rust lib :

[lib]
crate-type = ["cdylib"]

with pyo3 in my dependencies : pyo3 = { version = "0.19.0", features = ["auto-initialize", "abi3-py39"] }

Trying to build with cross or cargo for wasm32-unknown-unknown gets me this error :

  error: your Rust target architecture (32-bit) does not match your python interpreter (64-bit)

Trying again with cross: cross b --target wasm32-unknown-emscripten with in my cargo.toml : pyo3 = { version = "0.19.0", features = ["auto-initialize", "abi3-py39"] } I get

error: failed to run custom build command for `pyo3-build-config v0.19.0`

Caused by:
  process didn't exit successfully: `/target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build` (exit status: 1)
  --- stderr
  /target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build)
  /target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build)
  /target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/debug/build/pyo3-build-config-6833074d719a3ea0/build-script-build)

I would prefer to build for wasm32-unknown-unknown but if someone helps me get this work with wasm32-unknown-emscripten I might settle for that ! I'm stuck and I tried more than I reported here, any tips?

[messense]

See https://github.com/PyO3/pyo3/discussions/2306#discussioncomment-2564793.

We have support for wasm32-unknown-emscripten target in maturin, for example: https://github.com/pydantic/pydantic-core/blob/c8e38786c83ecf09803026f2f7794dfa367d514d/.github/workflows/ci.yml#L229-L277

[astrale-sharp]

reading cross docs led me to run cargo clean before running cross which leads me back to this error :

error: cannot set a minimum Python version 3.9 higher than the interpreter version 3.8 (the minimum Python version is implied by the abi3-py39 feature)
warning: build failed, waiting for other jobs to finish...

[messense]

@astrale-sharp I'd recommend reading https://pyo3.rs/v0.19.0/building_and_distribution#cross-compiling to learn more about how to cross compile pyo3 projects.

[astrale-sharp]

after having tried different new things getting inspiration from previous commands I have a new error. things I tried : pip install 'maturin>=0.15,<0.16' 'black>=22.3.0,<23' typing_extensions pyo3 = { version = "0.19.0", features = ["auto-initialize","generate-import-lib","abi3-py38"] } because only python3.8 seems to be selected while building

I get this new error :

linking with `emcc` failed: exit status: 1
%%%%%wall of text%%%%
wasm-ld: error: unable to find library -lpython3.8
%%%%%wall of text%%%%
error: could not compile `py-wasm` (lib) due to previous error

I'll do more reading and try to figure out what is happening/what's missing

[messense]

maturin only support python3.10+ for emscripten target, because that's the version pyodide supports.

[astrale-sharp]

Then in this message error: cannot set a minimum Python version 3.10 higher than the interpreter version 3.8 (the minimum Python version is implied by the abi3-py310 feature)

I'd like to change my interpreter version to 3.10, should be feasible

[astrale-sharp]

Thanks a lot for the quick response and the pointers so far by the way !

[astrale-sharp]

PYO3_PYTHON=python3.10 cross b --target wasm32-unknown-emscripten doesn't seem to fix the problem

[astrale-sharp]

Moving back to cargo and away from cross PYO3_PYTHON=python3.10 cargo b --target wasm32-unknown-emscripten leads back to wasm-ld: error: unable to find library -lpython3.10 (I made sure to have python3.10-dev installed) and error: linking withemccfailed: exit status: 1

[messense]

with pyo3 in my dependencies : pyo3 = { version = "0.19.0", features = ["auto-initialize", "abi3-py39"] }

You have enabled auto-initialize feature so you need to provide a pre-compiled wasm32 python interpreter path to pyo3 via PYO3_CROSS_LIB_DIR env var.

[astrale-sharp]

Maybe I'll try disabling auto-initialize then cause I'm totally unsure how to get a pre-compiled wasm32 python interpreter.

if i try with PYO3_PYTHON=python3.11 cargo b --target wasm32-unknown-emscripten I get a new error as well.

thread 'main' panicked at 'Path does not exist: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /home/astrale/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-build-config-0.19.0/src/impl_.rs:1278:33

[astrale-sharp]

I'm also trying with features = ["generate-import-lib", ...]

[messense]

Have you tried PYO3_CROSS_PYTHON_VERSION as per https://pyo3.rs/v0.19.0/building_and_distribution#cross-compiling?

[astrale-sharp]

removing "auto-initialize" I still get thread 'main' panicked at 'Path does not exist: Os { code: 2, kind: NotFound, message: "No such file or directory" }'

I have not tried PYO3_CROSS_PYTHON_VERSION, yet, this will be the last try for today !

[astrale-sharp]

PYO3_CROSS_PYTHON_VERSION= 3.11 or 3.10 or 3.9 or 3.8 all fail with

thread 'main' panicked at 'Path does not exist: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /home/astrale/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-build-config-0.19.0/src/impl_.rs:1278:33

with features = ["generate-import-lib"]

I have to go now, thanks for the help !

[adamreichold]

with features = ["generate-import-lib"]

As per https://pyo3.rs/v0.19.0/building_and_distribution#cross-compiling, this feature is relevant only when compiling for Windows targets so you should not use it.

As a more fundamental question, if you do not have a Python interpreter for your WASM target, how do you intend to run the resulting binary which will need to load said interpreter?

If you do want to build in isolation without access to an interpreter, you need to set the environment variable PYO3_NO_PYTHON and enable one of the abi3-py* features as per the above linked documentation.

[astrale-sharp]

Trying to go with PYO3_PYTHON=python3.11 cargo b --target wasm32-unknown-emscripten or PYO3_NO_PYTHON=true c b --target wasm32-unknown-emscripten

and pyo3 = { version = "0.19.0", features = ["abi3-py311"] } and

[build-dependencies]
wlr-libpy = { git = "https://github.com/vmware-labs/webassembly-language-runtimes.git", features = ["build"] }

// build.rs
fn main() {
    use wlr_libpy::bld_cfg::configure_static_libs;
    configure_static_libs().unwrap().emit_link_flags();
}

to get access to libpython

I get linking error linking withemccfailed: exit status: 1 wasm-ld: error: duplicate symbol: pthread_mutex_init

, with latest(3.1.42) emscripten version and PYO3_PYTHON=python3.11 I get

  = note: wasm-ld: error: duplicate symbol: pthread_mutex_init
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_mutex_destroy
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_getspecific
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_setspecific
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_cond_wait
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_cond_signal
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_cond_init
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_cond_destroy
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_condattr_init
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)

          wasm-ld: error: duplicate symbol: pthread_condattr_setclock
          >>> defined in /home/astrale/Coding/rust/py-wasm-plugin/target/wasm32-wasi/wasi-deps/lib/wasm32-wasi/libpython3.11.a(thread.o)
          >>> defined in /home/astrale/Coding/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/libc-debug.a(library_pthread_stub.o)
          emcc: error: '/home/astrale/Coding/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_68mgykqb.rsp.utf-8' failed (returned 1)

warning: `py-wasm-plugin` (lib) generated 1 warning
error: could not compile `py-wasm-plugin` (lib) due to previous error; 1 warning emitted

with PYO3_NO_PYTHON=true I get something very similar

[adamreichold]

The project you are using says

It will download all needed pre-built static libraries for wasm32-wasi and configure the linker to use them.

but you are not targetting WASI, you are targetting Emscripten.

[astrale-sharp]

Thank you very much! it is finally compiling, i'll test without the build script to see if it still works!

while trying to load it with wasmer i have the following error now : wasi_snapshot_preview1"."clock_time_get" is there a way to fix it without actually importing anything?

[adamreichold]

Maybe https://stackoverflow.com/questions/68265851/cannot-execute-standalone-webassembly-file-with-wasmer? Maybe try Wasmtime as discussed there?

[astrale-sharp]

Well don't know what I changed exactly but it's not compiling anymore... PYO3_CROSS=true PYO3_NO_PYTHON=1 PYO3_CROSS_LIB_DIR=/usr/lib/x86_64-linux-gnu/ c b --target wasm32-wasi

I'm trying to give it my python lib but it fails with rust-lld: error: unknown file type: /usr/lib/x86_64-linux-gnu/libc.a(errno.o)

it seems it doesn't recognize my lpython

[astrale-sharp]

turns out the build script was in a weird state and that's why it didn't compile but I got it working again, still interested by why it didn't like my libpython if someone has a clue

[adamreichold]

PYO3_CROSS_LIB_DIR=/usr/lib/x86_64-linux-gnu/

Why did you point it to a Python library for x86-64 if you are building for WASM?

[astrale-sharp]

Anyway I finally understand that pyo3 is about accessing a python interpreter and that you can't (tell me if i'm wrong) bundle an interpreter in your wasm to make it work anywhere.

[davidhewitt]

This isn't something that we've attempted to support but there's no reason that we shouldn't be able to. I don't have firsthand experience trying to do that, e.g. it's also been proposed at https://github.com/indygreg/PyOxidizer/issues/507 - just needs some collaboration to work it out.

[davidhewitt]

I'm closing this as it's a long and winding thread with the original question resolved. The follow-up about packaging a Python interpreter is a separate question which is probably best solved by PyOxidizer.