search

RustPython / Parser

MIT License
66 stars 24 forks source link

Error building wheel for rustpython_ast (pyo3) #115

Closed pawamoy closed 3 months ago

pawamoy commented 3 months ago

I cloned the repository, entered the ast-pyo3 directory, and ran pyproject-build: 

% pyproject-build 
* Creating venv isolated environment...
* Installing packages in isolated environment... (maturin>=0.15,<0.16)
* Getting build dependencies for sdist...
* Building sdist...
Running `maturin pep517 write-sdist --sdist-directory /media/data/dev/griffe/rustpython_parser/ast-pyo3/dist`
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at /tmp/build-env-m11693t3/bin/python3
📡 Using build options features from pyproject.toml
📦 Built source distribution to /media/data/dev/griffe/rustpython_parser/ast-pyo3/dist/rustpython_ast-0.0.1.tar.gz
rustpython_ast-0.0.1.tar.gz
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (maturin>=0.15,<0.16)
* Getting build dependencies for wheel...
* Building wheel...
Running `maturin pep517 build-wheel -i /tmp/build-env-ra719lal/bin/python --compatibility off`
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at /tmp/build-env-ra719lal/bin/python
📡 Using build options features from pyproject.toml
   Compiling autocfg v1.1.0
   Compiling libc v0.2.153
   Compiling proc-macro2 v1.0.79
   Compiling cfg-if v1.0.0
   Compiling unicode-ident v1.0.12
   Compiling target-lexicon v0.12.14
   Compiling version_check v0.9.4
   Compiling once_cell v1.19.0
   Compiling ppv-lite86 v0.2.17
   Compiling ahash v0.8.11
   Compiling siphasher v0.3.11
   Compiling num-traits v0.2.18
   Compiling phf_shared v0.11.2
   Compiling libm v0.2.8
   Compiling zerocopy v0.7.32
   Compiling syn v1.0.109
   Compiling pyo3-build-config v0.19.2
   Compiling getrandom v0.2.12
   Compiling quote v1.0.35
   Compiling rand_core v0.6.4
   Compiling either v1.10.0
   Compiling hashbrown v0.14.3
   Compiling rand_chacha v0.3.1
   Compiling itertools v0.11.0
   Compiling rand v0.8.5
   Compiling ryu v1.0.17
   Compiling phf_generator v0.11.2
   Compiling phf_codegen v0.11.2
   Compiling unicode-width v0.1.11
   Compiling crunchy v0.2.2
   Compiling getopts v0.2.21
   Compiling malachite-base v0.4.5
   Compiling num-integer v0.1.46
   Compiling num-bigint v0.4.4
   Compiling anyhow v1.0.81
   Compiling log v0.4.21
   Compiling tiny-keccak v2.0.2
   Compiling paste v1.0.14
   Compiling unicode_names2_generator v1.2.2
   Compiling pyo3-ffi v0.19.2
   Compiling syn v2.0.53
   Compiling lock_api v0.4.11
   Compiling parking_lot_core v0.9.9
   Compiling unic-char-range v0.9.0
   Compiling unic-common v0.9.0
   Compiling memchr v2.7.1
   Compiling Inflector v0.11.4
   Compiling convert_case v0.4.0
   Compiling unicode_names2 v1.2.2
   Compiling rustpython-parser-vendored v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-parser-vendored)
   Compiling unic-ucd-version v0.9.0
   Compiling unic-char-property v0.9.0
   Compiling memoffset v0.9.0
   Compiling smallvec v1.13.2
   Compiling scopeguard v1.2.0
   Compiling derive_more v0.99.17
   Compiling rustpython-parser v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-parser)
   Compiling phf v0.11.2
   Compiling pyo3-macros-backend v0.19.2
   Compiling pyo3 v0.19.2
   Compiling malachite-nz v0.4.5
   Compiling is-macro v0.3.5
   Compiling static_assertions v1.1.0
   Compiling rustpython-parser-core v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-parser-core)
   Compiling parking_lot v0.12.1
   Compiling pyo3-macros v0.19.2
   Compiling unic-emoji-char v0.9.0
   Compiling unic-ucd-ident v0.9.0
   Compiling num-complex v0.4.5
   Compiling unindent v0.1.11
   Compiling lalrpop-util v0.20.2
   Compiling indoc v1.0.9
   Compiling rustc-hash v1.1.0
   Compiling malachite-q v0.4.5
   Compiling malachite v0.4.5
   Compiling malachite-bigint v0.2.0
   Compiling rustpython-ast v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-ast)
error[E0252]: the name `bigint` is defined multiple times
  --> local_dependencies/rustpython-ast/src/lib.rs:28:9
   |
26 | pub use malachite_bigint as bigint;
   |         -------------------------- previous import of the module `bigint` here
27 | #[cfg(feature = "num-bigint")]
28 | pub use num_bigint as bigint;
   |         ^^^^^^^^^^^^^^^^^^^^ `bigint` reimported here
   |
   = note: `bigint` must be defined only once in the type namespace of this module
help: you can use `as` to change the binding name of the import
   |
28 | pub use num_bigint as other_bigint;
   |         ~~~~~~~~~~~~~~~~~~~~~~~~~~

warning: unused import: `num_bigint as bigint`
  --> local_dependencies/rustpython-ast/src/lib.rs:28:9
   |
28 | pub use num_bigint as bigint;
   |         ^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

For more information about this error, try `rustc --explain E0252`.
warning: `rustpython-ast` (lib) generated 1 warning
error: could not compile `rustpython-ast` (lib) due to 1 previous error; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/tmp/build-env-ra719lal/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/build-env-ra719lal/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/Cargo.toml" "--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/build-env-ra719lal/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

ERROR Backend subprocess exited when trying to invoke build_wheel

Any hint on how to fix that :slightly_smiling_face:?

As to why I'm trying to build it, well, I'd like to try it in a project of mine to see how well it performs and how much work it would need to use it instead of Python's ast.

Quick question by the way: it builds an Abstract Syntax Tree and not a Concrete Syntax Tree right?

youknowone commented 3 months ago

Hi, welcome to ast pyo3 support. I usually built it with maturin, like maturin develop for dev version. I am not very familiar with this ecosystem. Is pyo3 projects usually expected to be built with pyproject-build?

pawamoy commented 3 months ago

Thanks for your help!

Is pyo3 projects usually expected to be built with pyproject-build?

pyproject-build is a build frontend for Python projects. It reads pyproject.toml and uses the specified build-system metadata to build sdists and wheels for Python projects. In your case, pyproject.toml specifies maturin>=0.15,<0.16 as the build-backend, so pyproject-build uses maturin to build this project.

After your comment, I tried building with maturin develop directly, and it worked. I thought removing the cap to 0.16 in the build-system metadata would fix it (latest is 1.5.1), but no. pyproject-build apparently runs the following command:

maturin pep517 build-wheel -i /tmp/build-env-3os4z0xe/bin/python --compatibility off

(/tmp/build-env-3os4z0xe/bin/python is a temporary virtualenv)

Full logs:

% pyproject-build 
* Creating venv isolated environment...
* Installing packages in isolated environment... (maturin>=0.15)
* Getting build dependencies for sdist...
* Building sdist...
Running `maturin pep517 write-sdist --sdist-directory /media/data/dev/rustpython-parser/ast-pyo3/dist`
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at /tmp/build-env-wrgp8mbb/bin/python3
📡 Using build options features from pyproject.toml
📦 Built source distribution to /media/data/dev/rustpython-parser/ast-pyo3/dist/rustpython_ast-0.0.1.tar.gz
rustpython_ast-0.0.1.tar.gz
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (maturin>=0.15)
* Getting build dependencies for wheel...
* Building wheel...
Running `maturin pep517 build-wheel -i /tmp/build-env-3os4z0xe/bin/python --compatibility off`
error: failed to load manifest for workspace member `/tmp/build-via-sdist-f2sr0kj1/rustpython_ast-0.0.1/ast-pyo3`

Caused by:
  failed to load manifest for dependency `rustpython-ast`

Caused by:
  failed to load manifest for dependency `rustpython-literal`

Caused by:
  failed to read `/tmp/build-via-sdist-f2sr0kj1/rustpython_ast-0.0.1/literal/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
💥 maturin failed
  Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
  Caused by: `cargo metadata` exited with an error: 
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/build-env-3os4z0xe/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

ERROR Backend subprocess exited when trying to invoke build_wheel

I'm not familiar with maturin so I don't know how to make maturin pep517 build-wheel work like maturin develop.

pawamoy commented 3 months ago

Weirdly enough if I use this command with maturin 1.5.1 directly, it works...

% maturin pep517 build-wheel --compatibility off
⚠️  Warning: You specified maturin >=0.15, <0.16 in pyproject.toml under `build-system.requires`, but the current maturin version is 1.5.1
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at /home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.11.5/bin/python3
📡 Using build options features from pyproject.toml
   Compiling rustpython-ast-pyo3 v0.0.1 (/media/data/dev/rustpython-parser/ast-pyo3)
    Finished release [optimized] target(s) in 7.03s
📦 Built wheel for CPython 3.11 to /media/data/dev/rustpython-parser/target/wheels/rustpython_ast-0.0.1-cp311-cp311-linux_x86_64.whl
/media/data/dev/rustpython-parser/target/wheels/rustpython_ast-0.0.1-cp311-cp311-linux_x86_64.whl
youknowone commented 3 months ago

This crate now placed in https://github.com/RustPython/rustpython-ast-pyo3 This issue seems to be happened with parsers default bigint feature malachite. I fixed this issue in the new repository.

pawamoy commented 3 months ago

Thanks @youknowone, I'll try the new repository and report back :slightly_smiling_face:

pawamoy commented 3 months ago

Yay that worked! Thanks a lot!