search

PyO3 / maturin

Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages
https://maturin.rs
Apache License 2.0
3.69k stars 253 forks source link

maturin develop builds for mixed python versions #2198

Open rumpelsepp opened 2 weeks ago

rumpelsepp commented 2 weeks ago

Bug Description

The bug on my machine is that maturin builds for a faulty combination of python versions.

$ tree .venv/lib/                                                                                                                                                                        .venv/lib/                                                                                                                                                                                                                                      
└── python3.11   <--- Python 3.11
    └── site-packages
        ├── […]
        ├── my_package
        │   ├── __init__.py
        │   └── my_package.cpython-312-x86_64-linux-gnu.so   <--- Python 3.12
        ├── […]
        ├── _virtualenv.pth
        └── _virtualenv.py

I have multiple Python versions on my system.

$ uv python list
cpython-3.12.5-linux-x86_64-gnu     <download available>
cpython-3.12.4-linux-x86_64-gnu     /nix/store/04gg5w1s662l329a8kh9xcwyp0k64v5a-python3-3.12.4/bin/python3.12
cpython-3.12.4-linux-x86_64-gnu     /nix/store/04gg5w1s662l329a8kh9xcwyp0k64v5a-python3-3.12.4/bin/python3 -> python3.12
cpython-3.12.4-linux-x86_64-gnu     /nix/store/04gg5w1s662l329a8kh9xcwyp0k64v5a-python3-3.12.4/bin/python -> python3.12
cpython-3.11.9-linux-x86_64-gnu     /nix/store/53kwxjyrlpj47nhv6w8hv4kaa8x6rmka-python3-3.11.9/bin/python3.11
cpython-3.11.9-linux-x86_64-gnu     /nix/store/53kwxjyrlpj47nhv6w8hv4kaa8x6rmka-python3-3.11.9/bin/python3 -> python3.11
cpython-3.11.9-linux-x86_64-gnu     /nix/store/53kwxjyrlpj47nhv6w8hv4kaa8x6rmka-python3-3.11.9/bin/python -> python3.11
cpython-3.11.9-linux-x86_64-gnu     <download available>
cpython-3.11.2-linux-x86_64-gnu     /usr/bin/python3.11
cpython-3.11.2-linux-x86_64-gnu     /usr/bin/python3 -> python3.11
cpython-3.11.2-linux-x86_64-gnu     /bin/python3.11
cpython-3.11.2-linux-x86_64-gnu     /bin/python3 -> python3.11
cpython-3.10.14-linux-x86_64-gnu    <download available>
cpython-3.9.19-linux-x86_64-gnu     <download available>
cpython-3.8.19-linux-x86_64-gnu     <download available>
pypy-3.7.13-linux-x86_64-gnu        <download available>

I have pinned it to 3.11: 

$ cat .python-version
3.11

Your maturin version (maturin --version)

maturin 1.7.0

Your Python version (python -V)

Python 3.11.9

Your pip version (pip -V)

pip 24.0 from /nix/store/z5xf4y7v8b5g31x0cxj683x38n5gakam-python3.12-pip-24.0/lib/python3.12/site-packages/pip (python 3.11)

What bindings you're using

pyo3

Does cargo build work?

Steps to Reproduce

$ RUST_LOG=maturin=debug maturin develop --uv                                                                                                                                            
2024-08-27T08:36:12.435637Z DEBUG into_build_context: maturin::project_layout: Found pyproject.toml in working directory at "/home/steff/Projects/private/penlog/pyproject.toml"                                                                
2024-08-27T08:36:12.435799Z DEBUG into_build_context: maturin::project_layout: Using cargo manifest path from working directory: "/home/steff/Projects/private/penlog/Cargo.toml"                                                               
2024-08-27T08:36:12.435878Z DEBUG into_build_context:resolve_cargo_metadata: maturin::project_layout: Resolving cargo metadata from "/home/steff/Projects/private/penlog/Cargo.toml"                                                            
2024-08-27T08:36:12.472807Z  INFO into_build_context:resolve_cargo_metadata: maturin::project_layout: close time.busy=36.9ms time.idle=1.40µs                                                                                                   
📦 Including license file "/home/steff/Projects/private/penlog/LICENSE"                                                                                                                                                                         
2024-08-27T08:36:12.473001Z DEBUG into_build_context: maturin::project_layout: Project layout resolved project_root=/home/steff/Projects/private/penlog python_dir=/home/steff/Projects/private/penlog rust_module=/home/steff/Projects/private/
penlog/penlog python_module=/home/steff/Projects/private/penlog/penlog extension_name=penlog module_name=penlog                                                                                                                                 
🔗 Found pyo3 bindings                                                                                                                                                                                                                          
2024-08-27T08:36:12.530416Z DEBUG into_build_context:check_executable: maturin::python_interpreter: Found CPython interpreter at /home/steff/Projects/private/penlog/.venv/bin/python executable=/home/steff/Projects/private/penlog/.venv/bin/p
ython                                                                                                                                                                                                                                           
2024-08-27T08:36:12.530437Z  INFO into_build_context:check_executable: maturin::python_interpreter: close time.busy=23.4ms time.idle=2.04µs executable=/home/steff/Projects/private/penlog/.venv/bin/python                                     
🐍 Found CPython 3.11 at /home/steff/Projects/private/penlog/.venv/bin/python                                                                                                                                                                   
📡 Using build options features from pyproject.toml                                                                                                                                                                                             
2024-08-27T08:36:12.530455Z  INFO into_build_context: maturin::build_options: close time.busy=94.8ms time.idle=2.88µs                                                                                                                           
2024-08-27T08:36:12.553541Z DEBUG check_executable: maturin::python_interpreter: Found CPython interpreter at /home/steff/Projects/private/penlog/.venv/bin/python executable=/home/steff/Projects/private/penlog/.venv/bin/python              
2024-08-27T08:36:12.553559Z  INFO check_executable: maturin::python_interpreter: close time.busy=23.1ms time.idle=1.25µs executable=/home/steff/Projects/private/penlog/.venv/bin/python                                                        
2024-08-27T08:36:12.578320Z DEBUG maturin::develop: Found uv binary in PATH version=uv 0.3.1

2024-08-27T08:36:12.578340Z  INFO install_dependencies: maturin::develop: close time.busy=1.46µs time.idle=1.46µs
2024-08-27T08:36:12.578452Z DEBUG build_wheels: maturin::compile: Setting PYO3_PYTHON to /home/steff/Projects/private/penlog/.venv/bin/python
2024-08-27T08:36:12.578458Z DEBUG build_wheels: maturin::compile: Running env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/home/steff/Projects/private/penlog/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/home/steff/Projects/private/penlog/.venv/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/home/steff/Projects/private/penlog/Cargo.toml" "--lib"
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s
2024-08-27T08:36:12.624348Z  INFO build_wheels:warn_missing_py_init: maturin::compile: close time.busy=5.85ms time.idle=2.15µs
2024-08-27T08:36:12.627097Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/METADATA
2024-08-27T08:36:12.627217Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/WHEEL
2024-08-27T08:36:12.627306Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/license_files/LICENSE from /home/steff/Projects/private/penlog/LICENSE
2024-08-27T08:36:12.627766Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding penlog/__init__.py
2024-08-27T08:36:12.627910Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding penlog/penlog.cpython-312-x86_64-linux-gnu.so from /home/steff/Projects/private/penlog/target/debug/libpenlog.so
2024-08-27T08:36:12.922780Z  INFO build_wheels:write_bindings_module: maturin::module_writer: close time.busy=295ms time.idle=1.99µs
2024-08-27T08:36:12.922806Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/RECORD
📦 Built wheel for CPython 3.11 to /tmp/nix-shell.wDHYOK/.tmp2POqRu/penlog-0.1.0-cp311-cp311-linux_x86_64.whl
2024-08-27T08:36:12.923395Z  INFO build_wheels: maturin::build_context: close time.busy=345ms time.idle=651ns
✏️   Setting installed package as editable                                                                              
2024-08-27T08:36:13.528730Z  INFO pip_install_wheel:fix_direct_url: maturin::develop: close time.busy=536ms time.idle=2.18µs wheel_filename=/tmp/nix-shell.wDHYOK/.tmp2POqRu/penlog-0.1.0-cp311-cp311-linux_x86_64.whl
2024-08-27T08:36:13.528749Z  INFO pip_install_wheel: maturin::develop: close time.busy=605ms time.idle=971ns wheel_filename=/tmp/nix-shell.wDHYOK/.tmp2POqRu/penlog-0.1.0-cp311-cp311-linux_x86_64.whl
🛠 Installed penlog-0.1.0

The error is here:

2024-08-27T08:36:12.627910Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding penlog/penlog.cpython-312-x86_64-linux-gnu.so from /home/steff/Projects/private/penlog/target/debug/libpenlog.so

For reasons in this line there is the wrong python version.

messense commented 2 weeks ago

That's odd, could you play around with maturin source code (use cargo run develop) to debug it? I'm asking because it might not be easy to reproduce it elsewhere. Thanks!

https://github.com/PyO3/maturin/blob/a40f720482b7dcf7ce7eec644d67cf5a719df5c2/src/module_writer.rs#L807-L824