PyO3 / maturin

Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages
https://maturin.rs
Apache License 2.0
3.91k stars 271 forks source link

With a Mixed Python/Rust project, only Rust functions appear in Python Wheel. #1649

Closed DanielChaseButterfield closed 1 year ago

DanielChaseButterfield commented 1 year ago

Bug Description

This may seem like a duplicate of #885, but that uses a different version of Maturin, and the solution for that thread is not the solution for my issue. I am trying to build into a conda virtual environment in WSL, and my pip wheel that is created by running maturin develop does not have any of my python functions or classes in it, just the functions defined by the Rust module. I have a mixed Python/Rust project with the following structure:

actag
├── Cargo.toml
├── python
│   └── actag
│       ├── __init__.py
│       ├── actag_src.py
│       ├── adaptive_threshold.py
│       ├── contour_identification.py
│       ├── median_filter.py
│       ├── quad_fitting.py
│       ├── sonar_parameters.py
│       ├── tag_decoding.py
│       └── tag_parameters.py
├── pyproject.toml
├── README.md
└── src
    ├── contours.py
    ├── filters.rs
    ├── lib.rs
    ├── quad_fitting.rs
    ├── tag_detection.rs
    └── tag_parameters.rs

My pyproject.toml contains:

[build-system]
requires = ["maturin>=0.13,<0.14"]
build-backend = "maturin"

[tool.maturin]
python-source = "python"

[project]
name = "actag"
requires-python = ">=3.9"
classifiers = [
    "Programming Language :: Rust",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]

My Cargo.toml contains:

[package]
name = "actag"
version = "1.0.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "actag"
crate-type = ["cdylib", "lib"]

[dependencies]
ndarray = "0.15.4"
num = "0.4"
libm = "0.1.4"
geo = "0.22.1"
num_cpus = "1.13.1"
crossbeam = "0.8"
indicatif = "0.17.0"
itertools-num = "0.1.3"
nalgebra = "0.31.1"
lstsq = "0.4.0"
float-cmp = "0.9.0"
randomkit = "0.1"
substring = "1.4.5"
rug = "=1.18.0"

[dev-dependencies]
criterion = "0.3"

[[bench]]
name = "my_benchmark"
harness = false

[dependencies.pyo3]
version = "0.16.5"

[features]
abi3-py37 = ["pyo3/abi3-py37"]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

My init.py file contains:

from .actag_src import AcTag

__doc__ = actag_src.__doc__
if hasattr(actag_src, "__all__"):
    __all__ = actag_src.__all__

My actag_src.py file contains an AcTag class. It also imports the actag module generated by the Rust side of the code.

With this setup, I can successfully build the pip wheel and install it with maturin develop, but I get the following error when I try to use the AcTag class from the python code:

Traceback (most recent call last):
  File "/home/dlittleman/actag/test.py", line 5, in <module>
    detector = actag.AcTag(min_range=0.1, max_range=1.5, horizontal_aperture=1.0472, tag_family="AcTag24h10", 
AttributeError: module 'actag' has no attribute 'AcTag'

However, I can see all of the Rust functions when I run print(dir(actag)):

['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'actag', 'adaptive_threshold', 'adaptive_threshold_multithread', 'bilateral_filter', 'bilateral_filter_multithread', 'check_quad_for_tag_python_wrap', 'convert_tag_detections_to_range_azi_locations', 'decode_tags', 'fit_quadrilaterals_to_contours', 'get_contours', 'get_data_bit_locations', 'get_intersection_of_lines', 'get_random_point_and_fit_line_python_wrap', 'least_squares_line_fit_python_wrap', 'median_filter', 'median_filter_multithread', 'parse_apriltag_family', 'plot_contours']

Your maturin version (maturin --version)

0.13.0

Your Python version (python -V)

3.9.16

Your pip version (pip -V)

23.0.1

What bindings you're using

pyo3

Does cargo build work?

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

Steps to Reproduce

  1. Install Conda and Initialize a conda environment: conda create -n tester python=3.9
  2. Activate the environment: conda activate tester
  3. Install maturin: pip install maturin
  4. Download the following repository: https://github.com/DanielChaseButterfield/maturinIssueDemonstration
  5. Run maturin develop --release from the main directory of the repository.
  6. Create a test.py file somewhere else with the following file contents:
    
    import actag

detector = actag.AcTag(min_range=0.1, max_range=1.5, horizontal_aperture=1.0472, tag_family="AcTag24h10", tag_size=0.130628571428644, median_filter_kernel_radius=4, adaptive_threshold_kernel_radius=8, adaptive_threshold_offset=1, quads_use_same_random_vals=True)


7. Run ``python test.py``
messense commented 1 year ago

Have you tried maturin 1.0 yet? No bugfixes will be backported to the old 0.13 versions.

DanielChaseButterfield commented 1 year ago

I didn't know maturin 1.0 existed! I've now updated, and I can now see my phython functions and classes. However, now I can't see my Rust functions. This is what print(dir(actag)) returns now:

['AcTag', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'actag_src', 'adaptive_threshold', 'contour_identification', 'median_filter', 'quad_fitting', 'sonar_parameters', 'tag_decoding', 'tag_parameters']

Am I correct in assuming that both the python and rust functions should be available?

messense commented 1 year ago

No, Rust functions/classes are contained in actag (the name of your Rust extension module) module, you can re-export it to top level module if you want.

messense commented 1 year ago

Please refer to https://www.maturin.rs/project_layout.html#mixed-rustpython-project for more information.

DanielChaseButterfield commented 1 year ago

Okay, thank you! I got it working by adding this to the init.py file:

import actag as actag_rust