Usage in PyO3 extension module

[bminixhofer]

Hi! Awesome project. I'm currently working on Python bindings for a Rust library. I occasionally have to call small Python snippets from the Rust code. This library would be a very elegant solution to that.

However, when I try to compile inline-python with the feature "extension-module" in PyO3 enabled I get an error:

error: /home/bminixhofer/Documents/string_sum/target/debug/deps/libinline_python_macros-2b68321f767fc207.so: undefined symbol: PyTuple_SetItem
   --> /home/bminixhofer/.cargo/registry/src/github.com-1ecc6299db9ec823/inline-python-0.5.1/src/lib.rs:143:9
    |
143 | pub use inline_python_macros::python;
    |         ^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: could not compile `inline-python`.

Caused by:
  process didn't exit successfully: `rustc --crate-name inline_python --edition=2018 /home/bminixhofer/.cargo/registry/src/github.com-1ecc6299db9ec823/inline-python-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -Cbitcode-in-rlib=no -C debuginfo=2 -C metadata=6055137a725d9ef2 -C extra-filename=-6055137a725d9ef2 --out-dir /home/bminixhofer/Documents/string_sum/target/debug/deps -L dependency=/home/bminixhofer/Documents/string_sum/target/debug/deps --extern inline_python_macros=/home/bminixhofer/Documents/string_sum/target/debug/deps/libinline_python_macros-2b68321f767fc207.so --extern pyo3=/home/bminixhofer/Documents/string_sum/target/debug/deps/libpyo3-df6fd04a55a3c6e8.rmeta --cap-lints allow` (exit code: 1)

To reproduce

Cargo.toml

[package]
name = "string-sum"
version = "0.1.0"
edition = "2018"
authors = [""]

[lib]
name = "string_sum"
crate-type = ["cdylib"]

[dependencies]
inline-python = "0.5.1"

[dependencies.pyo3]
version = "0.9.0"
features = ["extension-module"]

src/lib.rs

#![feature(proc_macro_hygiene)]
use inline_python::python;

use inline_python::pyo3::prelude::*;
use inline_python::pyo3::wrap_pyfunction;

#[pyfunction]
/// Formats the sum of two numbers as string.
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    python! {
        print("Hello!")
    }

    Ok((a + b).to_string())
}

#[pymodule]
/// A Python module implemented in Rust.
fn string_sum(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(sum_as_string))?;

    Ok(())
}

Rust version: rustc 1.45.0-nightly (fa51f810e 2020-04-29)

Thanks for any help!

[m-ou-se]

Interesting!

It seems that enabling feature = "extension-module" is not a strict addition to the pyo3 crate, but will disable some linking options (cargo:rustc-link-search=native=...).

Cargo automatically takes the union of all requested features of a crate through the whole dependency graph, which includes the procedural macros. This means that feature = "extension-module" will also be enabled for inline-python-macros, which uses PyO3 in a proc macro to compile the Python code at (Rust) compile time. That crate is not an extension module, as it will not be loaded by Python, but by the Rust compiler. That fails.

I don't know much about how this works on Windows and Mac, but on Linux it works fine to make an extension module without extension-module enabled.

[m-ou-se]

See https://github.com/PyO3/pyo3/issues/904

[bminixhofer]

Thanks for the quick response!

The follow up is a bit beyond my expertise, but I agree that this is an issue with PyO3 as features should be strictly additive. There has in fact already been discussion about this in https://github.com/PyO3/pyo3/issues/771.

Additionally, from PyO3s Cargo.toml:

# Use this feature when building an extension module.
# It tells the linker to keep the python symbols unresolved,
# so that the module can also be used with statically linked python interpreters.
extension-module = []

it seems that extension-module is only required for statically linked Python interpreters, which I am fine with not supporting for now.

IMO it is fine to close this issue.

But I would consider adding a small note to the How to Use section in the Readme indicating that the extension-module feature must not be enabled for inline-python to work. Otherwise this could be confusing, especially because the error message is rather cryptic.

[Jules-Bertholet]

Does the new Cargo feature resolver solve this issue (by no longer unifying proc-macro and build dependency features with "run-time" dependencies)?