`dyld: Symbol not found: _PyExc_SystemError` in tests if there is `#[pymethods]` in project

deaz commented 5 years ago

Tests fail with error when there are methods in #[pymethods] impl for struct:

Compiling pyo3-test v0.1.0 (/Users/ivan/pyo3-test)
    Finished dev [unoptimized + debuginfo] target(s) in 1.10s
     Running target/debug/deps/pyo3_test-34e869060858761f
dyld: Symbol not found: _PyExc_SystemError
  Referenced from: /Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f
  Expected in: flat namespace
 in /Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f
error: process didn't exit successfully: `/Users/ivan/pyo3-test/target/debug/deps/pyo3_test-34e869060858761f` (signal: 6, SIGABRT: process abort signal)

🌍 Environment

Your operating system and version: macOS 10.14.2 (18C54)
Your python version: 3.7.2
How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: i don't remeber, probably brew install python3. I used virtualenv via pipenv
Your rust version (rustc --version): 1.34.0-nightly (8ae730a44 2019-02-04)
Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")? I use 0.6.0-alpha.2. With latest master error is the same.

💥 Reproducing

Run cargo test for the following code:

use pyo3::prelude::*;

#[pyclass]
struct Class {}

#[pymethods]
impl Class {
    #[new]
    fn __new__(obj: &PyRawObject) -> PyResult<()> {
        obj.init(|| Class {})
    }
}

#[pymodule]
fn pyo3_test(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<Class>()
}

#[cfg(test)]
mod tests {
    #[test]
    fn test() {}
}

Also i created minimal project for reproducing the issue: https://github.com/deaz/pyo3-test

kngwyu commented 5 years ago

Thanks for reporting! Basically we need

[dependencies.pyo3]
features = []

to test project. For extension projects(i.e. projects with features = ["extension-module"]), you can use setuptools-rust to run test. @konstin Do you know any workaround other than that?

deaz commented 5 years ago

As workaround I added #[cfg(not(test))] to my struct and impl and it is working fine

konstin commented 5 years ago

First of all, thank you for the test repository!

As @kngwyu said, we need to deactivate features for the tests. A proper solution for this requires https://github.com/rust-lang/cargo/issues/2911, which unfortunately is unlikely to be implemented soon.

For now, you can use the following as workaround:

[package]
name = "pyo3-test"
version = "0.1.0"
authors = ["Ivan Vologdin <vologdin.nsk@gmail.com>"]
edition = "2018"

[lib]
name = "pyo3_test"
crate-type = ["cdylib"]

[dependencies.pyo3]
version = "0.6.0-alpha.2"

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

Tests can then be run with cargo test --no-default-features, while cargo build still works as expected.

With a bit context this workaround would be very useful in the guide.

deaz commented 5 years ago

Thanks for help!

boredstiff commented 5 years ago

Just ran into this issue - thank you both for posting really great instructions on how to work around this.

konstin commented 5 years ago

BTW this is also in the guide: https://pyo3.rs/v0.7.0-alpha.1/advanced.html#testing

liquidscorpio commented 5 years ago

The work-around mentioned in https://github.com/PyO3/pyo3/issues/340#issuecomment-461514532 does not work if the test is run on a workspace where one of member crates uses PyO3.

mvaled commented 4 years ago

I'm getting this error as well, however I'm not using #[pymethods] yet. My entire Rust code now is just:

use pyo3::prelude::*;

#[pymodule]
fn travertine(_py: Python, _m: &PyModule) -> PyResult<()> {
    Ok(())
}

#[cfg(test)]
mod test {
    use super::*;
    use pyo3::types::{IntoPyDict, PyDateTime};

    #[test]
    fn pydelta_conversion() {
        let gil = Python::acquire_gil();
        let py = gil.python();
        let datetime = py.import("datetime").unwrap();
        let locals = [("datetime", datetime)].into_py_dict(py);
        let now: &PyDateTime = py
            .eval("datetime.datetime.utcnow()", None, Some(&locals))
            .unwrap()
            .downcast()
            .unwrap();
        println!("{:?}", now);
    }
}

That suffices to trigger the error. The entire repository is in https://github.com/mvaled/pyo3bug340

The issue is reproducible with stable version of pyo3.

davidhewitt commented 4 years ago

Yes. TBH this issue is (I believe) really a dupe of #771

gilescope commented 4 years ago

Maybe we're going around this the wrong way, maybe when maturin builds it should add in --features=pyo3/extension-module because it knows it's required. That way the cargo test would just work.

We can already do:

maturin develop --cargo-extra-args="--features pyo3/extension-module"

Why not add that flag in automatically if it's clear that it's not there? Having pyo3/extension-module turned on as a default feature seems like a footgun as people will just run cargo test and expect stuff to work on cli, IDEs etc.

I've posted this in the maturin issues list here: https://github.com/PyO3/maturin/issues/325

attack68 commented 8 months ago

I just ran into this problem also. I had quite an extensive project which was working fine with cargo test --lib. A few commits later and the cc linker issue came up on my both my different Macs. The changes made did not seem to be that significant, and certainly not as obvious as the original post assumes some attachment of [pymethods]: I had these already working well so this was not the cause.

Rolling back removed the issue and I tried to remake the changes to identify the cause but couldn't unfortunately.

This was particularly confusing after appearing on a reasonably developed project and apparently after rather innocuous changes.

alex commented 8 months ago

Please see https://pyo3.rs/v0.21.0/faq#i-cant-run-cargo-test-or-i-cant-build-in-a-cargo-workspace-im-having-linker-issues-like-symbol-not-found-or-undefined-reference-to-_pyexc_systemerror

On Tue, Mar 26, 2024 at 2:22 AM JHM Darbyshire @.***> wrote:

I just ran into this problem also. I had quite an extensive project which was working fine with cargo test --lib. A few commits later and the cc linker issue came up on my both my different Macs. The changes made did not seem to be that significant, and certainly not as obvious as the original post assumes some attachment of [pymethods]: I had these already working well so this was not the cause.

Rolling back removed the issue and I tried to remake the changes to identify the cause but couldn't unfortunately.

This was particularly confusing after appearing on a reasonably developed project and apparently after rather innocuous changes.

— Reply to this email directly, view it on GitHub https://github.com/PyO3/pyo3/issues/340#issuecomment-2019487812, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBBI6JT4FQ77L5OKO7DY2EA2JAVCNFSM4GUTIVYKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBRHE2DQNZYGEZA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

glevco commented 6 months ago

@davidhewitt @alex I've been reading about this problem in the FAQ and related issues, and couldn't find any mentions or examples of running PyO3 code from a Rust test in a PyO3 lib.

For reference, I created a new hello world project with the exact steps from the Getting started guide, then added this to Cargo.toml (as suggested in the FAQ):

[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]

And then added this simple test to the lib.rs file:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

Running cargo tests works¹. If we uncomment the first line of the test, it fails with cc linking issues. I tried every combination suggested in the FAQ but couldn't get this to work. Do you know if it's possible?

¹ This is also weird by the way, because according to the FAQ, for the configuration I set above, I should call cargo test --no-default-features instead of cargo test for it to work. But in fact, the opposite happens. I wonder if the docs are outdated?

anuradhawick commented 3 months ago

@glevco good catch! There may be some inconsistency in docs I guess. Also the flag seems to be required for cargo build and cargo build --release not for testing. May be testing takes place in the default feature, while we need the pyo3/extension-module for the actual build.

rnag commented 3 weeks ago

@davidhewitt @alex I've been reading about this problem in the FAQ and related issues, and couldn't find any mentions or examples of running PyO3 code from a Rust test in a PyO3 lib.

For reference, I created a new hello world project with the exact steps from the Getting started guide, then added this to Cargo.toml (as suggested in the FAQ):
[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]
And then added this simple test to the lib.rs file:
#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        // pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}
Running cargo tests works1. If we uncomment the first line of the test, it fails with cc linking issues. I tried every combination suggested in the FAQ but couldn't get this to work. Do you know if it's possible?

1 This is also weird by the way, because according to the FAQ, for the configuration I set above, I should call cargo test --no-default-features instead of cargo test for it to work. But in fact, the opposite happens. I wonder if the docs are outdated?

I can't reproduce. The same code works for me. Except I had to add pyo3::prepare_freethreaded_python() in the new versions of pyo3.

My full code:

#[cfg(test)]
mod tests {
    #[test]
    fn test() {
        pyo3::prepare_freethreaded_python();
        pyo3::Python::with_gil(|py| {}); // uncommenting this breaks tests
        assert_eq!(1, 1);
    }
}

VS Code runs it like:

cargo test --package <mod> --lib --no-default-features -- path::tests::test --exact --show-output

By the way, to get that to work I added this in my .vscode/settings.json:

{
    "rust-analyzer.runnables.extraArgs": [
        "--no-default-features"
    ]
}

My cargo.toml is looking like:

# See https://github.com/PyO3/pyo3/issues/340#issuecomment-461514532
[dependencies.pyo3]
version = "0.22.5"

[features]
extension-module = ["pyo3/abi3-py39", "pyo3/extension-module"]
default = ["extension-module"]

I will add that I spent at least an hour trying to debug it, so I agree the docs could use a section on this - maybe something like "Calling Python from Rust in Cargo Unit Tests" or similar.

PyO3 / pyo3

`dyld: Symbol not found: _PyExc_SystemError` in tests if there is `#[pymethods]` in project #340

🌍 Environment

💥 Reproducing