pyo3 asyncio not working with pyembed

🐛 Bug Reports

error: future cannot be sent between threads safely --> src/main.rs:7:1	7	#[pyo3_asyncio::tokio::main]	^^^^^^^^^^^^^^^^^^^^^^^^^^^^ future returned by `main` is not `Send`

= help: within `impl Future<Output = Result<(), PyErr>>`, the trait `std::marker::Send` is not implemented for `std::sync::MutexGuard<'_, ()>`

note: future is not `Send` as this value is used across an await --> src/main.rs:25:6	16	let interp = pyembed::MainPythonInterpreter::new(config).unwrap();	------ has type `MainPythonInterpreter<'_, '_>` which is not `Send` ... 25	v.await;	^^^^^^ await occurs here, with `interp` maybe used later 26	Ok(()) 27	}	- `interp` is later dropped here note: required by a bound in `pyo3_asyncio::tokio::run` --> /Users/frederik/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-asyncio-0.17.0/src/tokio.rs:238:39
238	F: Future<Output = PyResult> + Send + 'static,
^^^^ required by this bound in `run`

= note: this error originates in the attribute macro `pyo3_asyncio::tokio::main` (in Nightly builds, run with -Z macro-backtrace for more info)

error: future cannot be sent between threads safely --> src/main.rs:7:1	7	#[pyo3_asyncio::tokio::main]	^^^^^^^^^^^^^^^^^^^^^^^^^^^^ future returned by `main` is not `Send`

= help: within `impl Future<Output = Result<(), PyErr>>`, the trait `std::marker::Send` is not implemented for `*mut c_void`

note: future is not `Send` as this value is used across an await --> src/main.rs:25:6	16	let interp = pyembed::MainPythonInterpreter::new(config).unwrap();	------ has type `MainPythonInterpreter<'_, '_>` which is not `Send` ... 25	v.await;	^^^^^^ await occurs here, with `interp` maybe used later 26	Ok(()) 27	}	- `interp` is later dropped here note: required by a bound in `pyo3_asyncio::tokio::run` --> /Users/frederik/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-asyncio-0.17.0/src/tokio.rs:238:39
238	F: Future<Output = PyResult> + Send + 'static,
^^^^ required by this bound in `run`

= note: this error originates in the attribute macro `pyo3_asyncio::tokio::main` (in Nightly builds, run with -Z macro-backtrace for more info)

🌍 Environment

Your operating system and version: Macos
Your python version: 3.10
How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: PyOxidizer
Your Rust version (rustc --version): rustc 1.69.0 (84c898d65 2023-04-16)
Your PyO3 version: 0.17
Have you tried using latest PyO3 master (replace version = "0.x.y" with git = "https://github.com/awestlake87/pyo3-asyncio")?:

💥 Reproducing

#[pyo3_asyncio::tokio::main]
async fn main()  -> PyResult<()>{
    // Get config from default_python_config.rs.
    let config = default_python_config();

    let interp = pyembed::MainPythonInterpreter::new(config).unwrap();

    // `py` is a `pyo3::Python` instance.
    let v = interp.with_gil(|py| {
        let asyncio = py.import("asyncio").unwrap();
       pyo3_asyncio::tokio::into_future(asyncio.call_method1("sleep", (1.into_py(py),)).unwrap())
        // convert asyncio.sleep into a Rust Future
    }).unwrap().await;
    Ok(())
}

Please also write what exact flags are required to reproduce your results.

Not sure I would consider this a bug with PyO3 Asyncio or PyEmbed. The error makes sense to me.

Basically what's happening is that you're creating a Python interpreter using PyEmbed and holding it over an await boundary. Tokio requires its futures to be Send (except in some niche cases with LocalSet), but PyEmbed added negative trait impls for !Send and !Sync to its interpreter. Presumably if these negative impls were not there in PyEmbed, you could get some nasty memory safety issues, so the Rust compiler is just protecting you from doing something bad here.

From your example, it looks like you want to use PyEmbed to create your interpreter and then perform some async operations on the embedded interpreter. In order to do something like this, I think you'll have to accept that your async operations will have to use Python::with_gil instead of interp.with_gil. interp.with_gil is just a wrapper around Python::with_gil except it holds a reference to the interpreter to ensure that you don't drop the interpreter before the operation is finished. According to the docs Python::with_gil should work fine, but you will need to be very careful about managing the lifetime of your interpreter. See the interpreter Usage and Safety.

pyo3-asyncio does not unify the Python and Rust async runtime, it glues them together across (at least) two threads. You'll need to have your embedded interpreter running on one thread (ideally the main thread), and your tokio runtime initialized separately. In order to perform async operations you'll spawn futures onto the tokio runtime, and those futures will be allowed to use Python::with_gil to interact with your interpreter.

Sadly this is a bit more complicated than the typical usage, but I do actually have something similar in my tests with run_forever:

In your case I think using pyo3_asyncio::tokio::run would suffice though. Essentially your situation would boil down to something like this:

fn main() {
    // Get config from default_python_config.rs.
    let config = default_python_config();

    let interp = pyembed::MainPythonInterpreter::new(config).unwrap();

    // `py` is a `pyo3::Python` instance.
    let v = interp.with_gil(|py| {
        // Wraps a Rust future with Python's `asyncio.run`
        pyo3_asyncio::tokio::run(py, async move {
            // Inside the tokio event loop, reference the interpreter via `Python::with_gil`
            Python::with_gil(|py| {
                let asyncio = py.import("asyncio").unwrap();
                pyo3_asyncio::tokio::into_future(asyncio.call_method1("sleep", (1.into_py(py),)).unwrap())
            }).await.unwrap();

            Ok(())
        })
        .map_err(|e| {
            e.print_and_set_sys_last_vars(py);
        })
        .unwrap();
    });

    // interp will be finalized when it goes out of scope. In this simple example, 
    // we know tokio's tasks are all wrapped up, but in a real world example we should 
    // shutdown our tokio runtime before dropping the interpreter to ensure that none 
    // of our background tasks try to access the interpreter after finalization.
    drop(interp);
}

Full Disclosure, I haven't compiled and tested this yet

@Omicronlawful were you able to figure this out, facing similar issue

awestlake87 / pyo3-asyncio