search

PyO3 / rust-numpy

PyO3-based Rust bindings of the NumPy C-API
BSD 2-Clause "Simplified" License
1.11k stars 106 forks source link

Error importing numpy from Dotnet on Ubuntu #401

Open Sparika opened 11 months ago

Sparika commented 11 months ago

Hi,

I am building a Rust dynamic library embedding Python using rust-numpy. My project allows me to build a pure Rust binary, a C binary loading the Rust library and a Dotnet binary loading the Rust library. All three scenarios works on Windows but fails on Linux with "Error importing numpy".

The following code built as a cdylib is enough to reproduce the problem when imported into a Dotnet SDK 6.0 or 7.0 program

use numpy::{pyo3::{Python, self}, PyArray};

#[no_mangle]
pub extern "C" fn hello_world() {
    // Demonstrates the DLL alone is compatible with Dotnet AnyCPU
    println!("Hello, world!");
    // Demonstrates the ORT DLL is compatible with Dotnet AnyCPU
    println!("Using ort::LoggingLevel::Info : {:?}", ort::LoggingLevel::Info);
    // Demonstrates the Numpy crate is compatible with Dotnet (only uses ndarray though)
    println!("Using numpy::array![1, 2] {:?}", numpy::array![1, 2]);
    // Demonstrates the Numpy crate using actual Python code (numpy array) => fails
    pyo3::prepare_freethreaded_python();
    Python::with_gil(|py| {
        let slice = &[1, 2, 3, 4, 5];
        let pyarray = PyArray::from_slice(py, slice);
        println!("Using numpy::array::Pyarray {:?}", pyarray);
    });
}

On Windows, the code succeeds 

PS C:\Users\****\workspace\demo-rs> .\dotnet\demo_dotnet\bin\Debug\net6.0\demo_dotnet.exe
Starting C# code
Hello, world!
Using ort::LoggingLevel::Info : Info
Using numpy::array![1, 2] [1, 2], shape=[2], strides=[1], layout=CFcf (0xf), const ndim=1
Using nump::array::Pyarray array([1, 2, 3, 4, 5])

On Ubuntu, the code fails

****:~/workspace/demo-rs/dotnet/demo_dotnet/bin/Debug/net7.0$ LD_LIBRARY_PATH=. ./demo_dotnet 
Starting C# code
Hello, world!
Using ort::LoggingLevel::Info : Info
Using numpy::array![1, 2] [1, 2], shape=[2], strides=[1], layout=CFcf (0xf), const ndim=1
thread '<unnamed>' panicked at 'Failed to access NumPy array API capsule: PyErr { type: <class 'ImportError'>, value: ImportError('Error importing numpy: you should not try to import numpy from\n        its source directory; please exit the numpy source tree, and relaunch\n        your python interpreter from there.'), traceback: Some(<traceback object at 0x7f47853a5900>) }', /home/****/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.20.0/src/npyffi/array.rs:52:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Aborted

The error triggers on import of numpy whether we use Rust PyArray or do a import numpy in Python. Removing references to numpy allows to execute Python code with success. For instance, I can print some informations from within Dotnet > Rust > Python as follows :

Numpy was installed using pip3, but I don't think it has anything to do with my Python setup since the dynamic library works when tested from a C code. It seems the error message may be giving a false indication as to the real cause of the error.

Any help would be greatly appreciated!

Sparika commented 11 months ago

The dotnet code to call into the library is : 

using System.Runtime.InteropServices;

namespace Demo
{
    static class Program
    {

        [DllImport("demo_rs", CallingConvention = CallingConvention.Cdecl)]
        private static extern void hello_world();

        static void Main(string[] args)
        {
            Console.WriteLine("Starting C# code");
            hello_world();
        }
    }
}
adamreichold commented 11 months ago

Uff, if this is specific to embedding into a .NET binary, then I cannot really add anything. Personally, I would start grepping the NumPy source tree for that error message to understand why NumPy thinks that it was loading from its source tree.

Chris00 commented 5 months ago

I had this kind of error on OSX when there was a mismatch between the Rust architecture and the Python one.