Ndarray views of numpy arrays

[daniel-vainsencher]

I write numerical code in Rust, but use Python and sometimes R to drive them (e.g., plot graphs comparing performance for different algorithms). This involves exporting an extern function that accepts pointers to (parts of) numpy arrays or R matrices, then treating these as Rusty datastructures (I implemented a basic version of for scirust in [1]).

As a user it would be great to have a function to which I pass a pointer to the raw Numpy array object (without doing any conversions in Python), and get back a view I can use safely. This involves looking at the Numpy array for the dtype, dimension, strides etc. Since ndarray encodes some of these in the type, a generic function that merely tests that the NP array passed conforms to its expectations (or panics with highly informative debug information) seems best.

Same applies to R, with different particulars of course.

Or is there a way to do this already?

[1] https://github.com/daniel-vainsencher/scirust/commit/48c9109abae32db06217576f03c22a18ad39e5d6 which doesn't compile on current nightly rust, sadly.

[bluss]

No numpy integration at the moment. How do you interface between Rust and numpy? just curious.

[daniel-vainsencher]

I am glad to help, of course :)

The Rust side of my current (ugly) solution looks like this:

#[no_mangle]
pub extern "C" fn ls_regression(l1reg: f64, iterations: usize,
    x_data: *const f64, x_raw_ndim: i32, x_raw_dims: *const i32, x_byte_strides: *const i32,
    y_data: *const f64, y_raw_ndim: i32, y_raw_dims: *const i32, y_byte_strides: *const i32,
    b_data: *const f64, b_raw_ndim: i32, b_raw_dims: *const i32, b_byte_strides: *const i32) {

    with_matrix_over_numpy(x_data, x_raw_ndim, x_raw_dims, x_byte_strides, &|x| {
        with_matrix_over_numpy(y_data, y_raw_ndim, y_raw_dims, y_byte_strides, &|y| {
            with_matrix_over_numpy(b_data, b_raw_ndim, b_raw_dims, b_byte_strides, &|b| {
                let answer = least_squares_regression(&x, &y, l1reg, iterations, &None);
                for r in (0..answer.data().num_rows()) {
                    b.set(r, 0, answer.data().get(r, 0).unwrap());
                }
            });
        });
    });
}

Each call to with_matrix_over_numpy sets up a context where we are essentially borrowing a given matrix.

[daniel-vainsencher]

I am currently calling this code from R, and with mere matrices, and everything is specialized to f64s, so I am not using any genericity.

[bluss]

It's sort of only an application of https://bluss.github.io/rust-ndarray/master/ndarray/struct.ArrayBase.html#method.from_slice_dim_stride

[bluss]

I know the ShapeError is not very detailed unfortunately.

[bluss]

I think a feature for ndarray to numpy integration should have as goal to be a separate crate. This crate should expand its api if the public api is not enough to allow the implementation.

[daniel-vainsencher]

Cool, I didn't now where that function was, but I can use that to get to the same level I had before.

[daniel-vainsencher]

The current issue is for the more ambitious fn numpy_array<D, A>(a: *const nparray) -> Result<Self, ShapeError>

but that might indeed be best served by its own crate, as it would depend on numpy internals :/

[bluss]

implementation details shouldn't be necessary if you have http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.array.html

[daniel-vainsencher]

That c-api is one made of macros, hence is not exported by any DLL, so we cannot use it directly from Rust, right?

[bluss]

I didn't know that, that makes it more messy. Still, a binding library can expose a macro through a new function.

[termoshtt]

FYI, I create numpy binding to rust while it is still developing. https://github.com/termoshtt/rust-numpy

That c-api is one made of macros, hence is not exported by any DLL, so we cannot use it directly from Rust, right?

Using this crate, we can call NumPy C API from Rust!

[termoshtt]

@bluss, I'd like to create a high-level interface of PyArray (corresponding to numpy.ndarray) using rust-ndarray. I already got shape, strides, and raw pointer to the data of the numpy array, so I think it is enough to construct rust-ndarray wrapping, e.g. ArrayBase<PyArray>. However, the document says that the Data trait is private, so I cannot implement it for my PyArray. Is there any option to do similar thing?

[termoshtt]

I found from_shape of ArrayView seems to work for it. Thanks

[landersson]

Just found this repo/issue when looking into ways of accessing numpy ndarrays down to rust functions called from python. Being able to create rust ndarray views of numpy ndarrays would be very useful. Are there any plans to support this in the near future?