Feature request: Add conversion from OpenCV to ndarray

[Machine-Jonte]

I think converting OpenCV to ndarray would be a good feature. As Pyo3 has good support for converting numpy arrays to ndarrays, seems like a good feature to be able to convert OpenCV mat directly to ndarray as that enables python/rust OpenCV usage.

A possible conversion could be

trait AsArray {
    fn try_as_array(&self) -> Result<ArrayView3<u8>>;
}impl AsArray for cv::core::Mat {
    fn try_as_array(&self) -> Result<ArrayView3<u8>> {
        if !self.is_continuous() {
            return Err(anyhow!("Mat is not continuous"));
        }
        let bytes = self.data_bytes()?;
        let size = self.size()?;
        let a = ArrayView3::from_shape((size.height as usize, size.width as usize, 3), bytes)?;
        Ok(a)
    }
}

Though, this only works for continuous arrays.

[jerry73204]

It is implemented in opencv-to-ndarray branch. You can see the usage in test code. Can you check if it works for you?

To run the test,

cargo test --features ndarray_0-15,opencv_0-63 mat_ref_to_array_view_conversion

To include the new future in your crate, add this to Cargo.toml.

[dependencies]
cv-convert = { git = "https://github.com/jerry73204/rust-cv-convert.git", branch = "opencv-to-ndarray", features = ["ndarray_0-15", "opencv_0-63"] }

[Machine-Jonte]

Hello, Thanks for the update! Unfortunately, I get a runtime error. Below is the code I'm running:

mod test {
    use cv_convert::{TryFromCv, TryIntoCv};
    use opencv::{self as cv, prelude::*};
    #[test]
    fn test_github() {
        let test_img = cv::imgcodecs::imread(
            "./test_img.png",
            cv::imgcodecs::IMREAD_COLOR,
        )
        .expect("Failed to read image");
        assert!(!test_img.empty());
        let nd_mat: ndarray::ArrayViewD<u8> =
            (&test_img).try_into_cv().expect("Failed conversion.");
    }
}

Error message:

running 1 test
thread 'test::test_github' panicked at 'Failed conversion.: ShapeError/OutOfBounds: out of bounds indexing', experiment/src/main.rs:34:39
stack backtrace:
   0: rust_begin_unwind
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
   1: core::panicking::panic_fmt
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
   2: core::result::unwrap_failed
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/result.rs:1613:5
   3: core::result::Result<T,E>::expect
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/result.rs:1255:23
   4: experiment::test::test_github
             at ./src/main.rs:34:13
   5: experiment::test::test_github::{{closure}}
             at ./src/main.rs:26:5
   6: core::ops::function::FnOnce::call_once
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:227:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test test::test_github ... FAILED

failures:

failures:
    test::test_github

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.33s

error: test failed, to rerun pass '-p experiment --bin experiment'
The terminal process "cargo 'test', '--package', 'experiment', '--bin', 'experiment', '--', 'test', '--nocapture'" terminated with exit code: 101.

I think it has to do with the 3 channels in the Mat. If I use a grayscale image, it does work (y)

[Machine-Jonte]

When I'm running the test you created I can see that it always add a 1 dim in the end

[experiment/src/main.rs:48] &mat.shape() = [
    31,
    17,
    8,
    13,
    1,
]
[experiment/src/main.rs:48] &mat.shape() = [
    19,
    10,
    1,
]
[experiment/src/main.rs:48] &mat.shape() = [
    19,
    27,
    24,
    8,
    1,
]
[experiment/src/main.rs:48] &mat.shape() = [
    6,
    27,
    1,
]
[experiment/src/main.rs:48] &mat.shape() = [
    2,
    11,
    8,
    1,
]

The shape for a RGB (BGR) image is

[experiment/src/main.rs:85] &test_img.shape() = [
    2880,
    5121,
    3,
]

This is why the test case doesn't detect it. This also explains why grayscale works as the last dim is 1.

[Machine-Jonte]

I tracked the issue to mat.as_slice(). It doesn't return the right number of elements for the RGB case.

The issue in the as_slice function is that the total() counts the number of pixels and not data points (i.e. that is why it works for 1 in the channel as total then counts the correct number of data points

https://stackoverflow.com/questions/16990510/opencv-how-to-get-the-number-of-pixels

The code can be fixed by replacing the total with

let numel = self.total() * self.shape().last().unwrap();

or

let numel = mat.shape().iter().fold(1, |a, b| a * b);

Did a pull request: https://github.com/jerry73204/rust-cv-convert/pull/8

[jerry73204]

The concept of OpenCV's Mat channel is absent in ndarray, tch and many multi-dimensional arrays. In this case, the channel is treated as the last dimension and thus you see one additional dimension there.

You PR is merged and the crate doc is changed accordingly. If you feel comfortable with this change, I will publish it on crates.io.

[Machine-Jonte]

Sounds good, thanks for your efforts! (Feel free to publish it)