Conversion between org.bytedeco.opencv.opencv_core.Mat (Java) and cv2.Mat (Python): NDArray or DirectNDArray?

[Daniel-Alievsky]

Please advise me the maximally correct way of passing OpenCV matrices between Java and Python via Jep.

On Java side, I work with images by using bytedeco javacpp bridge. I need to pass such an image from Java to Python, process their with help of Python OpenCV (cv2 module), then pass them back to Java.

Let's consider very simple Python function:

def blurMat(src, ksize):
    return cv2.GaussianBlur(src,(ksize,ksize),cv2.BORDER_DEFAULT)

In Java I do the following:

        final DirectNDArray<Buffer> ndSource = toNDArray(source);
...
        interp.exec("import tests.MatBlurTest as Test");
        Object ndResult = interp.invoke("Test.blurMat", ndSource, 21);

Here is toNDArray function:

    public static DirectNDArray<Buffer> toNDArray(Mat m) {
        final int[] dimensions = {m.rows(), m.cols(), m.channels()};
        Objects.requireNonNull(m, "Null Mat");
        ByteBuffer byteBuffer = m.data().position(0).capacity(m.arraySize()).asByteBuffer();
        // of course it is correct only for byte matrix
        return new DirectNDArray<>(byteBuffer, isUnsigned(m), dimensions);
    }

Really, on the Python side, NDArray is automatically converted into Mat and vice versa. Very well.

On Java side I must use DirectNDArray: OpenCV is oriented to native memory, that is represented in Java bridge via ByteBuffer. But what will be the type of ndResult, result of calling Python blurMat? It is NDArray! Why?

So, I need to write something like this:

    public static Mat toMat(NDArray<byte[]> ndArray) {
        final ByteBuffer byteBuffer = ByteBuffer.wrap(ndArray.getData());
        //of course, only for byte elements
        final int[] dimensions = ndArray.getDimensions();
        if (dimensions.length == 2 || (dimensions.length == 3 && dimensions[2] == 1)) {
            return toMat(dimensions[1], dimensions[0], opencv_core.CV_8UC1, byteBuffer);
        } else if (dimensions.length == 3) {
            return toMat(dimensions[1], dimensions[0],
                    opencv_core.CV_MAKETYPE(opencv_core.CV_8U, dimensions[2]),
                    byteBuffer);
        } else {
            throw new IllegalArgumentException("Unsupported NDArray dimensions: " + Arrays.toString(dimensions));
        }
    }

Why do I need here additional conversion byte[] -> ByteBuffer? Why, on Python side, OpenCV matrix is converted to NDArray<byte[]>, not to DirectNDArray?

Moreover, do I implement all conversions correctly (besides not supporting non-byte elements yet)? Do you really guarantee that NDArray (from Java) can be user as OpenCV matrices on Python side, and, vice versa, that OpenCV matrices from Python will be always transformed to NDArray on Python side? It is not obvious from documentation.

I think it is important to provide clear and stable bridge between Java and Python image processing, and OpenCV seems to be one of most popular libraries on the both sides.

[bsteffensmeier]

Why do I need here additional conversion byte[] -> ByteBuffer? Why, on Python side, OpenCV matrix is converted to NDArray<byte[]>, not to DirectNDArray?

The way that JNI and cpython are implemented makes it much easier for jep to allow memory allocated in Java to be used directly in Python and much more difficult to do the other way around. Java offers no prompt method to notify native code that the DirectByteBuffer is no longer used and also offers no API to ensure Java cannot access memory that has been freed by Python. I have some ideas how Jep may be able to allow direct access to Python memory in very limited circumstances but it is not implemented in the current version and due to the complexity and limitations it is unlikely to be implemented in the near future.

Moreover, do I implement all conversions correctly (besides not supporting non-byte elements yet)? Do you really guarantee that NDArray (from Java) can be user as OpenCV matrices on Python side, and, vice versa, that OpenCV matrices from Python will be always transformed to NDArray on Python side? It is not obvious from documentation.

Nothing looks incorrect about your conversions. I do not know anything about OpenCV. When Jep is built with numpy enabled it will always convert an NDArray(or DirectNDArray) in Java to a numpy ndarray in Python and numpy ndarray in Python will be converted to an NDArray in Java. I assume from your code that OpenCV is also using numpy ndarrays for input and output so it should be guaranteed to work.

I do see one small optimization that could improve the Python->Java conversion. Currently the python ndarray is converted to a java NDArray and the java NDArray is converted to another type, so the data is copied twice. If you pass another, empty, DirectNDArray into python you should be able to copy the result of the cv2 operation to the direct array in python using numpy.copyto() or something similar. This would result in only one copy, from Python allocated memory to java allocated memory without the java byte[] in between.

[Daniel-Alievsky]

Thank you for your explanations, I understand.

By the way, I recommend to draw attention to importance of good support of OpenCV. Let me explain.

Java has relatively poor set of libraries in area of computer science. If was not a focus while developing Java language, and now it is relatively difficult to find necessary algorithm, implemented in Java with high quality or having bridge to Java. On the other hand, out experience while last 20 years proved very good features of Java in area of developing new algorithms. Having comparable efficiency with C/C++, Java provides high safety of programming and allow to avoid a lot of "stupid" bugs like memory damage or freeing used memory. Moreover, you may forget about system crash - your server, processing a lot of data, will work stably. It allows you to focus on algorithms themselves - their quality, speed, supported features, more optimal data structures etc. For example, some time ago I've implemented in Java graph-based segmentation algorithm, which is already implemented in OpenCV, and my Java implementation works in 3-10 times faster than C implementation.

Unlike this, Python has a very wide set of professional libraries in almost all areas, and allows to use them very simply . Python code, using OpenCV or machine learning libraries, is much more simple and obvious than equivalent code in Java or C++. But implementing new algorithms is Python is usually bad idea due to low efficiency of built-in operations in CPython.

Your bridge, Jep, opens an ability to join best sides of Java and Python. Java gets an ability to use thousands of powerful libraries, available in Python, and Python gets an ability to use libraries, written in Java instead of C++ - above I explained advantages of such approach. Our system, Stare, allows to combine solutions written in different platforms, and now my goal is providing smart integration between Java and Python executors via Jep. I even think we could collaborate with you in future.

All this means that it is important to support main computer-science libraries, popular from both sides - Java and Python - and to deeply investigate possible problems and maximal simple ways of integration. OpenCV is one of such libraries, and maybe it makes sense for you to look on it more thoroughly. One of possible issues is UMat, i.e. Mat allocated in GPU memory - it is a very efficient approach in some algorithms and I suggest (not investigated yet) that Python supports it as good as JavaCPP. But you understand that passing GPU memory via NDArray is not a good idea. And so on and so forth.

In any case thank you for your support.