Closed kono94 closed 1 year ago
That's a lot easier to do with indexers, and they are more efficient as well: http://bytedeco.org/news/2014/12/23/third-release/
But if you need to use the "official" Java API, @Craigacp would be guy to ask about that.
So presumably the Mat.createBuffer
call hits this - https://github.com/bytedeco/javacpp-presets/blob/master/opencv/src/main/java/org/bytedeco/opencv/opencv_core/AbstractArray.java#L46? If you give ORT a valid buffer it should be fine, but I don't know how JavaCPP converts from the C++ Mat object into a java.nio.Buffer, as it happens before any ORT code executes.
Adam, how many times do I need to tell you before you remember? JavaCPP doesn't do anything with data from OpenCV, ORT, TF, PyTorch, or any other library mapped in the presets. But the data from OpenCV, for example, might not be "compact", that is, it can have arbitrary strides. I don't believe ORT offers any support for that, so that's why I recommend using JavaCPP to work around that, but I may be mistaken, so that's why I'm asking you. What are the limitations of data supported by your bindings?
ORT needs dense row major data. However I think @kono94 is saying that when they convert the data from the Mat
into a FloatBuffer
it's all zeros when inspected using the methods on the FloatBuffer
, which is before any ORT code is executed.
One thing I would check is that the ONNX model you want to use does in fact require float inputs. Many image models accept int8
or unit8
inputs, which you can wrap the byte array you already have into a ByteBuffer
then create the tensor. If the model does require float inputs, then it's a fairly straightforward transformation to put the cast & rescale on the front of the ONNX model by rewriting it's protobuf slightly.
Is there a way to validate that inputBlob
has the right values in it? The codepath you use to access the image elements in the array is quite different to the one which emits the float buffer, and so maybe one step of that transformation is failing.
ORT needs dense row major data. However I think @kono94 is saying that when they convert the data from the
Mat
into aFloatBuffer
it's all zeros when inspected using the methods on theFloatBuffer
, which is before any ORT code is executed.
Ah, I see, looks like the problem is this line:
1/255,
@kono94 That's equal to 0, you probably wanted to write 1.0/255.0.
Still, it's probably faster to use indexers anyway since it doesn't look like we can coerce blobFromImage() to use external buffers either.
Oh man, such a silly mistake! Thank you very much @saudet and @Craigacp for the help!
Using the blob approach instead of the for-loops even improved the detection results somehow.
Two questions are still on my mind:
ai.onnxruntime.*
or org.bytedeco.onnxruntime.*;
like in the example https://github.com/bytedeco/javacpp-presets/blob/master/onnxruntime/samples/CXXApiSample.java
- In terms of performance, should I be using the official Microsoft maven Java package
ai.onnxruntime.*
ororg.bytedeco.onnxruntime.*;
like in the example https://github.com/bytedeco/javacpp-presets/blob/master/onnxruntime/samples/CXXApiSample.java
It's only "official" in the sense that @Craigacp is from Oracle, and it's designed to be easy to use with a small number of features, making it easier for Microsoft to accept pull requests, but it's not designed for performance, for example, we can't access buffers allocated on GPU memory. For most workloads the overhead probably doesn't matter though.
- @saudet Isn't every Indexer .get() call a single JNI-Call, so I end up with 700k JNI calls? Or do I misunderstand you? You want me to loop through the data with an indexer to fill the Java 4D array, no?
No, that uses sun.misc.Unsafe, it's fast: http://bytedeco.org/news/2014/12/23/third-release/
- In terms of performance, should I be using the official Microsoft maven Java package
ai.onnxruntime.*
ororg.bytedeco.onnxruntime.*;
like in the example https://github.com/bytedeco/javacpp-presets/blob/master/onnxruntime/samples/CXXApiSample.javaIt's only "official" in the sense that @Craigacp is from Oracle, and it's designed to be easy to use with a small number of features, making it easier for Microsoft to accept pull requests, but it's not designed for performance, for example, we can't access buffers allocated on GPU memory. For most workload the overhead probably doesn't matter though.
It's official in the sense that it is the supported ORT Java API, and developed in the ORT source tree as part of the ORT project (along with the Python & C# interfaces). The fact that it has an Oracle copyright on it is irrelevant.
The plan is to eventually add IOBinding support which will allow the persistence of tensors on GPUs, though not to the extent that JavaCPP provides. Training support and better CPU memory pinning are going to happen first though.
I'll also point out that it is used in production at several massive tech companies, and they don't complain about performance.
It's official in the sense that it is the supported ORT Java API, and developed in the ORT source tree as part of the ORT project (along with the Python & C# interfaces). The fact that it has an Oracle copyright on it is irrelevant.
No, it's very relevant. Microsoft can't easily accept pull requests from random dudes. The fact that you are at Oracle does help a lot.
The plan is to eventually add IOBinding support which will allow the persistence of tensors on GPUs, though not to the extent that JavaCPP provides. Training support and better CPU memory pinning are going to happen first though.
I'll also point out that it is used in production at several massive tech companies, and they don't complain about performance.
Yes, I know, the overhead of your bindings is not typically relevant for server applications, but it's very relevant, for example, in embedded or mobile applications. The fact that Oracle, Microsoft, and other large corporations running ONNX Runtime on the cloud where performance isn't too relevant doesn't change the fact that all these corporations don't care about embedded and mobile applications, at least when it comes to Java. Please keep an open mind!
Hello, I am trying to get a self-trained YOLOv7-tiny network to run in a Java application. I tried using OpenCV's dnn, but it does not currently include the nms-module and exporting the onnx-model with the detection layer results in a weird error, so I switched to the Onnx inference runtime.
In short, where the problem is the output in the OpenCV inference (for the input, just use the blob created by OpenCV's "blobFromImage"), the input is cumbersome in the Onnx approach as it does not accept Mats, but float-arrays or FloatBuffers.
The problem arises when converting the blob (NCHW Mat) to a FloatBuffer as its values are all 0.0 somehow?! What am I doing wrong?
See the code and output as comments:
It works perfectly, when I am "converting" the Mat to a 4d Java float-array "by hand" but I want to avoid this costly looping...
Thank you in advance for any help!