An example Script for loading models and inferencing use kinference in Java?

[com3dian]

Hi,

I am quite new to Java and barely know nothing about Kotlin. I am currently attempting to use the Kinference package in my Java project. While exploring the example codes in the public repo (kinference-examples) and it seems I could use Engine.loadmodel(path) to get a model, therefore I have the following code to achieve this:

import io.kinference.ort.model.ORTModel;
import io.kinference.ort.ORTEngine;

public class ORTEngineExample {
    public static void main(String modelPath) {
        // Create an instance of ORTEngine
        ORTEngine ENGINE = new ORTEngine();
        // Load model
        ORTModel ortModel = ENGINE.loadModel(modelPath);
    }
}

yet I get

error: ORTEngine() has private access in ORTEngine
        ORTEngine ENGINE = new ORTEngine();
                           ^

As a newbee, I am struggling to understand how to resolve this issue so I am reaching out for assistance. Could someone maybe share an example code demonstrating how to load an ONNX model and run inferencing effectively in a Java project? Thanks in advance!

[cupertank]

Hi! ORTEngine is a singleton class, you shouldn't create it. Just use ORTModel ortModel = ORTEngine.loadModel(modelPath);.

[com3dian]

Hi Ilya, thanks for the very quick reply! So now I update my code into

public class ORTEngineExample {
    protected ORTEngineExample (String modelPath) throws IOException {
//        Path path = Paths.get(modelPath);
//        byte[] modelBytes = Files.readAllBytes(path);

        // ORTEngine engine();
        ORTModel ortModel = ORTEngine.loadModel(modelPath);
        return ortModel;
    }

    public static void main(String[] args) throws IOException {
        ORTModel ortModel = new ORTEngineExample("/my/actual/model/path.onnx");
    }
}

and I have the following error message:

error: no suitable method found for loadModel(String)
        ORTModel ortModel = ORTEngine.loadModel(modelPath);
                                     ^
    method ORTEngine.loadModel(byte[],Continuation<? super ORTModel>) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(byte[],SessionOptions) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(byte[],boolean,OrtLoggingLevel) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(Path,Continuation<? super ORTModel>) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(byte[],boolean,Continuation<? super ORTModel>) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(Path,boolean,Continuation<? super ORTModel>) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(String,Continuation<? super ORTModel>) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(Path,SessionOptions) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(String,SessionOptions) is not applicable
      (actual and formal argument lists differ in length)
    method ORTEngine.loadModel(String,boolean,Continuation<? super ORTModel>) is not applicable
      (actual and formal argument lists differ in length)

Also, I have tried to provide other inputs for loadModel like passing a byteArray, but it returns the same error with fewer listed not-applicable methods, do you have any ideas on this? Thank you!

[cupertank]

Hi! I see your problem, it happens because it's suspendable function from Kotlin. Let's try this code:

package me.cupertank;

import io.kinference.ort.model.ORTModel;
import io.kinference.ort.ORTEngine;
import kotlinx.coroutines.BuildersKt;
import kotlinx.coroutines.Dispatchers;
import java.io.IOException;

public class ORTEngineExample {
    protected static ORTModel getModel(String modelPath) throws IOException, InterruptedException {

        ORTModel model = BuildersKt.runBlocking(
            Dispatchers.getDefault(),
            (scope, continuation) -> ORTEngine.INSTANCE.loadModel(modelPath, continuation)
        );

        try {
            return model;
        } finally {
            model.close();
        }
    }

    public static void main(String[] args) throws IOException, InterruptedException {
        ORTModel ortModel = getModel("/path/to/onnx/file");
        //...Some code...
        //Don't forget to close model since you don't need it
        ortModel.close();
    }
}

[com3dian]

Hi Ilya @cupertank , many thanks for your help! Though I still don't quite understand about how to handle the kotlin coroutines in java, I eventually switch my project to kotlin.