Create a single Android package

[dkurt]

ARM64 and x86_64 native libraries should be in a single package. The problem in plugins.xml location resolving.

[keyur2maru]

@dkurt any progress here? I am using your latest release on arm64-v8a running Android 8/11/12 but they all fail to resolve location of plugins.xml Please let me know if there's any known way to resolve this issue

[dkurt]

@keyur2maru, this issue is not about plugins.xml. You should resolve it in this way: https://github.com/dkurt/audio_recognition_android/blob/6400423e4d7f0cd872b517ac8298a506d908240c/app/src/main/java/com/example/audiorecognition/MainActivity.java#L179-L183

[keyur2maru]

@dkurt thank you for your response. Here is how I am currently using it and I have also tried adding repo from GitHub. My current code is copied as is from https://github.com/dkurt/audio_recognition_android. It fails to find plugins.xml and InputStream in is null

my code snippets build.gradle

dependencies {
    implementation 'androidx.appcompat:appcompat:1.6.1'
    implementation 'com.google.android.material:material:1.11.0'
    implementation files('libs/openvino-2023.0-android-arm64-v8a.aar')
    testImplementation 'junit:junit:4.13.2'
    androidTestImplementation 'androidx.test.ext:junit:1.1.5'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.5.1'
}

MainActivity.java

        // Initialize OpenVINO
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

        // Copy plugins.xml from resources
        InputStream in = Core.class.getClassLoader().getResourceAsStream("plugins.xml");
        String pluginsXml = getResourcePath(in, "plugins", "xml");

        // Create OpenVINO Core object using temporal plugins.xml
        core = new Core(pluginsXml);

Logcat

E  FATAL EXCEPTION: main
Process: io.esper.openvinofoundation, PID: 5003
java.lang.RuntimeException: Unable to start activity ComponentInfo{io.esper.openvinofoundation/io.esper.openvinofoundation.MainActivity}: java.lang.NullPointerException
         at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2913)
         at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3048)
         at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:78)
         at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:108)
         at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:68)
         at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1808)
         at android.os.Handler.dispatchMessage(Handler.java:106)
         at android.os.Looper.loop(Looper.java:193)
         at android.app.ActivityThread.main(ActivityThread.java:6669)
         at java.lang.reflect.Method.invoke(Native Method)
         at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
         at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
        Caused by: java.lang.NullPointerException
         at java.util.Objects.requireNonNull(Objects.java:203)
         at java.nio.file.Files.copy(Files.java:2984)
         at io.esper.openvinofoundation.MainActivity.getResourcePath(MainActivity.java:142)
         at io.esper.openvinofoundation.MainActivity.onCreate(MainActivity.java:180)
         at android.app.Activity.performCreate(Activity.java:7136)
         at android.app.Activity.performCreate(Activity.java:7127)
         at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1271)
         at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2893)
         at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3048) 
         at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:78) 
         at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:108) 
         at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:68) 
         at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1808) 
         at android.os.Handler.dispatchMessage(Handler.java:106) 
         at android.os.Looper.loop(Looper.java:193) 
         at android.app.ActivityThread.main(ActivityThread.java:6669) 
         at java.lang.reflect.Method.invoke(Native Method) 
         at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493) 
         at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858) 

[keyur2maru]

@dkurt i found out that plugins.xml is no longer available, it was last available in 2022.3 If I extract the aar and then classes, plugins.xml is available in 2022.3

but after that it is not visible, for instance in 2023.0.0.dev20230427

Please let me know if there's a different way to find plugins.xml now

[dkurt]

@keyur2maru, can you verify https://github.com/dkurt/openvino_java/actions/runs/8480462542/artifacts/1369656877 ?

[keyur-esper]

@dkurt yes it is able to find plugins.xml now, thank you! But your audio_recognition_android no longer works neither does the coco_detection_android_demo

After some digging, both of these apps have stopped working since these two commits that removed old arm plugin and replaced it by adding arm support in existing intel_cpu module https://github.com/openvinotoolkit/openvino_contrib/commit/4d35f32a797d0fbee38910bfedb127217ed9f3c9 https://github.com/openvinotoolkit/openvino/commit/c283d212151f38f00446db0fe43945e228193176

I can share detailed logs as well and open a new issue if you'd like, thanks!

[dkurt]

@keyur-esper, please share the logs, thank you!

As I see, the libopenvino_arm_cpu_plugin.so was added to the package and plugins.xml refers to it:

<ie>
<plugins>
<plugin name="AUTO" location="libopenvino_auto_plugin.so"> </plugin>
<plugin name="BATCH" location="libopenvino_auto_batch_plugin.so"> </plugin>
<plugin name="CPU" location="libopenvino_arm_cpu_plugin.so"> </plugin>
<plugin name="HETERO" location="libopenvino_hetero_plugin.so"> </plugin>
<plugin name="MULTI" location="libopenvino_auto_plugin.so"> </plugin>
</plugins>
</ie>

Is a problem at compile_model step? Maybe model upgrade is required.

[keyur2maru]

@dkurt yes it is at compile_model step, even though the error differs from your audio_recognition_android and coco_detection_android_demo. Here are the logs from your app

------- beginning of crash
2024-03-29 02:39:46.493 13330-13330 AndroidRuntime          pid-13330                            
E  FATAL EXCEPTION: main
              Process: io.esper.keyur_openvino, PID: 13330
              java.lang.RuntimeException: Unable to start activity ComponentInfo{io.esper.keyur_openvino/io.esper.keyur_openvino.MainActivity}: java.lang.Exception: 
              InferenceEngineException: 
               CompileModel: Exception from src/inference/src/cpp/core.cpp:109:
              Exception from src/inference/src/dev/plugin.cpp:54:
              Exception from src/plugins/intel_cpu/src/nodes/matmul.cpp:303:
              MatMul node with name '304' has invalid dims count                                                                                                                             
                           at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4169)
                           at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4325)
                           at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:101)
                           at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
                           at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
                           at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2574)
                           at android.os.Handler.dispatchMessage(Handler.java:106)
                           at android.os.Looper.loopOnce(Looper.java:226)
                           at android.os.Looper.loop(Looper.java:313)
                           at android.app.ActivityThread.main(ActivityThread.java:8762)
                           at java.lang.reflect.Method.invoke(Native Method)
                           at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:604)
                           at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1067)
                          Caused by: java.lang.Exception: 
                          InferenceEngineException: 
                           CompileModel: Exception from src/inference/src/cpp/core.cpp:109:
                          Exception from src/inference/src/dev/plugin.cpp:54:
                          Exception from src/plugins/intel_cpu/src/nodes/matmul.cpp:303:
                          MatMul node with name '304' has invalid dims count

[keyur2maru]

@dkurt Also the classes under classes.jar is empty, org/intel/openvino only contains BuildConfig.class. It is due to the version of your openvino_contrib contains org/* folder at the root of java_api module