How to load the trained model using setTrainingPath in an Android application.

[mogamusa31]

Hello

I want to get the bounding box of a recognized object in an Android application. So I decided to do that using ObjectnessBING.

So, I generated ObjectnessBING and performed object recognition by computeSaliency. As a result, when I tried to get the bounding box with getobjectnessValues, I could not get any. After investigating the reason, I guessed that the cause was not loading the trained model using setTrainingPath. Is this guess correct?

If that guess is correct, could you tell me how to load the trained model using setTrainingPath in an Android application?

Also, please let me know if there is a way to load the trained model without using setTraningPath.

Thank you for your response.

[mogamusa31]

example code

        Bitmap bitmap = BitmapFactory.decodeResource(context.getResources(), R.drawable.pepper);
        Mat bitmapMat = new Mat();
        Utils.bitmapToMat(bitmap, bitmapMat);
        opencv_core.Mat imageMat = TestJavaCV.convertMat(bitmapMat);
        opencv_saliency.ObjectnessBING objectnessBING = opencv_saliency.ObjectnessBING.create();
        // objectnessBING.setTrainingPath();
        opencv_core.Mat bingMat = new opencv_core.Mat();
        Boolean bool = objectnessBING.computeSaliency(imageMat, bingMat);
        FloatPointer pointer = objectnessBING.getobjectnessValues();
        if(Objects.isNull(pointer)) {
            System.out.println("pointer is null");
        }

[saudet]

Yes, it looks like we need to call setTrainingPath(): https://github.com/opencv/opencv_contrib/blob/master/modules/saliency/samples/computeSaliency.cpp#L152 Simply do as shown in that sample code and it should work. What is the question exactly?

[mogamusa31]

thank you for your answer.

I put the trained training models in a folder called ObjectnessTrainedModel in the assets folder And tried to load the trained model placed in the ObjectnessTrainedModel in Method1, but as a result nothing changed. As the cause, I think that there was no change because reading by Method1 was not the correct method. So, if you know how to load on Android, I'd like to know how to do it. . .

Method1: objectnessBING.setTrainingPath ("file: /// android_asset / ObjectnessTrainedModel");

The file that I placed in the assets / ObjectnessTrainedModel folder uses the one found at the link below. https://github.com/opencv/opencv_contrib/tree/master/modules/saliency/samples/ObjectnessTrainedModel

[mogamusa31]

By the way, when using the ComputeSaliency of ObjectnessBING with OpenCV of Python for the image used on the PC for trial, the object was recognized normally as follows.

[saudet]

If you put the file in your application's assets, you'll need to extract it back to a file: https://stackoverflow.com/questions/4447477/how-to-copy-files-from-assets-folder-to-sdcard

[mogamusa31]

thank you for your answer.

As advised, I implemented it as follows, and I was able to confirm that I could copy the files in the asset folder to the data folder and get the files, but it seems that the object recognition is not going well.

https://github.com/mogamusa31/opencv-android-test/blob/master/app/src/main/java/tokyo/masayuki/image/processing/android/opencvtest/TestJavaCV.java#L60-L109

Is it not working because the copy processing part of the following file is wrong? I'm sorry to ask you many times.

the copy processing part https://github.com/mogamusa31/opencv-android-test/blob/master/app/src/main/java/tokyo/masayuki/image/processing/android/opencvtest/TestJavaCV.java#L103

[saudet]

It's probably something wrong with the way you're loading the image. Try to use imread() instead.

[mogamusa31]

Thank you for your reply.

I Implemented as follows using "Imgcodecs.imread". As a result, object recognition did not work. https://github.com/mogamusa31/opencv-android-test/pull/4/files#diff-a96d4a5736a89edad095c3c9fcbb2baeR46

Therefore, I verified that "Imgcodecs.imread" is functioning properly. As a verification method, I convert the opencv_core.Mat read by imread into org.opencv.core.Mat again, and confirm that the conversion to Bitmap is working properly with Utils.matToBitmap. As a result, running Utils.matToBitmap resulted in the following error: Is this an error caused by imread failing?

verifing test code

static Bitmap testLoadBitmapInDataFolder(Context context) throws IOException {
        String folderName = "image";
        String outputDir = "/data/data/" + context.getPackageName() + "/" + folderName;
        copyAssets(context, outputDir, folderName);
        String[] files = context.getAssets().list(folderName);
        Mat bitmapMat = Imgcodecs.imread(outputDir + "/" + files[0]);
        Bitmap bitmapForInfo = BitmapFactory.decodeResource(context.getResources(), R.drawable.pepper);
        Bitmap outputBitmap = Bitmap.createBitmap(bitmapForInfo.getWidth(), bitmapForInfo.getHeight(), Bitmap.Config.ARGB_8888);
        Utils.matToBitmap(bitmapMat, outputBitmap);
        return outputBitmap;
    }

error detail

E/cv::error(): OpenCV(3.4.3) Error: Assertion failed (src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols) in void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean), file /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-3.4.3/modules/java/generator/src/cpp/utils.cpp, line 101
E/org.opencv.android.Utils: nMatToBitmap caught cv::Exception: OpenCV(3.4.3) /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-3.4.3/modules/java/generator/src/cpp/utils.cpp:101: error: (-215:Assertion failed) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function 'void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean)'
D/AndroidRuntime: Shutting down VM
E/AndroidRuntime: FATAL EXCEPTION: main
    Process: tokyo.masayuki.image.processing.android.opencvtest, PID: 3223
    java.lang.RuntimeException: Unable to resume activity {tokyo.masayuki.image.processing.android.opencvtest/tokyo.masayuki.image.processing.android.opencvtest.MainActivity}: CvException [org.opencv.core.CvException: OpenCV(3.4.3) /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-3.4.3/modules/java/generator/src/cpp/utils.cpp:101: error: (-215:Assertion failed) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function 'void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean)'
    ]
        at android.app.ActivityThread.performResumeActivity(ActivityThread.java:4430)
        at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:4470)
        at android.app.servertransaction.ResumeActivityItem.execute(ResumeActivityItem.java:51)
        at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:145)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:70)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2199)
        at android.os.Handler.dispatchMessage(Handler.java:112)
        at android.os.Looper.loop(Looper.java:216)
        at android.app.ActivityThread.main(ActivityThread.java:7625)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)
     Caused by: CvException [org.opencv.core.CvException: OpenCV(3.4.3) /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-3.4.3/modules/java/generator/src/cpp/utils.cpp:101: error: (-215:Assertion failed) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function 'void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean)'
    ]
        at org.opencv.android.Utils.nMatToBitmap2(Native Method)
        at org.opencv.android.Utils.matToBitmap(Utils.java:123)
        at org.opencv.android.Utils.matToBitmap(Utils.java:132)
        at tokyo.masayuki.image.processing.android.opencvtest.TestJavaCV.testLoadBitmapInDataFolder(TestJavaCV.java:138)
        at tokyo.masayuki.image.processing.android.opencvtest.MainActivity.onResume(MainActivity.kt:33)
        at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1456)
        at android.app.Activity.performResume(Activity.java:7614)
        at android.app.ActivityThread.performResumeActivity(ActivityThread.java:4412)
            ... 11 more

[saudet]

You're using a very old version of OpenCV. Please try again with the latest version.

[mogamusa31]

Thank you for your reply.

As you pointed out, I changed javacv to 1.4.4 and javacpp-presets to 4.0.1-1.4.4 by referring to the following javadoc to change the version of each package to the latest version.

https://www.javadoc.io/doc/org.bytedeco.javacpp-presets/opencv/latest/index.html

As a result, I still get the same error.

the commit of changing version https://github.com/mogamusa31/opencv-android-test/pull/4/commits/16f2e9866c878634c45db0ec41d7a50bb87b723c

Error detail

E/cv::error(): OpenCV(4.0.1) Error: Assertion failed (src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols) in Java_org_opencv_android_Utils_nMatToBitmap2, file /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.0.1/modules/java/generator/src/cpp/utils.cpp, line 101
E/org.opencv.android.Utils: nMatToBitmap caught cv::Exception: OpenCV(4.0.1) /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.0.1/modules/java/generator/src/cpp/utils.cpp:101: error: (-215:Assertion failed) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function 'Java_org_opencv_android_Utils_nMatToBitmap2'
D/AndroidRuntime: Shutting down VM
E/AndroidRuntime: FATAL EXCEPTION: main
    Process: tokyo.masayuki.image.processing.android.opencvtest, PID: 13223
    java.lang.RuntimeException: Unable to resume activity {tokyo.masayuki.image.processing.android.opencvtest/tokyo.masayuki.image.processing.android.opencvtest.MainActivity}: CvException [org.opencv.core.CvException: OpenCV(4.0.1) /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.0.1/modules/java/generator/src/cpp/utils.cpp:101: error: (-215:Assertion failed) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function 'Java_org_opencv_android_Utils_nMatToBitmap2'
    ]
        at android.app.ActivityThread.performResumeActivity(ActivityThread.java:4430)
        at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:4470)
        at android.app.servertransaction.ResumeActivityItem.execute(ResumeActivityItem.java:51)
        at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:145)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:70)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2199)
        at android.os.Handler.dispatchMessage(Handler.java:112)
        at android.os.Looper.loop(Looper.java:216)
        at android.app.ActivityThread.main(ActivityThread.java:7625)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)
     Caused by: CvException [org.opencv.core.CvException: OpenCV(4.0.1) /home/travis/build/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.0.1/modules/java/generator/src/cpp/utils.cpp:101: error: (-215:Assertion failed) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function 'Java_org_opencv_android_Utils_nMatToBitmap2'
    ]
        at org.opencv.android.Utils.nMatToBitmap2(Native Method)
        at org.opencv.android.Utils.matToBitmap(Utils.java:123)
        at org.opencv.android.Utils.matToBitmap(Utils.java:132)
        at tokyo.masayuki.image.processing.android.opencvtest.TestJavaCV.testLoadBitmapInDataFolder(TestJavaCV.java:138)
        at tokyo.masayuki.image.processing.android.opencvtest.MainActivity.onResume(MainActivity.kt:33)
        at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1456)
        at android.app.Activity.performResume(Activity.java:7614)
        at android.app.ActivityThread.performResumeActivity(ActivityThread.java:4412)
            ... 11 more

[saudet]

I see you're still not using the right imread() though. Make sure to use the one from the C++ API: http://bytedeco.org/javacpp-presets/opencv/apidocs/org/bytedeco/opencv/global/opencv_imgcodecs.html#imread-java.lang.String-int-

[mogamusa31]

Thank you for your reply.

The error I reported was due to a mismatch between the size in createBitmap and the bitmap size generated in matToBitmap. Sorry I didn't notice

The image of BGR Format was displayed, but when I checked the result of computeSaliency, it did not seem to recognize the object at all. . .

[saudet]

Please show your code. We can't guess what you might be doing wrong without taking a look.