Custom object detection model giving wrong predictions on Android

DeepakSinghRawat commented 2 years ago

Hi,

I am trying to use a custom object detection model in the example app provided by you. I am using the latest version of google_mlkit_object_detection: ^0.4.0 plugin.

I replaced the object_labeler.tflite model in the example app with a custom model. I want to get predictions using camera live feed. The model is predicting correct labels on iOS. But for some reason it's not predicting correct labels on Android. I am testing on Android 10. On Android it's predicting random values. If I use a single image from the gallery or take a picture then I am getting good predictions on Android also.

I used the same model in the ML Kit Vision Quickstart Sample Android App and it's giving correct predictions.

My current settings:

    final options = LocalObjectDetectorOptions(
      mode: mode,
      modelPath: modelPath,
      classifyObjects: true,
      multipleObjects: true,
    );

CameraController(
      camera,
      ResolutionPreset.high,
      enableAudio: false,
    );

I have tried to change ResolutionPreset to different values but nothing is working. Please guide me on how to resolve this issue.

fbernaly commented 2 years ago

@DeepakSinghRawat: could you share your custom model, I will take a look.

DeepakSinghRawat commented 2 years ago

@fbernaly Please download the model from here.

I trained a fruit detection model using a pre-trained tf.keras.applications.MobileNetV3Large model. This model has in-built image normalization layer so I disabled normalization in TFLite model's metadata by setting mean=0 and std=1. This model is able to detect fruits like banana, tomato, apple, coconut in MLKit's android app but it's giving random predictions using your example app. I hope you will be able to point me in the right direction. Please let me know if you need any other information.

fbernaly commented 2 years ago

@DeepakSinghRawat : I created this branch with you model: https://github.com/bharat-biradar/Google-Ml-Kit-plugin/tree/fruits_model

Give it a try.

I am not an expert in Machine Learning or trying models. I cannot tell you if the model is well trained or not, but so far the testing I did with my Pixel4 I am getting this:

Screenshot_20220602-151324

Screenshot_20220602-151309

DeepakSinghRawat commented 2 years ago

Dear @fbernaly Thank you for taking a look at the issue I am facing. I tried your repo on a few real fruits but got wrong predictions. For example here are the predictions from your flutter app and mlkit's android app. Flutter app is predicting apple which is wrong. mlkit's android app is predicting tomato for the same image which is correct. I even changed multipleObjects: false in the flutter app but still got wrong predictions most of the time. For the same images, the flutter app on iPhone is working fine. I am facing this issue only on Android.

I have created a new repo with mlkit's android app and my model in case you want to install it and compare the results. Here's the link Launch the app and choose: Run the ML Kit quickstart written in Java -> LivePreviewActivity -> Custom Object Detection

Original Image

original_tomatoes

Flutter App Prediction

flutter_app

MLKIT Android App Prediction

android_app

Heech commented 2 years ago

I'm having a similar problem when trying to use our custom model in the object detection example project. Seeing the same results as when the default assets/object_labeler.tflite model is used. Even when changing to a non-existing file path the app gives no errors but showing same results regardless.

minhquangkid commented 6 months ago

@DeepakSinghRawat: I don't know how to trained a custom detection model using a pre-trained model as you did, please show me the way to do it, thank you!

flutter-ml / google_ml_kit_flutter