Wrong Classification with custom model

[diogogonnelli]

Hello everyone.

I'm testing the implementation with a model trained by myself, and I'm facing some issues during the classification process. I had to modify the code to export the YOLO classification model, and I did it like this:

import torch
import os
from torch import *
from torch.utils.mobile_optimizer import optimize_for_mobile

dev = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

model = torch.hub.load('main_folder', 'custom', 'yolov5s-cls.pt', source='local')
model.cpu()

example = torch.zeros((1, 3, 224, 224))
traced_script_module = torch.jit.trace(model, example)
optimized_traced_model = optimize_for_mobile(traced_script_module)
optimized_traced_model._save_for_lite_interpreter("model.pt")

It seems like the export is working, but when I use it in Flutter, it always gets classified with the wrong class.\

In my case, my model was trained with only 3 classes, and I updated your code to accommodate this number of classes.

I also calculated the mean and standard deviation based on my dataset.

In addition to changing the number of classes in the code, do I need to make any other modifications in the code to use a custom model with a different number of classes?

[abdelaziz-mahdy]

Can you share your code, Also did you update the labels.txt?

And for yolo no need to change the model the mean and std if you're using the default values, so can you test with the default?

[abdelaziz-mahdy]

Can you share the model and test image if you can't share the code?

Since i need something to test with if classification is wrong

[diogogonnelli]

Certainly, I will provide the codes, model, and classes along with the image I am testing. model.zip

Yes, I tested it with the mean and standard deviation values as well.

For this attached image, the return class should be 'andorinha', but regardless of which image I use, it always returns the class 'objeto'.

Here are some parts of my code.

https://github.com/zezo357/pytorch_lite/assets/34580126/769e4529-11a3-493c-bbe7-30b4384f5e5f

[abdelaziz-mahdy]

Okay I see that all of your codes are correct, but I can't figure out why the results are not correct,

Anyway I am working on a new release for iOS support and changing how the package reads and gets results so can you wait for new release?

When I do I can further test why the classification is not correct, and improve it if I can.

[diogogonnelli]

Of course, I will wait and conduct new tests with the new build. Thank you for your attention!

[diogogonnelli]

Hello @zezo357, were you able to reproduce my code and test it? Do you need any additional information? If you can help with anything, please let me know, as I ran some tests and still have the same result. Thanks!

[abdelaziz-mahdy]

I was not free lately, thank you for reminding me to test again sry for the wait

I did try some stuff and changed variables without any useful results, so I will try some different normalization and see what works and let you know, still no promises that I will be able to make it work

[diogogonnelli]

Meanwhile, I will continue running some tests. If there is progress, I will let you know.

To improve reproduction, here are the codes used, original model, and converted model, along with the output tensors.

https://drive.google.com/file/d/149grCV5f6vQkXw9j4xoDufcQN6LW3z1I/view?usp=share_link

[abdelaziz-mahdy]

Meanwhile, I will continue running some tests. If there is progress, I will let you know.

To improve reproduction, here are the codes used, original model, and converted model, along with the output tensors.

https://drive.google.com/file/d/149grCV5f6vQkXw9j4xoDufcQN6LW3z1I/view?usp=share_link

i have sent a request to access, hopefully i can figure something this weekend

[diogogonnelli]

Meanwhile, I will continue running some tests. If there is progress, I will let you know. To improve reproduction, here are the codes used, original model, and converted model, along with the output tensors. https://drive.google.com/file/d/149grCV5f6vQkXw9j4xoDufcQN6LW3z1I/view?usp=share_link

i have sent a request to access, hopefully i can figure something this weekend

Access granted. Any news let me know. Thanks

[abdelaziz-mahdy]

can you guys try on this branch?

it maybe be the fix for the problem not sure, but lets see

  pytorch_lite:
    git: 
      url: https://github.com/zezo357/pytorch_lite
      ref: fix-classification-bug

[abdelaziz-mahdy]

Also sorry for asking but, can you provide a GitHub repo i can test on, instead of asking you to test?

Just to be more efficient while testing and having the same configuration

[ebdjesus]

@zezo357 hello my friend!

How you doing?

I have shared with you the code we used to test pytorch_lite

https://github.com/ebdjesus/ri_novo/

[abdelaziz-mahdy]

@zezo357 hello my friend!

How you doing?

I have shared with you the code we used to test pytorch_lite

https://github.com/ebdjesus/ri_novo/

thank you will test and let you know if new fix is enough

[ebdjesus]

@zezo357 hello my friend! How you doing? I have shared with you the code we used to test pytorch_lite https://github.com/ebdjesus/ri_novo/

thank you will test and let you know if new fix is enough

I'm sending the images in case I need to test with this model we generated

[abdelaziz-mahdy]

@ebdjesus i cant find a way to test the model? the code is not using it?

the camera icon prints the path, but doesnt run the classification function

[abdelaziz-mahdy]

can you let me know how to run the model in your app ?

[ebdjesus]

After capturing the image, you need to click the + button to run the classification.

[abdelaziz-mahdy]

i just tested on the package example (just replacing model and labels files )

all of the results are correct? where is the problem?

[abdelaziz-mahdy]

my function

    final XFile? image = await _picker.pickImage(source: ImageSource.gallery);
    print(image!.path);
    Stopwatch stopwatch = Stopwatch()..start();

    String textToShow = await _imageModel!
        .getImagePrediction(await File(image.path).readAsBytes());
    textToShow = "${textToShow ?? ""}, ${inferenceTimeAsString(stopwatch)}";

[ebdjesus]

this image is gallo label its correct

Now try with this image that is andorinha label

[abdelaziz-mahdy]

this image is gallo label its correct

Now try with this image that is andorinha label

check my comment i did try the 3 labels

[abdelaziz-mahdy]

After capturing the image, you need to click the + button to run the classification.

it doesnt run the classification it runs object detection

[abdelaziz-mahdy]

this image is gallo label its correct

Now try with this image that is andorinha label

also i tested this image and it works too ?? ?

[abdelaziz-mahdy]

in your code you have

i am thinking this is the problem, since using the default works correctly

[ebdjesus]

@zezo357 all your tests are correct

I downloaded your code and included the branch you provided and the models and label I trained, but I was unsuccessful.

[abdelaziz-mahdy]

Did you close app and doing a pub get and rerun it after changing the models?

Please set it model name as different name to make sure it's replaced correctly

[abdelaziz-mahdy]

https://github.com/zezo357/pytorch_lite/tree/classifer-test this is the code i tested on and showed the results to you

[ebdjesus]

I deleted the .pub-cache and tried to clear everything, changed the name as you said and nothing. I was creating a new project to import your classes.

[abdelaziz-mahdy]

The model I used is the one in your repo, and it works correctly,

And I did give you the branch I tested on, this is weird

Please can you try on the branch I provided?

[ebdjesus]

I'll test it now with him. I'll get back to you.

Thanks

[ebdjesus]

@zezo357

I have now tested it directly in my project and it is working very well. The problem was that @diogogonnelli had given me the old classification file, I'm going to kick his ass here ;D Once again thank you very much for your support

[abdelaziz-mahdy]

Glad to help and good luck , also don't be too harsh on him we all mess up sometimes. All we can do is learn from mistakes.

[ebdjesus]

I'm just kidding, he's largely responsible for us having this functionality today and you for making it possible to run this in the app.

[abdelaziz-mahdy]

I'm just kidding, he's largely responsible for us having this functionality today and you for making it possible to run this in the app.

Hopefully all goes well, and don't hesitate to ask for assistance if needed