Object Detection - Result & Model size torchscript

[ebdjesus]

• Object Detection Hello, I'm running the getImagePrediction method and for any photo it always gives me only 15 results, I've already changed the iOUThreshold value and nothing changes. I tested it with the example of your project and the same thing happens.

• Model size

When we use the model in 960x960 we get this error below, we already tried changing the resolution in the code and it didn't work

[abdelaziz-mahdy]

Currently I am doing a refactor for the whole package to native, to improve size and most probably speed too, while fixing support for iOS

Sadly you will have to wait for it 😢 or you can use latest version of 2.*

And let me know if it works for you

Keep in mind it only works on Android

[ebdjesus]

@zezo357 thanks for attention.

I'm using Flutter and Android for this test.

I'm sorry, but I think I wasn't clear when I sent my questions. The first one is that when I perform inference, only 15 objects are detected. Is it possible to increase this quantity? My second question is that when I use a Yolov5 model with a resolution of 960 and convert it to torchscript, I encounter this error related to the number of classes, but my class list is always the same, with only 1 class. Do I need to change this resolution somewhere in the code? When I use a 640 model, it works fine.

Thank you very much for the quick response!

[abdelaziz-mahdy]

@zezo357 thanks for attention.

I'm using Flutter and Android for this test.

I'm sorry, but I think I wasn't clear when I sent my questions. The first one is that when I perform inference, only 15 objects are detected. Is it possible to increase this quantity? My second question is that when I use a Yolov5 model with a resolution of 960 and convert it to torchscript, I encounter this error related to the number of classes, but my class list is always the same, with only 1 class. Do I need to change this resolution somewhere in the code? When I use a 640 model, it works fine.

Thank you very much for the quick response!

I did understand your question correctly, the 15 is not set my me , and I don't think there is a limit anywhere in code, but I will check it again

And for the resolution one it's a problem from my implemention so as I mentioned I am making a conversation from my current implemention to native one, to avoid using ffi Which will fix this problem

Most probably you won't find it in version 2.* since it was not using ffi

[ebdjesus]

@zezo357 Thanks for the help.

[abdelaziz-mahdy]

@ebdjesus the limit is 15 yes,i can remove it, can you wait for next release? or you need it now?

[ebdjesus]

@zezo357 I wait for the next release. Thank you so much

[diogogonnelli]

Hello everybody! Firstly, your work is incredible on this solution.

@zezo357 For this release where the limit of detected objects will no longer be 15, do you have a priority queue? Or do you have an estimate of when we can use it? I have the same problem as @ebdjesus

Thanks for the contribution!

[abdelaziz-mahdy]

Hello everybody! Firstly, your work is incredible on this solution.

@zezo357 For this release where the limit of detected objects will no longer be 15, do you have a priority queue? Or do you have an estimate of when we can use it? I have the same problem as @ebdjesus

Thanks for the contribution!

i am planning to work on it this weekend, i may release a beta version whenever something has reached stable point

keep in mind the only thing i cant test is ios camera (but hopefully the rest can be tested and ensured it works)

[diogogonnelli]

Excellent! I'll be waiting to test it and give my contribution if possible.

Thanks for the answer.

[abdelaziz-mahdy]

please check version https://pub.dev/packages/pytorch_lite/versions/4.1.0-dev

camera code is tested on android but not tested on ios hopefully it works (will need someone to test it and report its status)

[ebdjesus]

please check version https://pub.dev/packages/pytorch_lite/versions/4.1.0-dev

camera code is tested on android but not tested on ios hopefully it works (will need someone to test it and report its status)

Hello @zezo357, how are doing?

I tested Pytorch using the IOS camera and it didn't work.

Android - Its ok

https://github.com/zezo357/pytorch_lite/assets/51535131/ef76b78e-fd06-46a1-a76f-f13c4f1e346a

Iphone 13 Pro Max - did not work

https://github.com/zezo357/pytorch_lite/assets/51535131/6913e4d7-f3cd-4c22-a7c5-ac865e58921a

Iphone 8 - did not work

https://github.com/zezo357/pytorch_lite/assets/51535131/f614b725-9c6a-405f-9f53-02c464d8c820

[abdelaziz-mahdy]

please check version https://pub.dev/packages/pytorch_lite/versions/4.1.0-dev

camera code is tested on android but not tested on ios hopefully it works (will need someone to test it and report its status)

Hello @zezo357, how are doing?

I tested Pytorch using the IOS camera and it didn't work.

Android - Its ok

https://github.com/zezo357/pytorch_lite/assets/51535131/ef76b78e-fd06-46a1-a76f-f13c4f1e346a

Iphone 13 Pro Max - did not work

https://github.com/zezo357/pytorch_lite/assets/51535131/6913e4d7-f3cd-4c22-a7c5-ac865e58921a

Iphone 8 - did not work

https://github.com/zezo357/pytorch_lite/assets/51535131/f614b725-9c6a-405f-9f53-02c464d8c820

Hello, thank you for testing

Could you please share the iOS logs, since I want to retrace it,

Edit: No need, I was able to get it from the video, will fix it and let you know

[abdelaziz-mahdy]

@ebdjesus

can you test again on 4.1.1-dev, hopefully it should be fixed now https://pub.dev/packages/pytorch_lite/versions/4.1.1-dev

sorry for not being able to test it myself, but i dont have access to ios physical device :(

[ebdjesus]

@zezo357

I tested again with this version 4.1.1-dev on iPhones 8 and 13 and now it works fine.

Now I'm validating the other items I mentioned above

thank you very much for the support and whatever I can help to contribute, you can count on me

https://github.com/zezo357/pytorch_lite/assets/51535131/d067aa9e-b1d6-4aa0-9299-b39fd36c14b4

[ebdjesus]

@zezo357

Has the number of object detection results not yet been adjusted in this release?

I/PytorchLitePlugin(22487): result length before processing 1781 I/PytorchLitePlugin(22487): result length after processing 10

Thanks you

[abdelaziz-mahdy]

@ebdjesus

Adjust the boxes limit in any inference it should take the passed limit,

The default is 10

[abdelaziz-mahdy]

Also android and iOS prediction doesn't give me the same results, I think iOS is wrong for some reason, so until I fix that I won't release the package

But when I do fix it I will release it as stable

You can test the latest dev whenever I get it updated, and thank you so much for letting me know that iOS camera logic works finally ❤️

[ebdjesus]

Also android and iOS prediction doesn't give me the same results, I think iOS is wrong for some reason, so until I fix that I won't release the package

But when I do fix it I will release it as stable

You can test the latest dev whenever I get it updated, and thank you so much for letting me know that iOS camera logic works finally ❤️

Sorry for the lack of attention, I tested it and it's working fine now.

If you need to test something on IOS device, count me in.

The other point is that I have the same problem as @diogogonnelli with classification model.

https://github.com/zezo357/pytorch_lite/issues/43

Thank you very much for your help and have a great weekend ❤️

[abdelaziz-mahdy]

Also android and iOS prediction doesn't give me the same results, I think iOS is wrong for some reason, so until I fix that I won't release the package

But when I do fix it I will release it as stable

You can test the latest dev whenever I get it updated, and thank you so much for letting me know that iOS camera logic works finally ❤️

Sorry for the lack of attention, I tested it and it's working fine now.

If you need to test something on IOS device, count me in.

The other point is that I have the same problem as @diogogonnelli with classification model.

https://github.com/zezo357/pytorch_lite/issues/43

Thank you very much for your help and have a great weekend ❤️

Does the classification problem happen on both android and iOS?

Just to be 100% sure where is the problem.

You too❤️

[ebdjesus]

This happens on Android and iOS

On Android, 95% of the classification returns as object

On IOS 100% as object

I believe I did something wrong when exporting the classification model

Android

IOS

Labels

I'm using this image to test

my code

[abdelaziz-mahdy]

This happens on Android and iOS

On Android, 95% of the classification returns as object

On IOS 100% as object

I believe I did something wrong when exporting the classification model

Android

IOS

Labels

I'm using this image to test

my code

how did you export the model? do you have the code to it?

[ebdjesus]

Hi @zezo357, how you doing?

Sorry for the delay, I was doing some tests before sending it to you.

I would like to first explain the tests I conducted. When I export a model that hasn't been retrained, the classifier works correctly. I took the Yolov5 classification model "yolov5s-cls.pt" as an example, which was trained with the ImageNet dataset of 1000 classes. This model functions correctly when exported.

The code used was this:

model = torch.hub.load('C:\inetpub\wwwroot\RI_object_cls\yolov5', 'custom', 'yolov5s-cls.pt', source='local') model.cpu()

example = torch.zeros((1, 3, 224, 224)) traced_script_module = torch.jit.trace(model, example) optimized_traced_model = optimize_for_mobile(traced_script_module) optimized_traced_model._save_for_lite_interpreter("model.pt")

However, when I train a custom Yolov5 classification model and export it in the same way, the classification always returns the wrong class, regardless of the image used.

The folders of the trained classes are organized as follows

[abdelaziz-mahdy]

Hi @zezo357, how you doing?

Sorry for the delay, I was doing some tests before sending it to you.

I would like to first explain the tests I conducted. When I export a model that hasn't been retrained, the classifier works correctly. I took the Yolov5 classification model "yolov5s-cls.pt" as an example, which was trained with the ImageNet dataset of 1000 classes. This model functions correctly when exported.

The code used was this:

model = torch.hub.load('C:\inetpub\wwwroot\RI_object_cls\yolov5', 'custom', 'yolov5s-cls.pt', source='local') model.cpu()

example = torch.zeros((1, 3, 224, 224)) traced_script_module = torch.jit.trace(model, example) optimized_traced_model = optimize_for_mobile(traced_script_module) optimized_traced_model._save_for_lite_interpreter("model.pt")

However, when I train a custom Yolov5 classification model and export it in the same way, the classification always returns the wrong class, regardless of the image used.

The folders of the trained classes are organized as follows

I am still having problems matching android and iOS results, I can't figure out why it give me wrong results when I used the same logic as ios-demo from pytorch, so it will take me a while to figure out what and why is wrong

Whenever I fix it I can trace why the yolov5 classifier fails, can you test it on python? Just Incase to make sure the model it self is not corrupted, or it's not extracted correctly somehow!

And if you can provide the model mean and std that will be helpful

[ebdjesus]

Hello @zezo357 Yes, I performed the tests in Python, both with the original Yolo model and the exported model. The tensor results were exactly the same:

original Yolo: tensor([[ 1.61764, -1.07584, -0.65396]]) converted model: tensor([[ 1.61764, -1.07584, -0.65396]])

When we tested it in Flutter, the tensor result was very different:

To load the image in Flutter and apply inference, I use the following code below:

I calculated the mean and the std, and they are as follows:

mean: [0.39676202, 0.38142911, 0.33674075] std: [0.26444618, 0.25301057, 0.24330125]

I'm sending my code and how the tests were done, in case you want to replicate.

https://we.tl/t-9GPsGeTmI3

[abdelaziz-mahdy]

I couldn't find the cause of it, so I am going to expose the layers of infernnce and expose a dart algorithm for normalization and resizing, and hopefully we make it run correctly,

If that worked I may leave it since it will ensure both android and iOS share the same results

I am currently working on it and if all is well, I will update the release with it

[abdelaziz-mahdy]

Hello @zezo357 Yes, I performed the tests in Python, both with the original Yolo model and the exported model. The tensor results were exactly the same:

original Yolo: tensor([[ 1.61764, -1.07584, -0.65396]]) converted model: tensor([[ 1.61764, -1.07584, -0.65396]])

When we tested it in Flutter, the tensor result was very different:

To load the image in Flutter and apply inference, I use the following code below:

I calculated the mean and the std, and they are as follows:

mean: [0.39676202, 0.38142911, 0.33674075] std: [0.26444618, 0.25301057, 0.24330125]

I'm sending my code and how the tests were done, in case you want to replicate.

https://we.tl/t-9GPsGeTmI3

I can't download the code right now, but does it include the python code for testing? If not, can you add it

[ebdjesus]

Yes, the zip file contains a Python notebook with the conducted tests and the models used. I'll be waiting, thank you!

[abdelaziz-mahdy]

Thank you, I am sorry its taking sometime, will let you know when I am done with it

[ebdjesus]

Thank you, I am sorry its taking sometime, will let you know when I am done with it

@zezo357 Don't worry and thanks for your patience

[abdelaziz-mahdy]

fixed in 4.2.0 model size and the output boxes limit

for yolo classifier will leave the other issue open until we figure out what is the problem

[aprashantz]

Detection rarely worked in iOS using 4.2.5. After resizing or compressing image, it worked in iOS too.

[abdelaziz-mahdy]

Detection rarely worked in iOS using 4.2.5. After resizing or compressing image, it worked in iOS too.

Can you elaborate more?

[aprashantz]

Detection rarely worked in iOS using 4.2.5. After resizing or compressing image, it worked in iOS too.

Can you elaborate more?

• Using the package in Flutter 3.13.4
• I am using startImageStream of camera controller and passing the CameraImage object to the getCameraImagePrediction method. On Android it works fine for my custom model. But running on iOS device, prediction mostly returns empty array.
• Not sure whether my trained model is not being able to use image from the iOS or is there changes required in the Pytorch lite.