search

BlueMirrors / cvu

Computer Vision deployment tools for dummies and experts. CVU aims at making CV pipelines easier to build and consistent around platforms, devices, and models.
https://pypi.org/project/cvu-python/
Apache License 2.0
88 stars 19 forks source link

Any Idea on mAP value? #17

Closed jaskiratsingh2000 closed 3 years ago

jaskiratsingh2000 commented 3 years ago

Do you have any idea about this? I am struggling past 1 month on my research due to this - https://docs.google.com/document/d/1ej8hIktTFmsqTH0GCYVw-qD8sF9C9fa_Pe61EenNmkU/edit?usp=sharing

Please let me know if you have any idea about this. Thanks!

shivamswarnkar commented 3 years ago

Hi @jaskiratsingh2000 did you try to run inference on single image, just to see if output boxes are fine? (using detect.py from ultralytics/yolov5).

Also, did you match torch version between server and raspi?

jaskiratsingh2000 commented 3 years ago

Hi @shivamswarnkar Firstly, Thank you so much for replying. I actually didn't test the interference on single image. Should I do that and let you know here?

and regarding the torch version those are same.

shivamswarnkar commented 3 years ago

Yeah @jaskiratsingh2000, please let me know if output matches (feel free to share outputs here if it's not sensitive info). I would recommend to look at both the output image and box coordinates, just to make sure that there isn't any precision loss.

Got it (about torch). Just a follow up, are you using any compute sticks?

jaskiratsingh2000 commented 3 years ago

Hi @shivamswarnkar I tested the interference and I got to know that when I tested that on Server there were bounding boxes in the image and when I tested on Raspberry Pi there were no bounding boxes in the image in the Raspberry Pi.

Please help me out in resolving this @shivamswarnkar Would highly appreciate your response. Thanks

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar Please let me know that what could be an issue?

shivamswarnkar commented 3 years ago

@jaskiratsingh2000 Can you run a similar test with the pre-trained weights (yolov5s, it'll download itself when you run detect.py without specifying the weights)? If possible, maybe make a fresh clone of the Yolov5 repo and run detect.py (just to confirm if something is wrong with weights, code, or raspi-system).

jaskiratsingh2000 commented 3 years ago

Yes so @shivamswarnkar I already ran this test with both the weights that is with pre-trained (yolov5s.pt) and weights that I got after training

shivamswarnkar commented 3 years ago

If you didn't make any changes to the original code base then I think it's something with installations of libraries. Have you tried the tflite version of Yolov5?

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar No, I haven't tried the tflite version of Yolov5. Can you let me know how to go with that?

shivamswarnkar commented 3 years ago

You can checkout submissions in the Raspi export. Alternatively you can use this repo to convert pt weights to tflite weights and run a sample detect.py inference.

We are also working on our own tflite support but it'll probably take some more time.

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar I actually checked the submissions from the Export but none of them has implemented the mAP value mechanism. Moreover, the repo you shared has linkage with android so I am bit confused that how to actually go with that.

shivamswarnkar commented 3 years ago

I would recommend to use those repos to confirm if single image inference is working. Once that's confirmed, you can modify the mAP script to work with tflite.

Tflite conversion part is independent of android. You can use the exported tflite with that repo's version of detect.py to run inference.

You can use any repo. Point is to make sure if libraries are the culprit. If you don't get acceptable results with tflite then something is probably wrong with raspberry Pi itself (which would be weird)

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar Let me get back to you in the morning with all the details. Like libraries info, weights info etc in detail. Would that be okay?

shivamswarnkar commented 3 years ago

Yeah that works, no problem. I would still recommend to try out tflite route because I don't have access to Raspi right now and so I probably won't be able to confirm any library related issues (specially related to pytorch). I have tried with tflite in past and it worked fine so I'm hoping it'll work for you as well.

shivamswarnkar commented 3 years ago

Hi @jaskiratsingh2000,

Please let me know if this issue is no longer relevant and if I can close it now.

jaskiratsingh2000 commented 3 years ago

Hey! I am still working on it. Please keep it open so that we can discuss here. I'll come up with an update here

On Mon, Sep 6, 2021 at 9:56 PM Shivam Swanrkar @.***> wrote:

Hi @jaskiratsingh2000 https://github.com/jaskiratsingh2000,

Please let me know if this issue is no longer relevant and if I can close it now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-913759258, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHK567HERRNZ5L34VI6LUATTVBANCNFSM5C4EZV3A .

shivamswarnkar commented 3 years ago

Sure. I'll keep it open, thanks for letting me know.

jaskiratsingh2000 commented 3 years ago

Hi @shivamswarnkar I just want to ask that I am trying to install Tensorflow on my Raspberry Pi as we discussed previously but everything went well hence I am getting this following error. Not sure why because I have tried all the possibilities

error:

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

@shivamswarnkar Can you help with this if possible?

shivamswarnkar commented 3 years ago

@jaskiratsingh2000 what's your python version and NumPy version?

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar My Python version is 3.7.3 and the Numpy version is 1.19.5. I followed all the steps mentioned here https://qengineering.eu/install-tensorflow-2.6-on-raspberry-64-os.html

jaskiratsingh2000 commented 3 years ago

Do you have any idea? Please let me know. I am kinda stuck.

On Tue, Sep 7, 2021 at 2:46 PM Shivam Swanrkar @.***> wrote:

@jaskiratsingh2000 https://github.com/jaskiratsingh2000 what's your python version and NumPy version?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-914137145, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHKYYWJTKQDEFJ2GPDATUAXJ6RANCNFSM5C4EZV3A .fo you have any

shivamswarnkar commented 3 years ago

@jaskiratsingh2000 sorry, I was busy with something else. From what I can gather from StackOverflow, most people solved it by reinstalling the NumPy. If that doesn't help then share your Raspi details to debug it further.

Btw, if you are trying the tflite version, then I think you don't need the whole TensorFlow. You should just need tflite_runtime.

Side note, we are also planning to release our tflite_runtime version in the upcoming week (it'll only require opencv, numpy and tflite_runtime). If your issue doesn't get solved by that time then perhaps you could try CVU.

jaskiratsingh2000 commented 3 years ago

Hi @shivamswarnkar Thank you very much for responding back. I was able to resolve this issue. Now my main goal is to check the mAP value on RPi with this tflite version. I tried this exported version https://github.com/karanjakhar/yolov5-export-to-raspberry-pi but it does not have any mAP value mechanism which could provide me with the accuracy performance on RPi. Can you help in this if possible? That would really be great if possible.

shivamswarnkar commented 3 years ago

I guess now you'll have to edit the official yolov5 repo's code to work with tflite model. Just replace all the PyTorch model-related code with tflite version and that should do it. Though, it'll require some manual work

jaskiratsingh2000 commented 3 years ago

I actually tried but couldn't make through it.😔 Can you help me with that in some guidance because my end goal is just to get the mAP value and I am not able to figure out that where is the mAP function and where actually to replace things.

On Thu, 9 Sep 2021, 1:10 am Shivam Swanrkar, @.***> wrote:

I guess now you'll have to edit the official yolov5 repo's code to work with tflite model. Just replace all the PyTorch model-related code with tflite version and that should do it. Though, it'll require some manual work

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-915517727, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHK7SSRUJ7KABRPRZSLTUA634FANCNFSM5C4EZV3A .

shivamswarnkar commented 3 years ago

I can try but it might take some time (I have a very busy work schedule for this entire month, and I'm not familiar with that raspi repo). Can you share the link to the file you're using to evaluate your model? I'll try to see if there's an easy way.

shivamswarnkar commented 3 years ago

Btw, just to clarify, you are able to get proper object detection on Raspi now (even if for a single image), right?

jaskiratsingh2000 commented 3 years ago

I am just using original yolov5 val.py script which is used to evaluate the models. Since I am working as an intern but this thing is just out of my scope but anyways had to go through it so kinda frustrated after being stuck on this mAP value past 1.5 months.

Moreover, there is models/tf.py which do the almost same thing but it doesn't have way to show mAP value because val.py has only been made for pytorch not tflite

On Thu, 9 Sep 2021, 1:21 am Shivam Swanrkar, @.***> wrote:

Btw, just to clarify, you are able to get proper object detection on Raspi now (even if for a single image), right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-915523885, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHKYW42IRFNYF4JM447TUA65CXANCNFSM5C4EZV3A .

shivamswarnkar commented 3 years ago

Oh I see now. I'll try to see if I can find a way to make it work for you.

If not and val.py is useful, we might just put a PR or new repo for val.py with CVU (with tflite). I'll see if someone from our team is interested in it.

jaskiratsingh2000 commented 3 years ago

Thank you so much. Can I know that how long would it take? I really don't wanna put myself into the hole for another weeks.

On Thu, 9 Sep 2021, 1:30 am Shivam Swanrkar, @.***> wrote:

Oh I see now. I'll try to see if I can find a way to make it work for you.

If not and val.py is useful, we might just put a PR or new repo. I'll see if someone from our team is interested in it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-915529559, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHK3NPBGKUVEWQQQAFRLUA66FTANCNFSM5C4EZV3A .

shivamswarnkar commented 3 years ago

If I found an easy way to do it then it should be done by the weekends.

However, for creating the val.py ourselves, It's hard to say. But I don't think it'll be anything less than a few weeks. We already have a lot of stuff to get done.

jaskiratsingh2000 commented 3 years ago

Okay but the changes are to be required to be done in val.py directly right? We don't have to write val.py from scratch?

On Thu, 9 Sep 2021, 1:44 am Shivam Swanrkar, @.***> wrote:

If I found an easy way to do it then I should be done by the weekends.

However, for creating the val.py ourselves, It's hard to say. But I don't think it'll be anything less than a few weeks. We already have a lot of stuff to get done.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-915538149, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHK7RBSHC6IKW3PAS44TUA672BANCNFSM5C4EZV3A .

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar Here is the PR for the reference https://github.com/ultralytics/yolov5/pull/1127

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar Did you get a chance to look into the above? Please let me know

jaskiratsingh2000 commented 3 years ago

Hey when will you reply?

On Thu, Sep 9, 2021 at 1:42 AM Jaskirat Singh @.***> wrote:

Okay but the changes are to be required to be done in val.py directly right? We don't have to write val.py from scratch?

On Thu, 9 Sep 2021, 1:44 am Shivam Swanrkar, @.***> wrote:

If I found an easy way to do it then I should be done by the weekends.

However, for creating the val.py ourselves, It's hard to say. But I don't think it'll be anything less than a few weeks. We already have a lot of stuff to get done.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-915538149, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHK7RBSHC6IKW3PAS44TUA672BANCNFSM5C4EZV3A .

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar Hey! I am still waiting on to hear back from you. Please let me know that would really be helpful

shivamswarnkar commented 3 years ago

@jaskiratsingh2000 Hi, I had to take my vaccination dose so I wasn't able to reply to you sooner. I took a quick look at the val.py and I guess all you have to do is to replace all relevant code of model loading and inference. To get started, I would recommend reading TFLite's detect.py, it'll show you major differences between PyTorch and TFLite model loading/inference. Everything else mostly should remain unchanged. If you face any specific issues, then I can try to help you out.

We do plan to release something similar to val.py as a supplementary for the CVU test, but it's not on priority so I can't say when we'll be able to release it.

jaskiratsingh2000 commented 3 years ago

@shivamswarnkar I tried something similar quite a while back but couldn't figure out what or exact which changes to make. If you can give it a shot that would really be great @shivamswarnkar

Also, I am not able to find the TFlite detect.py earlier I could remember that it was achieved from tf.py model itself not sure what to do

shivamswarnkar commented 3 years ago

This has tflite inference code, this might help you. https://github.com/zldrobit/yolov5/blob/tf-android/detect.py

I am already preoccupied with other work, so I don't think I'll be able to work on val.py anytime soon.

jaskiratsingh2000 commented 3 years ago

Okay! Can you let me know exactly again that where to make changes and what to replace specifically?

On Tue, Sep 14, 2021 at 5:47 PM Shivam Swanrkar @.***> wrote:

This has tflite inference code, this might help you. https://github.com/zldrobit/yolov5/blob/tf-android/detect.py

I am already preoccupied with other work, so I don't think I'll be able to work on val.py anytime soon.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BlueMirrors/cvu/issues/17#issuecomment-919094737, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOBHK4HLHMUPY6FPM3KEELUB44OVANCNFSM5C4EZV3A .

shivamswarnkar commented 3 years ago

I can't give you exact lines, but just as we have been discussing, you'll have to read the file and see where pytorch model is getting loaded and performing inference, and replace those parts with tflite parts.

shivamswarnkar commented 3 years ago

Since it's not active and not relevant to the repo as of yet, I'll close it for now. Please feel free to reopen it if something changes.

jaskiratsingh2000 commented 3 years ago

Hey @shivamswarnkar As you were mentioning that you would probably work upon finding the mAP value for the TFlite model. Were you able to do that?