Please Add MediaPipe Objectron 3D detection

asus4 / tf-lite-unity-sample

TensorFlow Lite Samples on Unity

867 stars 254 forks source link

Please Add MediaPipe Objectron 3D detection #142

Closed nenenkosi closed 2 years ago

nenenkosi commented 3 years ago

The repo does not provide an easy way to use objection on the application it would be super nice to have the objectron example on the repo as it would provide a nice and easy way to integrate your on custom mode with objectron.

This feature would allow developers to do more than what the repo providers it will also help in understanding how 3D is used in mediapipe to Draw and understand the bounding boxes of a 2D image into 3D image.

asus4 commented 3 years ago

Hi @nenenkosi ,

The output of the Objectron model is 2D keypoints. In MediaPipe, they use Eigen library to convert 2D keypoints to 3D.

objectron_network_architecture https://google.github.io/mediapipe/solutions/objectron

And I hesitate to install such a big library for just using PnP solver in this sample project.

If you just need 2d keypoints, this Objectron implementation in Barracuda might be helpful.

nenenkosi commented 3 years ago

Thanks for the reply if it happens that you decide to include it please let us know coz it would make our life much easy thanks

olokobayusuf commented 3 years ago

Hey @asus4 @nenenkosi , I'm also looking at adding Objectron support to NatML. Would you be interested in collaborating to implement the EPnP algorithm with just C# (and maybe the Burst compiler)?

nenenkosi commented 3 years ago

@olokobayusuf I don’t mind if you would do such you would be helping the community a lot and I would be highly appreciated

asus4 commented 3 years ago

@olokobayusuf Yes, would be nice to have a PnP solver library for Unity!

asus4 commented 3 years ago

The OpenCV for Unity plugin also has a PnP solver method. So would it be a good starting point to make an example with it?

olokobayusuf commented 3 years ago

The OpenCV for Unity plugin also has a PnP solver method. So would it be a good starting point to make an example with it?

I explicitly don't want to add an OpenCV dependency, because it's massive and quite expensive. We can probably find a barebones EPnP implementation and port it over to C#.

nenenkosi commented 3 years ago

@olokobayusuf if you need any kind of help or even someone to test I am available and if you run into any kind of problems I think you can feel free to ask @asus4 to assist coz he once did the EPnP implementation but it was for work so he might be able to assist you to solve bugs faster

olokobayusuf commented 3 years ago

Just wanted to update this thread. Objectron is almost implemented (using NatML instead of TFLite):

All that's left is lifting the 2D points to 3D using EPnP.

nenenkosi commented 3 years ago

@olokobayusuf thanks for the update a quick question is NatML better than TFLite in terms of speed and accuracy ?

And when do you think it will be available for testing coz am very excited to try it out and thanks for updating us on this thread

olokobayusuf commented 3 years ago

thanks for the update a quick question is NatML better than TFLite in terms of speed and accuracy ?

They should be comparable on all platforms. NatML uses CoreML on iOS and macOS; NNAPI on Android; and DirectML on Windows.

And when do you think it will be available for testing coz am very excited to try it out and thanks for updating us on this thread

No ETA yet but I'll keep you updated as I go along. I'm also looking forward to completing this.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.