Unity support for MediaPipe?

[ROBYER1]

Our main issue when developing in Unity with ARCore/ARKit is limited device reach due to the amount of devices out there that don't support ARCore/ARKit.

Objectron and Instant Motion Tracking looks absolutely fantastic, and to my understanding doesn't require ARCore(?), are there any plans to make a plugin for Unity? The use-cases on older phones that don't support ARCore and pairing with Unity 3d animation/code are limitless and I am very keen on trying this out in Unity.

Unity forum post asking about this: https://forum.unity.com/threads/ar-foundation-support-for-3d-object-detection-with-mediapipe.1016224/

[TigerHix]

Agreed—this can bring a lot of people to use mediapipe in their VR/AR projects, many of which are built with Unity.

[ROBYER1]

Agreed—this can bring a lot of people to use MediaPipe in their VR/AR projects, many of which are built with Unity.

Good point, as a mixed reality developer, I wouldn't even dream of starting an AR project without Unity at this point. The platform has great support, getting started materials and asset management/optimisation in the engine for AR is second to none. Wasted a lot of time trying to craft AR projects using other software/frameworks so would love to see MediaPipe in Unity for object detection + motion tracking at the very least!

[ROBYER1]

Please close if need be as this is a duplicate of https://github.com/google/mediapipe/issues/36

The use case for me is Instant Motion Tracking without the need for initialisation/ARCore/ARKit hardware requirements

• The user journey of wiggling the phone around for AR initialisation is not simple for some of our users to grasp (even still)
• Running of Objectron object detection would bypass some strict limitations of Image/object tracking used already in ARCore/ARKit - it is quite challenging to detect some objects by shape so we would prefer to train them using ML with Objectron or Object Detection YOLO style to track them with box detection or Objectron tracking
• The most obvious use case is broader device support, as not all of the users we want to target have ARCore support, with around 50% device coverage of ARCore support and 90% device coverage for ARKit we are missing a LOT of end users if we want them to use our AR experience

[midopooler]

Any conclusion to this thread? I actually needed a human segmentation feature in unity (but for mobile applications)

[FerLuisxd]

+1, Having an API for Unity would be really nice to make 3D applications

[FerLuisxd]

@ROBYER1 BTW, you couldn't complete your task with https://github.com/homuler/MediaPipeUnityPlugin ?

[ROBYER1]

@ROBYER1 BTW, you couldn't complete your task with https://github.com/homuler/MediaPipeUnityPlugin ?

It's nearly there, instant motion tracking support on IOS isn't working yet there sadly

[sgowroji]

Hi @ROBYER1, Could you please provide more information regarding the above issue if you still looking resolution for it. Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[Thaina]

We are still waiting for official unity package. Or the confirmation that google won't do it

[ROBYER1]

You could be waiting a while 😂😭

[gkiernozek]

Is https://github.com/homuler/MediaPipeUnityPlugin not officially supported?

[Thaina]

Is https://github.com/homuler/MediaPipeUnityPlugin not officially supported?

I think that was independent dev owned repo that not maintain by google isn't it?

[ilmarioranen]

I'd also REALLY love to see this supported in an official Unity repo / asset maintained by Google. It would mean MediaPipe is accessible to the biggest AR/XR/VR developer community on the planet and thus would make a huge difference.

Granted such a huge community might require additional attention and effort from Google, but why make this code public at all if the goal is not to make is accessible to as many people as possible?

[kotran88]

i'm now really into implementing objectron on android device with s https://github.com/homuler/MediaPipeUnityPlugin repo but it's really hard to get there ...

[manwithsteelnerves]

We recently release Easy ML Kit which targets to get google's ML Kit, Media Pipe and Tensorflow in Unity. Currently we released 2 features (Barcode scanning and object detection) and others are in developement.

We have ARFoundation Camera as one of the input sources which will be quite useful for AR Apps. Would be great if you can share us what features you are expecting so we can prioritize.

[TZECHIN6]

Hi @manwithsteelnerves , at this moment, seem that the custom object detection is not ready yet, and only 5 classes can be detected.

Actually i am working on this idea, the more I read in Google, the more I got confused... Assume Unity is a must in my use case (for better graphic rending and flexibility of working on animation), which AI pipeline should I use for detecting the custom object and getting their corresponding location?

For some Google library likes ARCore, ML Kit, Mediapipe, Easy ML Kit, TensorFlowLite... which framework should be used for my application?

[Thaina]

@TZECHIN6 Currently ARCore has its own session system to access hardware and fabricate many system together internally. It very rigid and I don't think it could integrate with mediapipe easily. Given that mediapipe itself also very rigid too

Given that ARCore require specific hardware to be able to work. I think mediapipe is more usable

[manwithsteelnerves]

@TZECHIN6 for all the features we developed(barcode, object detection, text recognition and face), we didn't notice any issues using ar camera as input for processing. Regarding custom object detection model, thats something we will start on once we finish all features.

Easy ML Kit wraps all within unity and you just need to use our api alone.

[moheydy]

After many years still you don't support of Unity 😐, I'm sure The number of Unity developers who need mediapipe is 5 times that of Java and Android Studio ,objectron and body detection is really need for Ar and Xr Applications

[ys1317]

so does anyone find a solution for that ( human body occlusion )

[Thaina]

It seem like the new mediapipe still not finish upgrade full body human yet