Feature request : 3D Object Detection

[bilelmnasser]

Hello everybody , it will be quiet cool to recognize real objects from a simple 360° scan. ARkit 2.0 support this :dancer:

@inio @gordonbrander @eitanme @richlowenberg @jsantell

[inio]

Indeed, that's a really neat feature, isn't it?

It's among the many potential additions we're looking at.

[izdi]

Hey @inio While it's indeed a neat feature I'd like to discuss a little workaround I use.

Using any known object detection neural nets (SSD, YOLO, TF API) we can get in a separate thread objects from a frame, then pass satisfied results, in this case coordinates of a bounding box center, back onto drawing thread and make a hitTest for anchors in that frame region and hopefully attach our 3d object. While this works OK for some objects on a plain and there are lots of anchors, for objects that lack anchors this won't work. As I understood correctly at this moment ARCore knows distance to the real world object only via anchor hit calculation. So I have a question: what could be the workaround if detected object won't have anchors. In my mind it so:

• find the nearest anchor to the region of the real world object (probably if it's standing on a place then it's in the same distance)
• calculate distance this anchor instead
• use calculated distance to create anchor in the region without anchors

Another issue I have with this approach is placing 3d objects without a plane. Andy is inclined because anchor is not on the plane (see the picture): https://i.stack.imgur.com/JK3Pm.jpg What is the workaround render Andy by straight angle from the anchor ?

[paragbhagwat3575]

this is a good feature to have and will close the gap with Vuforia 3d object detection

[bryanliujh]

Hopefully arcore supports it, would love to use it.

[tony-topper]

Maybe start with migrating the TensorFlow Object detection example app over to using ARcore. That would be a nice way for people to experiment and give feedback.

[joao-carvalheira]

Any updates on this? Is anyone working on it?

[kinster]

Maybe start with migrating the TensorFlow Object detection example app over to using ARcore. That would be a nice way for people to experiment and give feedback.

Looks like image classification (2D stuff) not Objects

[Lucaffo]

@kinster , @tony-topper might he was referring to this

[sergeiromanov]

Any update on this feature?

[jackhub]

Any update on this feature?

[TZECHIN6]

I am working a project exactly what people trying to do in here. Environment: Android Edge device Workflow: Detect custom-labelled object --> use the class and coordinate of bounding box --> render a 3D text/flowing button on top of that object. (real time object detection, not image classification)

This is a integrate work for AI and AR, so far ARCore would be the choice for AR development, meanwhile AI object detection framework is hard to decide. Seems that not all the framework is compatible with ARCore, for example, if I want to use ARCore in Unity, the AI framework needs to be barracuda, no TensorFlowLite is supported (officially). ML Kit only works with image classification model, but not object detection model (YOLO).

Anyone has worked how to integrate both AI & AR information?