search

google-ar / arcore-android-sdk

ARCore SDK for Android Studio
https://developers.google.com/ar
Other
4.95k stars 1.22k forks source link

Android (Java or C++) API Feature Request: 2D keypoints, image descriptors #1114

Open brianm-sra opened 4 years ago

brianm-sra commented 4 years ago

Feature request:

ARCore is internally detecting points of interest in each camera frame image in order to determine 3D location of points in PointCloud, locations of Planes, tracking Anchors, etc.

But the API does not yet expose these internal 2D (pixel on screen) locations or image descriptors.

Developers and researchers working with computer vision and AI etc. could take the image frames and pass them to another library like OpenCV etc. for running computer vision algorithms.

Instead of recomputing keypoints/descriptors from images, It might be more efficient to reuse lower level informatiom already determined by ARCore but not currently exposed by the API if it was added in a future release (Android or Android NDK).

Is this something that Google would consider adding in a future release?

devbridie commented 4 years ago

Thank you for the feature request! I'm curious about the perceived performance benefit. Converting world coordinates to image space isn't expensive, and frame images are already available through acquireCameraImage for consumption by other means.

ARCore doesn't really have a concept of internal 2D locations as it tries to improve its understanding of the 3D world, so I don't think we can expose such an interface.

thomas-graphopti commented 3 years ago

Feature request too!

I am working on re-localization using ARCore. The 2D feature is extremely useful for re-localization, since the 2D pixel descriptor is more robust and informative than its 3D coordination. Otherwise, I could only registrate the 3D point cloud for re-localization, that is sensitive to environment change.

I thought ARcore is a spare feature based visual inertial system, that uses the 2D descriptor for tracking and point cloud building. If you open the 2D descriptor interface, it will be much helpful for re-localization application.