Quality of Point Cloud obtained from ARCore

[dhruvjaglan]

Hi,

I was trying to perform 3D Reconstruction from the point cloud being obtained from ARCore. However, The point cloud I was able to obtain using ARCore was not accurate or dense enough to perform 3D Reconstruction. Specifically, Points that were supposed to be on the same surface were off by a few miilimeteres due to which the surface looked as if it had a thickness.

Am I isolating the point cloud correctly or is it a limitation of ARCore ? What should be the standard approach towards isolating the point cloud and for 3D Reconstruction ?

I am attaching below the point cloud obtained from a laptop. ( The file is in the .PLY format, but since it won't allow me to upload a .ply file, I have changed the extension to .txt. The file can be viewed on Sketchfab ,Meshlab or any other software capable of rendering 3D models after changing the extension back to .ply .)

The Keyboard and the screen of the laptop here look as if they have a thickness, although all the points were supposed to be at the same depth.

Please have a look at the point cloud and guide me as to what is going wrong here, since I have been stuck at it for quite some time now.

Thank you Laptop.txt

[avirodov]

Hello.

1. It is normal for 3d points in the point cloud to be somewhat inaccurate. Few mm is definitely within the error we would expect.
2. Please make sure you are using anchors. Attaching each point cloud to an anchor will mean that any pose refinement will be properly accounted for. Otherwise you may end up with point cloud 1 being really far off point cloud 2, even if they represent same points accurately. As per [documentation](https://developers.google.com/ar/reference/java/com/google/ar/core/PointCloud.html#getPoints()) , "Point locations are in the world coordinate space, consistent with the camera position for the frame that provided the point cloud".

[avirodov]

Please reopen this issue if further discussion is need or if your question was not answered.

[dhruvjaglan]

I am using anchors and i am getting error in few mm because this point cloud is not able to produce a good 3D model after reconstruction. so can i do anything to improve the point cloud ? thanks a lot for your help

[Godric877]

Can you please elaborate on how to attach an anchor to a point cloud ? I was under the impression that an anchor can only be created on a trackable such as a single Point or a single Plane but not to a group of objects such as a point cloud ... ?

[avirodov]

Sure, sorry I wasn't clear. There is no api to attach anchor to point cloud or vice versa. However, at each frame where you obtain a point cloud (which is in world coordinates for that frame), you place an anchor at say Pose.Identity (origin, that is <0,0,0>). When the system is adjusted, the anchor will change the pose it reports, and you can then adjust the point cloud.

Numerical example: Say at time=2 sec, you obtain point_cloud_2 and place an anchor_2 at 0,0,0. Now at time=5 sec, you obtain another point_cloud_5. Now you would examine anchor_2, and if its pose changed, you would apply the inverse to all points of point_cloud_2 to make it in the same frame of reference as point_cloud_5.

Now to get absolute best adjustment, you would probably need an anchor per point cloud point, but that may be too expensive and unnecessary. My suggestion would be to start with one anchor, either at origin, or at center of point cloud. Or segment the point cloud to "sub-point-clouds" into say boxes or 1meter^3, or 10cm^3 or so, and make anchor per such box.

Note that the point cloud is exposed mostly for debug visualization, and is not specifically intended or optimized for reconstruction. I cannot recommend a way to improve the point cloud (we don't have a tunable setting or anything similar), but having good parallax motion generally helps depth estimation.

Oh one more thing, we do have confidence numbers for each point. Maybe use only high-confidence ones, or somehow merge giving more weight to high-confidence points?

[avirodov]

keeping issue open until all questions are addressed.

[Godric877]

Thanks a lot for such a detailed explanation. 😀

Suppose we make an anchor at each point in the point cloud, do you think this will result in an inaccurate point cloud ?

Please ignore the computational cost associated with the above-mentioned procedure while answering the question.

Thank you for your help.

[avirodov]

Suppose we make an anchor at each point in the point cloud, do you think this will result in an inaccurate point cloud ?

This will be the most accurate solution, and also the most expensive computationally. To elaborate, all such anchors will now be kept globally consistent between them across all frames (while the system is tracking of course).

Having additional anchors does not affect point clouds in any way, at least as long as the overall system performance is unaffected.

[Godric877]

Thanks a lot! I was just wondering if there was an update with regards to Tango 3DR Library being compatible with ARCore or another library being developed for 3D Reconstruction using ARCore?

What is the general approach you would be following to perform 3D Reconstruction using ARCore? WIll it be through isolation of point clouds and then performing surface reconstruction on it or through some other method?

Thank you!

[avirodov]

I was just wondering if there was an update with regards to Tango 3DR Library

Sorry, no update at this time.

What is the general approach you would be following to perform 3D Reconstruction using ARCore?

Not sure I have a good answer here. Note that Tango 3DR was relying on a hardware depth sensor that provided dense depth data, and this is something that is not currently available at large on ARCore-enabled devices.

Please reopen the ticket if the issue was not addressed.

[navalsheth]

Great Discussion , It has been mentioned in many articles now that new phones in 2019 will have TOF depth sensors. I think that may the first start for 3d Reconstruction. But is it really going to come in all ARCORE phones that is the question ?? whats yours take guys ??

[mode51software]

How about making ARCore detect and harness a USB attached hardware based depth sensor, so that Tango 3DR functionality could be enabled?

For example, the flexx: https://pmdtec.com/picofamily/flexx/

On this theme, ie. making ARCore accept external sensors, I'm thinking of multiple sources like the resident sensors in an autonomous vehicle, or a smart wheelchair, so that ARCore could harness additional sensors when they become available in the local environment.

[HripsimeS]

@dhruvjaglan Hello.

Can you please give some explanation how did you extract point cloud on your laptop as .PLY format. I would appreciate if you can share the Java code that you used to extract point clouds and save on your PC as .ply file.

Look forward to hearing from you. Thank you in advance!