How to use the PointCloud class to render world points from COLMAP ?

[alexs7]

Hello,

I am trying to localise a ARCore phone in a point cloud that was generated by COLMAP.

Most of the pose estimation has been dealt with. I can get the pose matrix that represents the camera's pose in that (COLMAP) point cloud.

The point cloud from COLMAP is a replica of the current room I am in. The end goal is to use ARCore's Pose to render my points (COLMAP) to the ARCore world coordinate system.

For this I am using the method Pose.toMatrix(). The maths look pretty much like this:

ARCore_3D_world_points = [ARCorePose]^-1 [R|t] COLMAP_3D_Points

Where [R|t] is the ARCore phone's pose in COLMAP point cloud, and ARCorePose is the pose from the current ARCore frame.

Now I have managed to overwrite the PointCloud class by doing something like this:

float[] myPointsFloatArray = PointLoader.my_point(0.0f, 0.0f, -0.3f);
FloatBuffer myPoints = FloatBuffer.wrap(myPointsFloatArray);

pointCloudRenderer.update(myPoints);
pointCloudRenderer.draw(viewmtx, projmtx);

Now this renders a point right in front of the phone as soon as it starts. It takes a while to stabilize but I guess that is because ARCore is trying to understand it's environment (?).

I am assuming since ARCore works in meter units, and uses the same coordinate system as OpenGL this point is 30cm in front of the camera ?

The problem is that "ARCore_3D_world_points" returns values that do not make sense given the above. For example a point is at (-6.5811 ,4.6097, 9.1296), that is 6 meters to the left and 4 meters up the Y axis, and 9 meters in front of me ? (The room isn't that big..)

How can you get a point rendered in ARCore's world coordinate system using its Pose class ?

PS: I have rendered the cameras and 3D Points, using Matlab and the results look OK, i.e the camera is pointing where it should be.

[purvigoel]

Hello! Were you ever able to resolve this issue? I'm running into a similar problem.

[alexs7]

@purvigoel Hello, yes I did. What exactly is your problem ? For mine I had to get my head around the different coordinate systems.

[purvigoel]

@alexs7 I'm trying to perform the inverse transformation: convert ARCore camera positions/rotations to the COLMAP format. My understanding is that the ARCore format is a world-space transformation and the COLMAP format is an extrinsic matrix. I'm unclear though on if the two systems have different conventions regarding the x/y/z coordinates.

Could I ask what transformation was required to convert from COLMAP->ArCore? I can try the backwards transformation to solve my task.

Thank you!

[alexs7]

@purvigoel COLMAP's axes point this way: y+ down, x+ right, z+ towards the screen. Now ARCore's world coordinate system axes point this way: y+ up, x+ right, z+ facing away from the screen. The transformation from COLMAP to ARCore's system is:

[[1, 0, 0, 0],
 [0, -1, 0, 0],
 [0, 0, -1, 0]]

To visualise everything here is a demo from a "ARCore debugger" I wrote,

https://www.youtube.com/watch?v=jj5WLkThNuI

You can see the thin lines being the ARCore world axes as I said and the red points are points from COLMAP, the green are the ARCore point clouds. I hope this helps ? If you have any other questions please let me know.

[purvigoel]

Thank you, this was very helpful. It's working now!

[Kirang96]

Thank you, this was very helpful. It's working now!

Hi @purvigoel, I'm trying to do the same thing, to convert ARCore poses to colmap format. I couldn't get the poses right though. Could you please share how you managed to do it?

[alexs7]

@Kirang96 maybe this can help ? https://github.com/kornia/kornia/pull/1823

[Kirang96]

@Kirang96 maybe this can help ? kornia/kornia#1823

@alexs7, thanks for the reply. I tried that out and I'm still not getting proper results. Maybe the issue is with how I'm getting the poses from ARCore. I'm getting the pose using arcore Camera getDisplayOrientedPose. I had also tried using Pose w,x,y,z. Is this the right method to get the pose?

[alexs7]

ARCore poses are defined differently compared to COLMAP. COLMAP poses project 3D points to "camera space". ARCore poses do the inverse. So the pose you linked probably transforms 3D points from "camera space" to "world space".

[Kirang96]

@alexs7, I've been struggling with the pose conversion for some time now. Could you point me to the method that I should use to get the proper device pose in arcore?

[alexs7]

Hello, check this line of the code here: https://github.com/alexs7/Mobile-Pose-Estimation-Pipeline-Prototype/blob/0bf80c98bdbedc70454b43ca556cc605124e98e7/single_image_localization.py#L79

The unity pose is the pose from ARFoundation which for ARCore I think it will be the camera pose.

I think what you need is: https://developers.google.com/ar/reference/java/com/google/ar/core/Pose#inverse()

Coordinates can be a bit confusing

[Kirang96]

Thanks, I'll try it with inverse() pose. Would this pose be a transformation from world to camera coordinates?

[Kirang96]

@alexs7 ARCore pose documentation says: "As provided from all ARCore APIs, Poses always describe the transformation from object's local coordinate space to the world coordinate space (see below). That is, Poses from ARCore APIs can be thought of as equivalent to OpenGL model matrices." So the pose that we get would be from model2world right? and the inverse would be the world2model. Is this correct? If it is, is there any way to get the world2camera from it?

[alexs7]

Yes that should be the case. Inverting poses take you from one space to another and vice versa.

[Kirang96]

@alexs7 is't world2model and world2camera different? I read that the OpenGL also has a model coordinate system and arcore is outputting a transformation from model to world (model matrix in opengl). But colmap requires world2camera coordinates. Do you know how I convert it from model2world to world2camera?

[alexs7]

I would advice raising an issue in COLMAP repo. I am not very proficient at OpenGL

[Kirang96]

Okay. thanks @alexs7.