How to convert Open3d camera Pose to unity

[88ili88]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

My Question

Greetings! I've been running thru an issue where I do a camera extrinsic calibration in Open3D with global registration and ICP, the point clouds from each camera get aligned correctly but when I pass the camera pose to unity I get a really bad result. The Open3D Transform seems fine I already inverted the Y axis due to the difference in the coordinate system but I can't figure out where the variation error is coming from, is it just because of imprecision, or I'm missing something.

I invert the transform so I can extract the rotation matrix and the translation vector from the pose. My approach is to get the rotation and the translation and assign them to their correspondent kinect object in unity. The main Kinect stays in the origin of the coordinates and then I apply the transformations to each camera referencing the main one. I'll attach some data and captures of the setup.

right_transformation array([[ 0.96518757, -0.24386752, -0.09455995, -0.30049289], [ 0.14996032, 0.21974007, 0.9639638 , -0.97258158], [-0.21430085, -0.94458612, 0.24866084, 0.60323834], [ 0. , 0. , 0. , 1. ]])

First set of vectors: rotation angles right: [ 75.53550248 5.42599294 -14.17977523] translation right: [[0.56515514 -0.71024525 0.75911709]]

left_transformation array([[ 0.9690198 , 0.23938183, -0.06080263, -0.36967974], [-0.15101459, 0.37945827, -0.91280119, 1.0112917 ], [-0.19543595, 0.89370452, 0.40385273, 0.61812839], [ 0. , 0. , 0. , 1. ]])

First set of vectors: rotation angles left: [-66.13385402 3.48588443 13.87624411] translation left: [[ 0.63175131 0.84767252 0.65099793]]

The translation in unity kinda makes sense with the actual setup, but can't get a correct result.

When I manually calibrate the cameras in unity I get these vectors, and I really can't figure out if there's any correlation with the ones that I mentioned previously.

Second set of vectors: rotation angles right: [ 72.861, 7.016, 20.971] translation right: [0.3640004, 0.9750017, 0.832] rotation angles left: [-76.012, 22.273, -35.441] translation left: [ 0.331, -1.025, 0.556]

Any help or tip about what's happening would be greatly appreciated since I've been stuck with this for a long time, Thanks in advance!

[88ili88]

issue #2218 has a similar problem but it's from unity to open3d.

[theNded]

Please refer to #2218 and invert the coordinate transform. As clearly noted in that issue, the operation merely reverts y-axis, either unity to o3d or the way around.

[theNded]

Ah I see you have inverted axses. How did you do that, could you please put more details?

The logic of the post is also a bit confusing to me -- figure 3,4 looks good to me with minor distortions, while figure 2 looks incorrect. Could you please summarize a bit what you want to show?

[88ili88]

Hey, first of all, thanks for the fast reply! For the axis inversion, I just multiplied by -1 the Y-axis of the translation vectors that I get from o3d. In regard to the figures, 2 is what I get when I directly input the vectors that I get from o3d to unity, it looks completely off. After manually calibrating the cameras in unity I get 3 and 4, which gives me the second set of vectors that should be the right answer but I can't find any correlation between the first set of vectors (o3d) and the second set (unity).

[theNded]

For the axis inversion, I just multiplied by -1 the Y-axis of the translation vectors that I get from o3d.

This is incorrect from the beginning. The change of the coordinate system also affects the rotation. #2218 has given detailed transformation conversion. Please apply changes to the rotation as well.

[88ili88]

After computing the inverse transform of the point cloud to get the camera pose I proceed as shown in #2218 multiplying by -1 the whole second row of the pose (I don't know why he is multiplying the third one as this corresponds to Z and not Y).

And finally I get these vectors that should be similar to the second set of vectors, that I got from unity, but that is not the case. This gives me a even worse result. I attached some code captures in case u notice something that I'm missing.

[theNded]

No, the trick he did was mat = glm.rotate(mat, angle, [x, -1.0 * y, z]). glm is a fast math lib for graphics and there could be other equivalents for the equation. The coord transfer you referred to could be some system specific transform of his own code.

[88ili88]

But as I showed you, im already inverting the second row of my pose isn't that what's im supposed to do?

[theNded]

AFAIK this is incorrect. You need to invert the column as well because the basis also changes. Technically it can be done by r @ R @ r.inv() where r = [[1,0,0],[0,-1,0],[0,0,1]], but since you are not sure of the coordinate systems and have quaternions, Euler angles, transformations, and inversions back and forth, I would suggest you strictly go through #2218 and/or use glm as suggested.

I make a write up for the math FYI.

[theNded]

Also I'm not sure if it is correct to invert the transform in your setup. Transform is interpreted as transformation from camera to world, which is already its translation and rotation. The world is the shared coordinate system, usually aligned with the 1st Kinect's pose.

[88ili88]

Transform is interpreted as transformation from camera to world, which is already its translation and rotation.

The right_transformation and left_transformation that I mention at the begining of the issue are the pointcloud transformations that align them together, obtained from point2plane ICP, to the best of my knowldge if I inverse this transformation I get the extrensic matrix (rotation and translation) of the cameras

[theNded]

Extrinsic matrix is world to camera when you apply it to a point. There are different conventions in describing rotation and translation of the cameras, but if you need a coordinate system transform, you'd better stick to camera to world.

[88ili88]

Hi! First of all, thank you very much for the transformation of coordinate systems write up. I have tried to apply this transformation to the point cloud transform and from there, computing the camera pose, but the result wasn’t the expected.

Now, in order to see exactly what I obtain, I’ve tried to apply the coordinate system transformation and visualize the coordinates using o3d.geometry.TriangleMesh.create_coordinate_frame().

To do so, I have defined an identity matrix and applied the transformation of the system coordinates as you mentioned (I attach the code below). After visualizing the coordinates (before and after the transformation) this was the result: (The two coordinates are on top of each other)

To dissipate any doubts I have applied a translation in the y axis. We can verify that the y axis is inverted but there’s no rotation applied. Code: Then, instead of applying the transformation of the system coordinate to the given transformation matrix, I have inverted the y axis of the translation and I have applied a second rotation to the coordinate mesh, corresponding to the rotation of the coordinate system. And the result was the expected!

I have attached the code and the result below:

After trying a few things, I have seen that I could obtain the same result by making the following operation: (I don’t understand why, as the math you showed made very sense).

So I tried to do so and see the results in Unity, but the reconstruction wasn’t correct. Also, I applied first the transformation of the system coordinates and then, I extracted the rotation and translation of the camera, as you suggested too. I attach the code.

This time, to extract the rotation and translation of the camera from the transformation of the point cloud I didn’t use the inverse transform, but the results are the same. If you want me to add more detail on the maths, tell me so!

I don’t understand which could be the problem. Any suggestion will be appreciated! Thanks!

[Zulex]

Don't have the solution, but as I am trying to achieve something similar hereby some input:

Have you checked row-major, column major differences? As far as I understand, we have to negate the second column and not the second row, as that is the one storing the y axis information. I could also not find any information of the coordination frame open3D is using next to the other issue mentioned here. So if you have a link stating the open3D coordinate frame, please let me know.

Next to that, it seems that issue 2218 seems to negate the second column and row. If I am correct the glm function he calls is negating the second column and then afterwards he is also negating the second row manually. (which leaves [2,2] exactly the same as before. However he returns that value twice, so not sure if he negates both or already uses the matrix with only the column negated.

[prolove15]

Hi there. Nice to meet you. I am Unity developer and interested in use of Open3D library in Unity. You seems already using Open3D library in Unity. So I would appreciate if you let me know how to import Open3D library. Currently, I imported some Open3D project but I can't sure that it is right project for Unity. I look forward to hearing from you.

Thank you.