Pointcloud alignment inside unity application

[Birkenpapier]

Hello,

thank you very much once again for creating and documenting such a nice project! After tinkering with it around I want to visualize the results of the pointcloud directly inside the unity application. Therefore I am streaming the pointcloud array over a custom made rest api back to the unity app.

Unfortunately I found out that the constructed pointcloud from open3d is not aligned with unitys coordinate system. It seems to be inversed because the points are placed in my observations mirrored.

Do you have an idea or maybe any other developer how to align the reconstructed pointcloud from open3d inside unity to be placed at the same positions where they got recorded?

There is another developer which did exactly that but with unity version < 2021.2 because the methods for getting the unityWorldOrigin coordinatesystem got discontinued: IntPtr WorldOriginPtr = UnityEngine.XR.WSA.WorldManager.GetNativeISpatialCoordinateSystemPtr(); unityWorldOrigin = Marshal.GetObjectForIUnknown(WorldOriginPtr) as Windows.Perception.Spatial.SpatialCoordinateSystem;

I would appreciate any help and thankt you very, very much.

BR,

Birkenpapier

[Enzo-Kerkhof]

It seems to be inversed because the points are placed in my observations mirrored.

Yes, Unity's coordinate system is left-handed and Open3D + OpenCV are right-handed. You have to probably flip the Z axis just as in the viewer/unity_demo_stickers.py lines 130-132. Hope this helps!

[Birkenpapier]

Thank you very, very much for your explanation! The earliest I can test it is tommorrow morning so I'll come back to you again with my results.

[Birkenpapier]

I've testet your suggestion and you were correct! The rotation is now displayed correctly inside the unity application: Open3d + poinctloud registration result:

Unity visualization on Hololens screenshot:

The last problem I need to solve now is the position/translation for the object(the head in this case) in unity to correctly place it after the registration. Do you maybe have any idea what I can change to place it correctly inside unity's coordinate system? The red dot which you can see in the second image is the center of the application's coordinate system. My hand which is visualized as a green point cloud in the first image was in front of the red dot. The registration result is correct as you can see in the same image. After implementing your suggestions regarding the rotation the rotation is now correct inside the unity application leaving only the translation now which seems still to be mirrored.

I've tried to inverse it with the same method as the rotation and it works only when the object will gets recognized in front (0, 0, z) of the red dot otherwise it has offset to the hand in the real scene.

Thank you very, very much in advance again for your or any others help!

BR,

Birkenpapier

[Birkenpapier]

I fixed it with the creation of an mask array which consists of the vector [1, 1, -1] to convert it from right- to left-hand coordinate system before applying the registration step. After that the conversion of the translation matrix is not needed anymore and the position and rotation is calculated correctly.

Here's the snippet for the conversion: `
points = points[select, :] colors = colors[select, :]

    # convert from right- to left-hand coordinate system
    fill_vector = np.array([1, 1, -1])
    select_fill_vector_length = points.shape[0]
    select_fill_vector = np.zeros((select_fill_vector_length, 1), dtype=fill_vector.dtype) + fill_vector
    points = points * select_fill_vector

`

On unity site it looks like this if anybody wants to reproduce the results: (The spatial matrix was send over a Rest API to the unity application) ` Vector3 translation = new Vector3(t_1, t_2, t_3); var localToWorldMatrix = Matrix4x4.Translate(translation) * rotationMatrix;

    position.x = localToWorldMatrix.m03;
    position.y = localToWorldMatrix.m13;
    position.z = localToWorldMatrix.m23;

    Vector3 forward;
    forward.x = localToWorldMatrix.m02;
    forward.y = localToWorldMatrix.m12;
    forward.z = localToWorldMatrix.m22;

    Vector3 upwards;
    upwards.x = localToWorldMatrix.m01;
    upwards.y = localToWorldMatrix.m11;
    upwards.z = localToWorldMatrix.m21;

    rotation = Quaternion.LookRotation(forward, upwards);

`

Thank you very, very much once again for your help without it this wouldn't be possible.

BR,

Birkenpapier