Point Cloud Axes in Unity Flipped?

[agroenenberg]

Hi,

For my research I need the pointclouds recorded from the HoloLens 2 in the correct position w/ respect to the camera in Unity. But whenever I import a pointcloud into Unity, I either have to rotate the pointcloud a 180 degrees in the y-direction or mirror the pointcloud in Z and X axis in Meshlab for it to be in the same position w/ respect to the camera as when recording. Is there any way in the code to fix this so that when I process the PC in python, it is directly in the right configuration? Been searching for days now but I can't find it. Thank you :)

[Bornblack]

There is nothing wrong with the StreamRecorder or the python scripts. Pay attention to coordinate system in Meshlab and Unity and you will find the culprit. The coordinate system that Unity uses is left-handed and the direction of main camera is positive z-axis. The HoloLens app and processing scripts use is right-handed coordinate system, and the camera viewing orientation is negative z-axis. Coordinate conversion is needed when you import the pointcloud into Unity.

[agroenenberg]

Hi, thanks for helping out! I know the scripts are correct, I'm just looking for the part in the code where the CS is defined so I can rewrite it to match with the Unity righthanded CS directly. I'm doing some registrations and preprocessing steps in python for which would be great if they'd be in a lefthanded CS directly.

[Bornblack]

You can modify the point cloud data to make it suits the Unity CS. I am not sure that Unity App and Holographic DirectX 11 App have the same world origin. I have not tried out the code below myself. Hope it helps ! https://github.com/microsoft/HoloLens2ForCV/blob/41ec9e542a7a84b3a68e5089b9906864bc7c526a/Samples/StreamRecorder/StreamRecorderConverter/save_pclouds.py#L217-L221

 def cam2world(points, rig2cam, rig2world): 
     homog_points = np.hstack((points, np.ones((points.shape[0], 1)))) 
     cam2world_transform = rig2world @ np.linalg.inv(rig2cam) 
     world_points = cam2world_transform @ homog_points.T 
     world_points[:,2] = - world_points[:,2] # Z = - Z
     return world_points.T[:, :3], cam2world_transform 

[chiarapalu]

Dear All, did you find out if the point cloud's coordinate system (except for the fact that is flipped in x) is the same as the global coordinate system?

[Bornblack]

Yes , the point cloud's coordinate system is the same as the global coordinate system that established when the app starts.

[chiarapalu]

Thanks!

[agroenenberg]

I solved it by hardcoding the axes in Unity to flip!

Op ma 3 mei 2021 om 07:24 schreef hwanyyy @.***>:

@agroenenberg https://github.com/agroenenberg Hi, I have the same problem. How did you solve it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/HoloLens2ForCV/issues/65#issuecomment-831027098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN34IUNGGYRG7NAYFV67GKLTLYXRRANCNFSM4WJCZBCQ .

[hwanyyy]

thanks! in my case i rotated x-axis about points

[spongebobbe]

Hi @Bornblack, regarding flipping the axis:

world_points[:,2] = - world_points[:,2] # Z = - Z

I think it should be

world_points[2,:] = - world_points[2,:]

As world_points is a 4d vector, the third element is the Z.

if the hololens coordinate system is left-handed but the processing scripts are right-handed, shouldn't we flip also the x-axis?