Camera Reference Frame to Visor Reference Frame

[nirajkark07]

Hello,

Thank you for the work you have done!

I am looking to find the transformation between the RGB camera on the Hololens2 and the visor reference frame. Would this be the 4x4 transformation matrix that is provided in the client_stream_pv.py as data.projection?

[jdibenes]

Hello, I'm not sure what you mean by visor reference frame, sorry. Could you elaborate? hl2ss only provides the following transformations for PV: PV <-> World (reference frame for holograms, provided by data.pose) PV <-> RigNode (reference frame for research mode sensors, provided by data.extrinsics) data.projection only contains the camera intrinsics.

[nirajkark07]

Hi,

In my application, I am taking an image from the client_stream_pv.py application and feeding it into an object detection model. The output is a pair of x,y coordinates in PV reference frame. I want to display these coordinates as a hologram. Given your previous reply, would I need the data.pose transformation to do this?

[jdibenes]

Yes. The transformation should be:

[x, y, 1] = [u, v, 1] @ intrinsics_pv[:3, :3]^(-1)
[X, Y, Z, 1]_world = [z * x, z * y, z, 1] @ extrinsics_pv^(-1) @ data_pv.pose

with

intrinsics_pv = hl2ss.create_pv_intrinsics(data_pv.payload.focal_length, data_pv.payload.principal_point)
extrinsics_pv = np.eye(4, 4, dtype=np.float32)
intrinsics_pv, extrinsics_pv = hl2ss_3dcv.pv_fix_calibration(intrinsics_pv, extrinsics_pv)

z can be obtained as shown in sample_pv_depth_lt.py. If you're using Unity, convert the result to left-handed coordinates.

[nirajkark07]

Thank you for your reponse.

If I am using unreal engine, would I need to convert it to left-handed coordinates as well?

[jdibenes]

Yes, also an additional transformation may be needed seeing that unreal engine is +z = up, +y = right and +x = forward, while hololens is +y = up, +x = right, and -z = forward.