Question about coordinate transforms

[aal-rvr]

Hi,

I am trying to get a point from the depth camera and view it in my Unity application. I use a background app to get the point and view it in my Unity app. I am doing the following steps but my 3d point is slightly off: 1) I get the (u,v) point from the depth image. 2) I usepCameraSensor->MapImagePointToCameraUnitPlane(uv, xy); to get a xy on the unit plane, followed by multiplying it with the depth of the (u,v) point to get the 3D coordinates. 3) I get my final 3D point in the left eye coordinates by pre-multiplying the 3d coordinates with the extrinsic matrix that I get from pCameraSensor->GetCameraExtrinsicsMatrix(&cameraPose).

I have to translate the point by ~6cms to the right to align it with the center. But even after doing that, the final 3d point is off from the expected position by a height(y) of 0.1 meters when I try to feed it to my Unity app? Am I doing something wrong with my transforms? OR does Unity apps have a different main camera pose from the rig pose?

Also, If I remove the translation from the extrinsic matrix the resultant point is only off by a 2-3 cm from the real world point.

Thanks!

[pkuma]

Hi,

I am trying to get a point from the depth camera and view it in my Unity application. I use a background app to get the point and view it in my Unity app. I am doing the following steps but my 3d point is slightly off:

1. I get the (u,v) point from the depth image.
2. I usepCameraSensor->MapImagePointToCameraUnitPlane(uv, xy); to get a xy on the unit plane, followed by multiplying it with the depth of the (u,v) point to get the 3D coordinates.
3. I get my final 3D point in the left eye coordinates by pre-multiplying the 3d coordinates with the extrinsic matrix that I get from pCameraSensor->GetCameraExtrinsicsMatrix(&cameraPose).

I have to translate the point by ~6cms to the right to align it with the center. But even after doing that, the final 3d point is off from the expected position by a height(y) of 0.1 meters when I try to feed it to my Unity app? Am I doing something wrong with my transforms? OR does Unity apps have a different main camera pose from the rig pose?

Also, If I remove the translation from the extrinsic matrix the resultant point is only off by a 2-3 cm from the real world point.

Thanks!

hi，I have a problem similar to yours. Have you solved it？

[felixvh]

Any update here?

[Olli1080]

@aal-rvr Just to make sure, are you multiplying with the extrinsics matrix or its inverse and did you take the potential difference in coordinate frames into account?