Unproject Depth

[griegler]

I rendered a scene using the provided ReplicaRenderer to get depth maps and color images. In addition, I saved the extrinsic and intrinsic camera parameters. Then I tried to unproject the depth maps again, but they do not align (see attached images).

For the intrinsic parameters I assume a pinhole camera and used the same parameters as for pangolin::ProjectionMatrixRDF_BottomLeft: focal length = [width / 2.0f, width / 2.0f], principal point = [(width - 1.0f) / 2.0f, (height - 1.0f) / 2.0f]. And rotation and translation are obtained from T = s_cam.GetModelViewMatrix() with R = T[:3,:3], t = T[:3, 3]. The depth is re-scaled by dividing the depth values as read from the .png files by d = d_png / (65535.0 * 0.1). The 3D point is then given by xyz = R^T . (K^(-1) . d . (u, v, 1)^T - t).

If you look at the blue cushion, or the wall to the right, you can notice the offsets. I assume that there is a problem how I use the intrinsics, or scale the depth?

[jstraub]

@simesgreen how do those calculations look to you?

[jstraub]

An alternate option for rendering depth and other quantities is to use the habitat simulator https://github.com/facebookresearch/habitat-sim

[erikwijmans]

You can read through the math of how habitat does the unprojection here: https://aihabitat.org/docs/habitat-cpp/namespaceesp_1_1gfx.html#a551e580d2075a4d118092fcc6e20045b

[griegler]

@erikwijmans calculateDepthUnprojection assumes a different form of the projection matrix, and it is not clear to me what p,q,a,b are referring to? Also Habitat is build on Magnum, whereas the code here is build on pangolin? Maybe @stevenlovegrove has an answer?

[roym899]

Just for reference; I had a similar issue, however it was just a mismatch of the focal length. Seems like standard projection works as it should.