Conversion between NDC coordinate and world coordinate

[gaochen315]

Hi, thanks for the great work!

I want to project a 3D point [x, y, z] to an arbitrary camera's image plane. This can be done easily if not using NDC, because we can use the w3c and c2w matrix Colmap provided.

If NDC is used, we then need to invert the NDC equation to get back world space values. You mentioned in issue #35, that the conversion is:

z = 2 / (z' - 1)
x = -x' * z * W / 2 / f
y = -y' * z * H / 2 / f

As a sanity check, I did the following experiments:

1. Construct rays_d and rays_o by get_rays_np.

2. Construct z_vals by near * (1.-tf.linspace(0., 1., 64)) + far * (tf.linspace(0., 1., 64)).

3. Construct 3D points pts by rays_o[...,None,:] + rays_d[...,None,:] * z_vals[...,:,None]. These points are in the world coordinate.

4. Construct rays_d and rays_o in the NDC coordinate by ndc_rays(H, W, focal, tf.cast(1., tf.float32), rays_o, rays_d).

5. Construct z_vals by 0. * (1.-tf.linspace(0., 1., 64)) + 1. * (tf.linspace(0., 1., 64)).

6. Construct 3D points pts_NDC by rays_o[...,None,:] + rays_d[...,None,:] * z_vals[...,:,None]. These points are in the NDC coordinate.

7. Change pts_NDC from the NDC coordinate to the world coordinate.

   pts_NDC2world_z = 2 / (pts_NDC[..., 2:] - 1)
   pts_NDC2world_x = - pts_NDC[..., 0:1] * pts_NDC2world_z * W / 2 / f
   pts_NDC2world_y = - pts_NDC[..., 1:2] * pts_NDC2world_z * H / 2 / f
   pts_NDC2world = concatenate(pts_NDC2world_x, pts_NDC2world_y, pts_NDC2world_z)

   Ideally, pts_NDC2world and pts should be the same. But they are not. Am I missing something? Thank you in advance!

[NagabhushanSN95]

@gaochen315 were you able to find a solution?

[NagabhushanSN95]

I think I found the reason. In step 2, you're uniformly sampling in depth. But in step 5, you're uniformly sampling in disparity. Obviously, both are bound to be different. Also, in NDC space, z_val=1 represents infinite depth, not the far value.

[nitishsaDire]

@gaochen315 Please check the additional NDC material provided towards the end of the nerf paper. What it says is for every point o+td in world coords (wc) there exists a point o'+t'd' in NDC. Then by derivation, it shows the relationship between t and t' in equation 15. What you are doing is taking samples in wc at some step size along the ray, but these sample points won't correspond to points in NDC at some regular interval, although they will definitely have their NDC representations, but the distance between these samples in NDC won't be uniform like in wc. I hope you get the point. Thanks