If NDC helps for forward-facing dataset?

[Harper714]

Hi, thanks for the awesome work.

I have tested the code on several forward-facing scenes captured by myself, and found that we need carefully tune the parameters such as offset and bound to get proper results. I want to ask if first converting to NDC space like original NeRF do can avoid these problems? Because NDC helps to map the whole scene into a unit cube. If possible, any suggestion to use the NDC space? Thanks!

[ashawkey]

@Harper714 Hi, you are right. Sorry that I have not planned to implement this recently, but any contribution will be welcome.

[Harper714]

I see. Many thanks~

[Harper714]

@ashawkey Hi, I tried to add the ndc conversion before generating xyzs, like this:

def run_cuda(self, rays_o, rays_d,  dt_gamma=0, bg_color=None, perturb=False, force_all_rays=False, max_steps=1024, T_thresh=1e-4, **kwargs):
    # rays_o, rays_d: [B, N, 3], assumes B == 1
    # return: image: [B, N, 3], depth: [B, N]

    prefix = rays_o.shape[:-1]
    rays_o = rays_o.contiguous().view(-1, 3)
    rays_d = rays_d.contiguous().view(-1, 3)

    N = rays_o.shape[0] # N = B * N, in fact
    device = rays_o.device

    # pre-calculate near far
    nears, fars = raymarching.near_far_from_aabb(rays_o, rays_d, self.aabb_train if self.training else self.aabb_infer, self.min_near)

    # ndc rays
    if kwargs['ndc']:
        near = 0.
        far = 1.
        rays_o, rays_d = self.ndc_rays(kwargs['height'], kwargs['width'], kwargs['focal'], near, rays_o, rays_d)
        nears, fars = near*torch.ones_like(rays_o[:, 0]), far*torch.ones_like(rays_d[:, 0])
    ......

but cannot work. Could you provide any suggestion to fix it? Thanks!

[serizba]

Hi @Harper714

Did you manage to make this work? I am also having issues with unbound outdoor scenes.

[Harper714]

@serizba I managed to do it but cannot get desired results. Hope someone address this issue.

[serizba]

@Harper714 Could you show what did you change?

Thanks a lot in advance

[ferrophile]

Hi guys,

Not sure if you are still working on this, but I tried experimenting with rendering under NDC coordinates as well. I observed that it mainly boils down to two aspects:

Bounds. Even when using NDC, it does not mean that the points are bounded perfectly within the [-1, +1] unit box. This is because the NDC frustum is relative to a single reference pose in the scene. The actual camera poses used for training are different from the ref pose, so the rays can start / end outside the NDC box.

Therefore during the calculation of the near / far values in near_far_from_aabb, I use the following bounds:

aabb = torch.FloatTensor([-bound, -bound, -1, bound, bound, 0.999])

• We know that the NDC z-coordinate is bounded within $[-1,1]$. Furthermore, since z=1 represents the plane at infinity, I use a slightly smaller upper bound to prevent numerical issues (in practice there is probably no need to consider samples that are too far away as well).
• The x / y coordinates are not bounded within $[-1,1]$ so I continue to use a scene-specific bound, bound. This is typically not much bigger than 1 (as long as the camera poses don't deviate too much from the reference pose).

Distances between samples. During ray marching, the distances between ray samples (i.e. deltas returned by the march_rays functions) are computed in NDC space. This needs to be changed to actual distances in world space. I use the following derivation based on z-coordinates:

Let $z_0$ and $z_1$ be the NDC z-coordinates of two consecutive samples on a ray. We can convert them to the corresponding z-coordinate in world space via the following relation: $$zw = \frac{2}{z{ndc}-1}$$ (You can see that this function maps $[-1, 1]$ to $[-1,-\infty)$.)

So the actual z-coordinate difference $\Delta z$ between the two points is $$\Delta z = \frac{2}{z_1 - 1} - \frac{2}{z_0 - 1}$$

and to convert this into Euclidean distance $\Delta t$, we can use similar triangles: $$\frac{\Delta z}{\Delta t}=\hat{z}$$

where $\hat{z}$ is the z-component of the ray direction unit vector in world space (i.e. rays_d prior to calling ndc_rays). So combining the above: $$\Delta t=\frac{1}{\hat{z}}\left(\frac{2}{z_1 - 1} - \frac{2}{z_0 - 1}\right)$$

Hope this helps.

[Harper714]

@ferrophile Hi, so nice of you to share your information. I was busy with other things before, and am going to address the NDC issue again, with your suggestions.

About the second point, there is a simple question. When do we need to convert deltas to actual distances? Before composite_rays_train or after that? Because the rgbs and sigmas are predicted under NDC, can we directly composite them using deltas retured by march_rays?

By the way, could I ask if you can get satisfactory results using NGP with NDC, especially on unbounded forward-facing dataset?

[ckLibra]

@ferrophile Hi, so nice of you to share your information. I was busy with other things before, and am going to address the NDC issue again, with your suggestions.

About the second point, there is a simple question. When do we need to convert deltas to actual distances? Before composite_rays_train or after that? Because the rgbs and sigmas are predicted under NDC, can we directly composite them using deltas retured by march_rays?

By the way, could I ask if you can get satisfactory results using NGP with NDC, especially on unbounded forward-facing dataset?

Hello, have you solved the problem now? I am working on face-forwarding scenes too and meet the same problem