Normalize rays_d in get_rays

[CSU-NXY]

Hi, thanks for your excellent work!

I'm wondering why the rays_d is normalized in get_rays()

def get_rays(directions, c2w):
    """
    Get ray origin and normalized directions in world coordinate for all pixels in one image.
    Reference: https://www.scratchapixel.com/lessons/3d-basic-rendering/
               ray-tracing-generating-camera-rays/standard-coordinate-systems

    Inputs:
        directions: (H, W, 3) precomputed ray directions in camera coordinate
        c2w: (3, 4) transformation matrix from camera coordinate to world coordinate

    Outputs:
        rays_o: (H*W, 3), the origin of the rays in world coordinate
        rays_d: (H*W, 3), the normalized direction of the rays in world coordinate
    """
    # Rotate ray directions from camera coordinate to the world coordinate, see https://github.com/kwea123/nerf_pl/issues/43
    rays_d = directions @ c2w[:, :3].T # (H, W, 3)
    rays_d = rays_d / torch.norm(rays_d, dim=-1, keepdim=True)
    # The origin of all rays is the camera origin in world coordinate
    rays_o = c2w[:, 3].expand(rays_d.shape) # (H, W, 3)

    rays_d = rays_d.view(-1, 3)
    rays_o = rays_o.view(-1, 3)

    return rays_o, rays_d

When rendering, rays_d is multiplied by z_vals which varies from near to far. If rays_d is normalized, it seems that we cannot sample points on the far plane and some points closer than near plane may be sampled.

I'm just confused about it. Could you please give me some hints?

[hjxwhy]

As I understand, it is because the ray's origins are shifted to near plan, so it works here, I have checked the official code it is not normalized here.

[kwea123]

@CSU-NXY You are right, I normalized the directions for simpler operations in the rendering code, but I was unaware about the bound problem. As you said, if the directions are normalized then some rays (around image boundaries) cannot reach far planes. Practically, it turns out that the objects are not too far from the cameras, so normalization works. However, the correct way should be to keep the directions unnormalized.