The sampling strategy in sample_ray_contracted()

[Tausc]

Hello! I am trying to understand this piece of code:

    def sample_ray_contracted(self, rays_o, rays_d, is_train=True, N_samples=-1):
        N_samples = N_samples if N_samples > 0 else self.nSamples
        N_samples = N_samples // 6

        t_vals = (
            torch.linspace(0.0, N_samples - 1, N_samples, device=rays_o.device)[None] / N_samples
        )

        interpx = t_vals.clone()

        if is_train:
            interpx += torch.rand_like(t_vals) / N_samples
            t_vals += torch.rand_like(t_vals) / N_samples

        near, far = [1, 1e3]
        interpx = torch.cat(
            [interpx, 1.0 / (1.0 / near * (1.0 - t_vals) + 1.0 / far * t_vals)], dim=1
        )
        interpx += 1e-1
        rays_pts = rays_o[..., None, :] + rays_d[..., None, :] * interpx[..., None]

        rays_pts = contract(rays_pts)

        mask_outbbox = torch.zeros_like(rays_pts[..., 0]) > 0
        return rays_pts, interpx, ~mask_outbbox

I find that this function didnt appear in TensoRF(https://github.com/apchenstu/TensoRF). I guess it is added in localrf so I decide to submit issue here. Would you mind explaining what exactly this code is trying to compute? 1.0 / (1.0 / near * (1.0 - t_vals) + 1.0 / far * t_vals) is it trying to get the inverse depth?

[Tausc]

And I wonder whether the depth_map renderred is depth or inv_depth. It is used directly in compute_depth_loss() with groundtruth invdepths, so i guess it is?

[ameuleman]

Hi, When rendering a ray, we sample half of the points uniformly in distance in [0, 1[ and the other half is sampled uniformly in inverse distance in [1, 1000[ (offset by 0.1). 1.0 / (1.0 / near * (1.0 - t_vals) + 1.0 / far * t_vals) samples within [near, far[ i.e. [1, 1000[. The goal is for our sampling strategy to loosely follow the resolution of the representation: due to the parameterization we use (Mip-NeRF360 contraction), the resolution is high within [-1, 1]^3 and then gets coarser as we get further from the origin. Another approach could be to sample in [0, 2[ before inverting the contraction. The rendered depth_map is inverted before calling compute_depth_loss here.

[Tausc]

So before contract(), the xyz of rays_pts are still in euclid space?

[ameuleman]

Yes, and the origin is the centre of the current local radiance field

[Tausc]

ok, now i understand! Thank you for replying