Rendering Equation and Visibility Tracing

[riccardodm97]

Hi. Thanks for sharing your great work! I have two questions regarding your method: 1) It looks like you compute the result of the rendering equation for every point in the GaussianModel instead of just for those points in the view frustrum. What's the reason for that? Wouldn't it be possible to compute the brdf_color after the world2camera projection and viewport culling ?

2) Your Visibility Computation via Ray Tracing doesn't allow backpropagation of gradients. Did you consider using a differentiable implementation ? Also why do you need the learned visibility ? To be able to actually explicitly computing it for every point instead of just for a subset ?

Thanks a lot

[yGaoJiany]

Hello, thank you for your interest.

1. It is perfectly fine to calculate the rendering equations after projection and culling. Thank you for raising this issue!
2. Mainly because raytracing is too time consuming. Using raytracing in each iteration of training would make the training time unbearably long. In addition, the current approach allows us to calculate the visibility via ray tracing and bake it into each Gaussian using SHs in advance of rendering, which facilitates further real-time rendering, especially when compositing multiple objects.

[riccardodm97]

Thanks for the very quick reply. Regarding point 2: for each training iteration in the _calculateloss function in _renderneilf.py you compute the visibility via raytracing but only for 10000 random points and only for a single random ray direction per point. Then you supervise the predicted visibility to be close to the random traced one. Is this accurate ? Please correct me if I am mistaken. Do you know how slow is the ray tracing operation on average ?

Again, thanks a lot

[yGaoJiany]

You're basically right. But I need to emphasise that we have already obtained a better geometry in the first stage. Since visibility is only related to geometry, we precomputed the visibility of each Gaussian point using randomly chosen rays before training in the second stage, as an initial value for optimisation afterwards. (See https://github.com/NJU-3DV/Relightable3DGaussian/blob/77827626db514d59b390da2474de5d71be61a2e6/scene/gaussian_model.py#L266C5-L266C5) Afterwards, in the second stage of optimisation, we supervised each point with a random ray in each iteration. Using the precomputed visibility as the initial value for the optimization with 10k iterations in the second stage is sufficient in my opinion. As for the time, this is closely related to the number of Gaussian points and the image size. On the NeRF/lego dataset, precomputation of visibility using ray tracing would take about a few seconds.

[riccardodm97]

I guess my question was why do you need to optimize a learned visibility initialized from the baked one if computing it via ray tracing only take a few seconds ? I understand that's too much for real-time rendering but it shouldn't be a problem for training, right? I am asking this because, in a setting where light is non-static (e.g. OLAT), it's not possible to optimize the geometry in a first stage and only after compute visibility.

[yGaoJiany]

If we use full ray tracing in the training process, it will take seconds per iteration. I think it's a problem to perform efficient training. However, if the lighting setup is dynamic, I agree with you that relying solely on pre-calculated visibility and fine tuning is really not a good option. In this case, full ray tracing seems necessary in the training process, as the geometry will change rapidly as the material decouples from the lighting.

[karranpandey]

Hey!

Great project, just following up on this thread. I'm struggling to get real-time rendering when using the neilf_composite mode for the render function (which I assume I need to use to update the gaussians to incorporate some change to the lighting or composition in the scene). You mention using some kind of baking to help achieve this?

Could you expand on this a bit more? In my case, it doesn't really help to set bake to True in the render_kwargs in terms of getting the fps up. Let me know if you have any further tips for this!

For instance, I am wondering if its possible to perform some relighting / composition, and update the gaussians to incorporate this change, and now render the relit scene with the faster 'neilf' mode again for real-time rendering?

For context, I'm testing on the lego scene from the nerf synthetic dataset within the repo.

[JiatengLiu]

Hello！ Great Project! And I have a question about rendering equation. In the paper, the formula for calculating the specular term is DFG/(n,wi)(n,wo), but only DFG is calculated in the code (in file:gaussian_renderer/neilf.py). What are the considerations behind this? Thanks for your reply, below is the code about how to calculate the specular term.

def _f_specular(h_d_n, h_d_o, n_d_i, n_d_o, base_color, roughness, metallic):
    # used in SG, wrongly normalized
    def _d_sg(r, cos):
        r2 = (r * r).clamp(min=1e-7)
        amp = 1 / (r2 * np.pi)
        sharp = 2 / r2
        return amp * torch.exp(sharp * (cos - 1))

    D = _d_sg(roughness, h_d_n)

    # Fresnel term F
    F_0 = 0.04 * (1 - metallic) + base_color * metallic  # [H, W, 1, 3]
    F = F_0 + (1.0 - F_0) * ((1.0 - h_d_o) ** 5)  # [H, W, S, 3]

    # geometry term V, we use V = G / (4 * cos * cos) here
    def _v_schlick_ggx(r, cos):
        r2 = ((1 + r) ** 2) / 8
        return 0.5 / (cos * (1 - r2) + r2).clamp(min=1e-7)

    V = _v_schlick_ggx(roughness, n_d_i) * _v_schlick_ggx(roughness, n_d_o)  # [H, W, S, 1]

    return D * F * V

f_s = _f_specular(h_d_n, h_d_o, n_d_i, n_d_o, base_color, roughness, metallic)
transport = incident_lights * incident_areas * n_d_i  # （num_pts, num_sample, 3)
rgb_s = (f_s * transport).mean(dim=-2)