Why using NeRF to model background?

[AIBluefisher]

Hi, @bennyguo

I wonder why you are using another NeRF to model backgrounds for neural reconstruction. In the original NeRF paper, I think the designation of using NeRF++ to model background is due to it being an earlier work and at that time there are not many works solving for the unbounded scenes. But since we are using NeRFAcc, we can directly model unbounded scenes in a single model.

[bennyguo]

Hi, @AIBluefisher! In the NeuS system, the foreground is modeled as a signed distance field, but the background is often modeled as a volumetric part, so we need a separate NeRF. In other words, the two NeRFs actually model different things (surface for foreground and volume for background). But it's worth mentioning that in BakedSDF the authors model the whole scene (foreground + background) with SDF but I think it could be harder to converge than using volume to model the background.

[AIBluefisher]

Hi, @bennyguo Thanks for your reply! "... but I think it could be harder to converge than using volume to model the background" - That's exactly what confuses me a lot. Since we can model the radiance field in a single model on unbounded scenes with NeRFAcc, theoretically, we can do the same for NeuS. And in NeuS, the main difference is that we just have another formula to compute alpha. Do you have any insights into this?

[bennyguo]

Although NeuS still uses volume rendering, it models the signed distance field which is restricted by the eikonal constraint. This makes it much harder to optimize than NeRF (no constraint at all).

[AIBluefisher]

It is a valid point. Actually, I tried to model NeuS without nerf for backgrounds on unbounded scenes, but the reconstruction quality is really poor.

[yipingp]

Hi, @AIBluefisher! In the NeuS system, the foreground is modeled as a signed distance field, but the background is often modeled as a volumetric part, so we need a separate NeRF. In other words, the two NeRFs actually model different things (surface for foreground and volume for background). But it's worth mentioning that in BakedSDF the authors model the whole scene (foreground + background) with SDF but I think it could be harder to converge than using volume to model the background.

Hi, since we already have the foreground point cloud, sphere radius, or mask, each of which can define the region we want to reconstruct, why is it necessary to model the background as well? Thanks!

[AIBluefisher]

Hi, @AIBluefisher! In the NeuS system, the foreground is modeled as a signed distance field, but the background is often modeled as a volumetric part, so we need a separate NeRF. In other words, the two NeRFs actually model different things (surface for foreground and volume for background). But it's worth mentioning that in BakedSDF the authors model the whole scene (foreground + background) with SDF but I think it could be harder to converge than using volume to model the background.

Hi, since we already have the foreground point cloud, sphere radius, or mask, each of which can define the region we want to reconstruct, why is it necessary to model the background as well? Thanks!

Because masks are not always available. If we already have masks, it is not necessary to use a background model.

[yipingp]

Hi @AIBluefisher , why it is still necessary to model the background when we have foreground point cloud or sphere radius to tell the ROI.

[AIBluefisher]

Theoretically speaking, it is unnecessary to model the background if we have the foreground/background masks. If we have a foreground point cloud, say it is dense enough, we can actually obtain the foreground masks by projecting the point clouds to the image plane. However, if we have only sphere radius, we have to model the foreground in the sphere and also the background that is out of the sphere - because we do not know which pixels should be modeled inside/outside the sphere.