search

bennyguo / instant-nsr-pl

Neural Surface reconstruction based on Instant-NGP. Efficient and customizable boilerplate for your research projects. Train NeuS in 10min!
MIT License
857 stars 84 forks source link

monosdf and surface normal implementation #21

Open pablovela5620 opened 2 years ago

pablovela5620 commented 2 years ago

Continue geometric cues w/scannet implementation. Start with MonoSDF as simpler to implement v.s. Neuris

pablovela5620 commented 2 years ago

Didn't know the best way to continue so I just made another PR. I'm trying to understand how to render the normal, right now I just copy pasted the NerfSystem as that's whats currently working

I'm using this as a reference https://github.com/autonomousvision/monosdf/blob/8e46ed516c80c30b73043ff163c2bd94b4b67062/code/model/network.py#L497

It looks like they're using the gradient outputs which I think are equivalent to this https://github.com/bennyguo/instant-nsr-pl/blob/c236bc369b91b8b044fbea654c67a0c16e69b70f/models/geometry.py#L149

I wanted to verify that I would then take those value and feed them into render_weight_from_density as you recommended as follows

ray_indices = unpack_info(packed_info)
sigmas = sigma_fn(t_starts, t_ends, ray_indices.long())
weights = render_weight_from_density(packed_info, t_starts, t_ends, sigmas)
normal_grad = grad / (grad.norm(2, -1, keepdim=True) + 1e-6)

normals = accumulate_along_rays(
    weights,
    ray_indices,
    values=normal_grad,
    n_rays=n_rays,
    )

I'm working on converting over from nerf to sdf, but wanted to start with the nerf system to ensure I was correctly loading the depth/normal maps.

Any advice on the normal map rendering would be appreciated!

pablovela5620 commented 2 years ago

Managed to get surface normal rendering working, though it seems to significantly slow down training, I think there's probably a better way of doing it as there's a bunch of redundant computation happening but it is not being included inside of rendering

https://user-images.githubusercontent.com/25287427/204103711-597fe207-be1e-4a34-83d1-1fa84cee3c28.mp4

bennyguo commented 1 year ago

Hi, sorry for the late reply! This is exactly what i meant for rendering the normal map. I think the slowdown mainly comes from second-order gradient calculation (which is not essentially required for NeRF). So if you experiment with NeuS, the slowdown could be negligible.

pablovela5620 commented 1 year ago

Thanks for the reply, this is taking a bit longer than I had hoped as I'm both implementing this and learning about Neus/Neuris/MonoSDF at the same time. May be a while before I'm able to get this working!