Could you help me with the normal graph generation?

Hi! Thanks for your great work! I've been devoting myself into the EndoNeRF related project recently, too.

I noticed that you added a normal graph into your project which is not in EndoNeRF, so I want to add it in my project. However, I met 2 problems.

Firstly, I'm wondering which 'out_normal_vr' are you use in the final images present?

out_rgb_vr = []  # Volume rendering
out_normal_vr = []
out_depth_vr = []
for rays_split in tqdm(rays, desc="DEMO|Render 2D images", leave=False):
    # Render with volume rendering
    render_out = self.renderer(rays_split, iter_step=global_step, eval=True) 
    out_rgb_vr.append(tensor2array(render_out["color_map"]))
    out_normal_vr.append(tensor2array(render_out["normal_map"]))
    out_depth_vr.append(tensor2array(render_out["depth_map"]))
    del render_out

out_rgb_vr, out_rgb_vr_show = gen_rgb(out_rgb_vr, n_frames, W, H)
out_depth_vr, out_depth_vr_show = gen_depth(out_depth_vr, n_frames, W, H, depth_max, filter=depth_filter)
out_normal_vr, out_normal_vr_show = gen_normal_from_depth(rays_in, out_depth_vr, self.device)

The code seems represent that there is a 'out_normal_vr' from the self.renderer and another one from gen_normal_from_depth. Are they the same? Or is there any connection between these two?

Secondly, I copied the gen_normal_from_depth function and utilize it into the original EndoNeRF project to generate normal graph. Whereas, the result seems unsatisfactory. One of the result presents like this:

Do you have any ideas why is that happened ? It bothered me for a long time.

Hoping for your reply soon!

Hi, thanks for your interest.

Q1: Two normal maps. The normal images from render_out is generated in a volume rendering manner. It is the weighted sum of gradients (related codes of EndoNeRF and EndoSurf, which is actually borrowed from IRON). The normal images from gen_normal_from_depth is calculated based on the depth maps (see codes here). The two results look quite similar and I think you can use either of them for visualization.

Q2: Noisy normal map. I guess what you got is the result before depth denoising. EndoNeRF heavily depends on a bilateral filter to process the rendered depth. Simply add filter gen_depth(..., filter=[24, 64, 32]) like code here and you should get similar results in the paper.

Ruyi-Zha / endosurf

Could you help me with the normal graph generation? #2