search

kwea123 / nerf_pl

NeRF (Neural Radiance Fields) and NeRF in the Wild using pytorch-lightning
https://www.youtube.com/playlist?list=PLDV2CyUo4q-K02pNEyDr7DYpTQuka3mbV
MIT License
2.74k stars 483 forks source link

How to get real depth values from approximated depth in certain ray. #124

Open dogyoonlee opened 2 years ago

dogyoonlee commented 2 years ago

Hi, I'm appreciate to your great work!

I'm implementing some light-field based neural network which have different input structure for MLP.

The MLP outputs the rgb color and depth values which are in range [0, 1] in NDC coordinates by sigmoid function to corresponding ray.

However, my questions is follows

  1. For NDC coordinates, if I correctly understand, the scene bounded in near(0) and far(1) plane. Then, do all components exist between near and far plane?

  2. As we can see the get_rays functions, z value is instantiate with -1. I can sure that camera origin is world coordinates from the evidence that c2w[:3,-1] part in following code. As I understand, the rays are still in world coordinate since the magnitude of direction vector doesn't matter. Am I right?

    image

  3. I get some depth value and compute the corresponding points using follow code. I intend the code works regardless of the coordinate system, NDC or world. z_vals contains N_rays x [near, far] values. Is this code right..?

    image

  4. If points are correctly computed in given ray with some depth values, how to compute the real depth value when NDC coordinate system is used? For example, if I have P(Xp, Yp, Zp) in NDC coordinates, following code can back project to real depth value? I use same coordinate system to NeRF, (right, up, backward), in world and I guess NDC coordinate system is (right, up, forward).

I read about the issue you raised in NeRF original GitHub, but I think it still not work properly for my code.

For given ndc_pts, which have N_rays x (Xp, Yp, Zp), we back-project ndc_pts to world following your issue.

image

But what I found in your GitHub issue, which refer the necessary of back project with K^-1 * d * (x y 1).

image

For depth direction, I think depth_real = 1/(1-depth_ndc) is different because of the different coordinate system. Then, does K^-1 * d * (x y 1) mean same process with the mentioned code from NeRF author?(which I used).

Thanks to read my questions. If you reply to this issue, it will be really helpful...!

mertkiray commented 2 years ago

Hey @dogyoonlee , did you find a way to convert the depth_map in NDC space to depth_map in world values?