search

liuyuan-pal / NeuRay

[CVPR2022] Neural Rays for Occlusion-aware Image-based Rendering
GNU General Public License v3.0
409 stars 32 forks source link

about depth using in init_net.py #4

Closed Fangkang515 closed 2 years ago

Fangkang515 commented 2 years ago

Thank you for your contribution！

I notice that "DepthInitNet" and "CostVolumeInitNet" both use the depth image. Does it mean the model is not end-to-end for the dataset without depth information, because you should get the depth using colmap firstly.

If my dataset is relatively large and there is no depth information or it is time-consuming to calculate depth, how should I use your code to run it?

Thank you very much.

liuyuan-pal commented 2 years ago

Hi, thanks for your interests. The CostVolumeInitNet does not require depth actually though I have passed it into the model. You can just implement a new database and return a dummy depth map like np.zeros([h,w],dtype=float32) in get_depth().

Fangkang515 commented 2 years ago

Thank you for your reply, which solves my problem. Then the new question is:

  1. How can I output a depth map when rendering, like the figure15 in your paper;
  2. I found that on my own dataset, the neuray method will lose some objects. I want to ask which parameters I should modify to improve the image quality?

the neuray results: some objects losed

image

Thank you very much.

liuyuan-pal commented 2 years ago

For your new questions:

  1. I have added an interface to render depth: https://github.com/liuyuan-pal/NeuRay/blob/fb3188d7fc4b17ab3b50a816c34b500eac404944/render.py#L195 and https://github.com/liuyuan-pal/NeuRay/blob/fb3188d7fc4b17ab3b50a816c34b500eac404944/network/renderer.py#L202 Results will be saved in data/render.
  2. Possible reasons for missing objects: 1) depth range does not include the object. 2) too few images seeing the object so that it cannot be correctly matched to construct the radiance field.
  3. Possible improvements: The MVSNet is only trained on the DTU dataset and it is fixed during the generalization training for NVS so it is not very accurate for outdoor scenes or the forward-facing scenes according to my experience. 1) I recommend reconstructing the scene by COLMAP and using the DepthInitNet, which performs better. 2) You may finetune the MVSNet in the NVS training, which however may consume lots of GPU memory.

Thank you!