search

ashawkey / torch-ngp

A pytorch CUDA extension implementation of instant-ngp (sdf and nerf), with a GUI.
MIT License
2.11k stars 275 forks source link

3D Mesh Quality Exploration #97

Open MaroonAmor opened 2 years ago

MaroonAmor commented 2 years ago

Hi @ashawkey, thanks for the great work!

I'm reaching out to explore the 3D mesh quality from torch-ngp. I'd like to understand from your perspective if you ever conducted any experiments in this direction.

  1. Could you share some experiments or metrics (for evaluating 3D model quality) if you performed on custom data that has a real-world background like the firekeeper dataset? Or any insights about how to evaluate the 3D quality?
  2. Specifically, you mentioned in the repo that training on real-world data using bg_radius can help deal with background data and help reduce noise. Does adding bg_radius also improve the output 3D mesh quality for the real-world data compared to the case without using bg_radius?
  3. Some follow-up works leverage foreground masks for view rendering and mesh extraction. Apart from using bg_radius, have you conducted experiments using 2D foreground masks to segment the object from images and then ran torch-ngp? Could you share some thoughts about if segmentation would help achieve better results vs. non-segmented data (in terms of extracted 3D model quality)?

Looking forward to hearing from you. Thanks again.

ashawkey commented 2 years ago
  1. As far as I know it's hard to evaluate 3D model quality if we don't have GT model. The most reasonable way maybe evaluating the 2D projections with captured GT images.
  2. Yes, it should be able to reduce floaters, which is bad for 3D model.
  3. I haven't tried using foreground mask for realistic dataset, but it is rather easy to do as long as you have the masks. Just like the nerf-synthetic (blender) dataset, add an alpha channel to the RGB image and only set the foreground to 1, then the codebase will handle the other things. It will surely lead to better results according to conclusions from lots of papers, like NeuS.
cubantonystark commented 2 years ago

@ashawkey thanks for this implementation. I have a question: would you add a colorized mesh generation step?

MaroonAmor commented 2 years ago

@ashawkey Thanks a lot for your reply. I have some follow-up questions.

For the 2nd point: The floaters will be reduced for sure. But is the 3D model quality of the central object also improved by using bg_radius?

For the 3rd point: From the NeuS paper, it is still not obvious to make a conclusion if using masks would improve the 3D model quality for NeRF-based approaches. We can see from the quantitative evaluation of the NeuS paper (figure below) that the Chamfer distances w/ mask are not always smaller than w/o mask, and the average distance w/o mask for NeRF is slightly smaller than the case w/ mask. Could you share more insights about it? Please correct me if I misunderstood something. Thanks again. image