Result mesh is very poor for indoor scenes

NVlabs / neuralangelo

Official implementation of "Neuralangelo: High-Fidelity Neural Surface Reconstruction" (CVPR 2023)

https://research.nvidia.com/labs/dir/neuralangelo/

Other

4.33k stars 387 forks source link

Result mesh is very poor for indoor scenes #78

Closed smandava98 closed 1 year ago

smandava98 commented 1 year ago

Hello, I took a video of an indoor scene but only from a frontal view (so I did not have the camera move around the whole room but just a part of it) and I initially expected myself to get color results of that portion where I can move in and out to view objects but the resulting mesh is this colorless blob looking object. I ran the full training loop (500k iterations with no changes in hyperparameters). How were you able to get such good looking meshes from your dataset? Does the data need additional requirements?

When I look at the wandb validation images they look great. It seems to be capturing all details well. Its just the mesh part that looks very off..I just want to somehow get a 3D scene of those wandb images.

Followup: Also why doesn't the mesh come with colors?

chenhsuanlin commented 1 year ago

Hi @smandava98, regarding your questions:

Have you set the scene_type to be indoor in the data processing step?
Have you tried looking inside the blob?
Please see #62 for issues on textured mesh extraction

mli0603 commented 1 year ago

Hi @smandava98

Sorry for the late reply.

Have you looked at the normal maps? Do they look reasonable?
If normal maps look great, my best guess is that your reconstruction result is inside the blob that you are looking at. You can zoom into the blob to look around. Another trick is to visualize the camera poses and mesh together to locate the reconstruction.

I hope these help!

mli0603 commented 1 year ago

We have pushed a commit that potentially fixes the issue of mesh extraction (https://github.com/NVlabs/neuralangelo/commit/c91af8d5098c858df8e8dfa35fba8666d314782b). Please let us know if you still run into the same problem.

smandava98 commented 1 year ago

Thanks both! I will try it out with that and experiment further with better ways to visualize it. Feel free to close the issue and I can re-open it if further need is there. Thanks again for the help.