Anttwo / SuGaR

[CVPR 2024] Official PyTorch implementation of SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering
https://anttwo.github.io/sugar/
Other
2.28k stars 175 forks source link

Bad opacity filtering with Binary Opacity Grids #168

Open rebecca-lay3rs opened 7 months ago

rebecca-lay3rs commented 7 months ago

Hi Antoine!

Thanks for the really great work!

I would like to ask if you have checked this paper "Binary Opacity Grids" (https://arxiv.org/pdf/2402.12377.pdf)? They also use an opacity threshold to identify the surface, and they further complement this by "filtering out" the wrong opacity predictions using volumetric fusion (by re-projecting the opacities on 2D depth maps, Fig 3 and Appendix A of the paper).

Have you thought about implementing some of this paper's features on SuGaR? I was wondering if this could help filtering the bad gaussians that we obtain when using objects without background such as:

image

Thank you!

Anttwo commented 7 months ago

Hello @rebecca-lay3rs,

Thank you so much for your nice words!

Indeed, I've checked this paper! You're right, this is a very interesting suggestion, thanks for that. I'm certain SuGaR mesh extraction process can benefit from such strategies. We will definitely explore this direction.

Also, for this specific problem of background, you're right, the current version of SuGaR has trouble with images of segmented objects, as flat monochrome backgrounds create monochrome artifacts (basically, SuGaR tries to reconstruct the background as a monochrome surface, which is actually a pretty dumb problem haha, we should just tell SuGaR not to consider this part of the image). I actually (more or less) fixed this problem with a very simple solution and will push a new version of the code in (I hope) a few days that can handle images of segmented objects (i.e. with masked background).

This fix could probably improve the quality of your reconstruction!