search

Anttwo / SuGaR

[CVPR 2024] Official PyTorch implementation of SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering
https://anttwo.github.io/sugar/
Other
2.36k stars 182 forks source link

What is the difference between $\hat{f}$ and $\bar{f}$? #9

Open mkocabas opened 12 months ago

mkocabas commented 12 months ago

Hi @Anttwo,

Congratulations on your preprint! Results look great. I have a couple of questions:

  1. I have a confusion about $\hat{f}$ and $\bar{f}$. What is the difference between them? Is $\hat{f}$ an approximation/efficient version of $\bar{f}$?
  2. How many points do you sample per Gaussian? To be more specific, I am asking the number of points sampled to obtain set $\mathcal{P}$.
  3. How do you compute the depth given multiple Gaussians with different $\alpha_g$?
Anttwo commented 11 months ago

Hi mkocabas,

Thank you so much for your kind words!

Here are some answers to your questions:

  1. The SDF $\bar{f}$ is an ideal SDF, that only holds true in the ideal case where Gaussians are flat, opaque and well spread on the surface. On the contrary, the estimator $\hat{f}$ aims to approximate the real SDF of the current scene, i.e. the distance to the surface represented by the current Gaussian Splatting that is being optimized, and that can be observed in the depth maps (so in theory, this SDF always holds true). Our strategy is to minimize the difference between the two, to enforce Gaussians to converge toward the ideal scenario (flat, opaque and well-spread Gaussians). Please note that we recently updated the paper on arxiv, as we noticed a minor typo in the definition of $\bar{f}$. Formally, we now introduce a more general SDF $f$ to better clarify this point.
  2. We tried (and will propose) different numbers of samples in the code, as this parameter can influence the training speed and the quality of the regularization. In the paper, we simply use 1,000,000 samples per training iteration.
  3. I understand your concerns, as Gaussians with $\alpha_g < 1$ may produce meaningless depth maps. Actually, as we explain in the paper, we also use an additional entropy term on opacities in the regularization, that enforces opacities to take binary values (either 0 or 1). This assumption seems logical as we want to extract an opaque mesh from the Gaussian Splatting. Consequently, we avoid semi-transparent Gaussians, and computing depth maps makes more sense. Please refer to this issue https://github.com/Anttwo/SuGaR/issues/8 for more details about depth computation.

I hope my message provides the answers you need! If not, of course, feel free to ask additional questions.

Best!

mkocabas commented 11 months ago

Thanks @Anttwo for the detailed answer. 1 and 3 is pretty clear, thanks!

Regarding 2, could you give more details how you sample this 1M points? Do you sample fixed number of points per Gaussian e.g. $10^6/n_\text{g}$?