How to improve quality of mesh?

[kusstox]

Hi there First of all, thanks for the amazing work. I am looking forward for the updates!

My question is, how can I improve the quality of my mesh? I tried to reconstruct an indoor plant. But I always get these grey artifacts around the leafs:

I am using 360° videos, use ffmpeg to create around 3 fps of images resulting in around 200 images. For the whole pipeline, I am using the standard parameters which are given by the train script.

Any ideas are welcome. Thank you very much.

[Anttwo]

Hey @kusstox,

Thanks for you nice words!

Good news is, I know why you have these artifacts. This is due to the fact that (a) your images do not cover both sides (especially the lower face) of the leaves, and (b) the leaves are very thin. Please see the end of this message to get a practical solution.

Here are more details: This is related to what I wrote in the Tips section of the README.md file:

Using a smartphone or a camera, capture images or a video that cover the entire surface of the 3D scene you want to reconstruct. The easiest way to do this is to move around the scene while recording a video. Try to move slowly and smoothly in order to avoid motion blur. For consistent reconstruction and easier camera pose estimation with COLMAP, maintaining a uniform focal length and a constant exposure time is also important. We recommend to disable auto-focus on your smartphone to ensure that the focal length remains constant.

For better reconstructions, try to cover objects from several and different angles, especially for thin and detailed parts of the scene. Indeed, SuGaR is able to reconstruct very thin and detailed objects, but some artifacts may appear if these thin objects are not covered enough and are visible only from one side in the training images. Detailed explanation SuGaR applies Poisson reconstruction with 3D points sampled on the parts of the surface that are visible in the training images. This visibility constraint is important to prevent sampling points on the backside of the Gaussian level sets, located behind the surface of the scene, which would produce a lot of self-collisions and many unnecessary vertices in the mesh after applying Poisson reconstruction. However, this visibility constraint also means that SuGaR cannot reconstruct parts of the surface that are not visible in the training images. If thin objects are visible only from one side in the training images, the Poisson reconstruction will try to reconstruct a closed surface, and will extend the surface of the thin objects, which produces an inaccurate mesh.

TODO: Add images illustrating such artifacts.

However, such artifacts are not visible in the hybrid representation, because the gaussian texturing gives low-opacity to these artifacts during refinement.

We already have simple ideas that could help to avoid such artifacts, such as (a) identifying new camera poses that cover parts of the surface non-visible in the training images that are likely to be on the same level set as the visible parts, and (b) adding these camera poses to the set of cameras used for sampling the points when applying Poisson reconstruction. We will update the code in a near future to include this.

Basically, if an object is very thin and covered only from one side, you can have this kind of artifacts. This is due to something very dumb: To be sure we do not reconstruct a lot of useless points inside the objects, we sample points only on the parts of the isosurface that are visible in the images.

The problem is, if the surface of a very thin object is visible only from one side in your training images, you may have artifacts on the border between visible and non-visible parts of your mesh.

I actually almost finished to implement a solution for that; The simple solution is to find virtual "camera poses" (that are not in the dataset) but that cover parts of the surface/volume that are not covered by the training images, and sample points on these parts. I'll not be able to push this in the coming days, though.

In the meantime, you have several potential solutions for that:

1. Try to cover more parts of your objects with your capture. Doing a 360° trajectory is good, but for thin objects like leaves you should also try to move your camera up and down to be sure you cover the lower faces of the leaves. SuGaR is able to reconstruct beautiful leaves/plants, as we show with the Counter scene from the Mip-NeRF360 dataset.
2. You can also try to pull the last changes I pushed in the code recently, and increase the parameter vertices_density_quantile at line 43 of sugar_extractors/coarse_mesh.py. This helps reducing the border artifacts.
3. If point (2) does not work, you can also try to use the option --postprocess_mesh True with train.py. This post-processing is a dumb solution for reducing bad-looking triangles on these border artifacts. Sometimes it works well, but I don't think it is a nice solution. The solution I'm currently implementing (using additional virtual cameras for coverage) is much better in my opinion.

I hope this will resolve your concerns. Looking forward to your reply!