How to generate a good splat of a room

[adricostas]

Hello,

I'm just opening this issue to see if someone can help me to increase the quality of my splat. I'm trying to generate a splat of a room. After several trials using datasets extracted from a video, I decided to try capturing images in a static mode to avoid the motion blur. My dataset has 76 images (3024 × 4032) with enough overlap between them.

I'm using this command to process the data: ns-process-data images --verbose --data $DATASET_PATH --output-dir $OUTPUT_PATH --no-gpu --matching-method exhaustive --matcher_type superpoint+lightglue

It seems that the result of the SfM is correct:

Reconstruction:
    num_reg_images = 76
    num_cameras = 1
    num_points3D = 13394
    num_observations = 48398
    mean_track_length = 3.61341
    mean_observations_per_image = 636.816
    mean_reprojection_error = 1.45261
    num_input_images = 76
[13:07:24] 🎉 🎉 🎉 All DONE 🎉 🎉 🎉                                                images_to_nerfstudio_dataset.py:132
           Starting with 76 images                                                   images_to_nerfstudio_dataset.py:135
           Colmap matched 76 images                                                  images_to_nerfstudio_dataset.py:135
           COLMAP found poses for all images, CONGRATS!                              images_to_nerfstudio_dataset.py:135

Then, I use the following command to generate the splat:

ns-train splatfacto --data $OUTPUT_PATH --pipeline.model.cull_alpha_thresh=0.005 --pipeline.model.continue_cull_post_densification=False --pipeline.model.use-scale-regularization True

The splat seems to be very good in some parts, however there is a lot of noise "in the air":

I don't know why this noise appears if it does not exist in the sparse pointcloud. Can be due to the parts that do not have features like walls, ceil, etc. or the light changes because of the window, for instance? Any advice to improve the result?

Thank you very much in advance!

[jb-ye]

Could you try remove --matcher_type superpoint+lightglue from the colmap commands? and remove --pipeline.model.use-scale-regularization True as well, or simply use splatfacto-big.

Also use the latest nerfstudio commit which has this fix: https://github.com/nerfstudio-project/nerfstudio/pull/3382

[adricostas]

Removing --matcher_type superpoint+lightglue from the colmap commands and --pipeline.model.use-scale-regularization True from train command the result is similar:

I20240829 06:36:11.320797 140647645972160 bundle_adjustment.cc:942]
    Residuals : 328420
   Parameters : 111823
   Iterations : 27
         Time : 10.5064 [s]
 Initial cost : 0.725944 [px]
   Final cost : 0.715718 [px]
  Termination : Convergence

I20240829 06:36:11.320816 140647645972160 timer.cc:91] Elapsed time: 0.181 [minutes]
[06:36:11] 🎉 Done refining intrinsics.                                                              colmap_utils.py:184
[06:36:13] 🎉 🎉 🎉 All DONE 🎉 🎉 🎉                                                images_to_nerfstudio_dataset.py:132
           Starting with 76 images                                                   images_to_nerfstudio_dataset.py:135
           Colmap matched 76 images                                                  images_to_nerfstudio_dataset.py:135
           COLMAP found poses for all images, CONGRATS!                              images_to_n

With splatfacto-big, also a similar result:

I will try with the last version then, but I'm a bit skeptical about that because I have already tried to generate the sparse reconstruction with a external software and execute ns-process with the --skip-colmap flag, and the result is also noisy.

[hariharan1412]

This is a problem with both the dataset and Gaussian splatting, If an area in image presented with very few features ( white/black area/walls ), the model try to fit that area with few larger Gaussians, which will cause the blurryness in the overall splats covers the finer details, No hyperparameter tuning will help you solve this also

If possible, Stick some wall painting in those walls and try again, this will increase the number of features in the wall, This is an experiment, if you could do this, please share your observation and output also

[AntonioMacaronio]

I've always found splatting inside a room to be very difficult - I recommend giving this paper a try: dn-splatter, it does a much better job inside rooms and is built on top of nerfstudio, so it should be very easy to install

[isso313]

I did something similar and obtained good results by turning around in spirals, and you might wanna increase the number of images and try it out again.