Improving Montmajour meshes - H100

[rebecca-lay3rs]

Improving Montmajour meshes - H100

Problem 1: Triangle vertices

• Change interpolation method in blender: https://docs.blender.org/manual/en/latest/render/shader_nodes/textures/image.html from linear to closest pixel (https://github.com/Anttwo/SuGaR/issues/119)

Problem 2: Filling up empty spaces in mesh

• mainly behind the object

n_points_in_range = 21

# We compute ray samples
points_range = torch.linspace(-range_size, range_size, n_points_in_range).to(self.device).view(1, -1, 1)  # (1, n_points_in_range, 1)
points_range = points_range * points_stds[..., None, None].expand(-1, n_points_in_range, 1)  # (n_points, n_points_in_range, 1)
samples = (all_world_points[:, None, :] + points_range * camera_to_samples[:, None, :]).view(-1, 3)  # (n_points * n_points_in_range, 3)
samples_closest_gaussians_idx = closest_gaussians_idx[:, None, :].expand(-1, n_points_in_range, -1).reshape(-1, self.knn_to_track)

Problem 3: Background:

• Idea 1) removing the points in the point cloud that we dont want! -> too complicated to do in COLMAP, would have to rewrite multiple files so lets put this in pause

• Idea 2) removing the background of the objects in every photo of the dataset and inserting it in COLMAP to obtain the point cloud only around the object, without the points of the background:

folder: Cloitre_Gal1-2_A2_sheep_tests command: python train.py -s data/Cloitre_Gal1-2_A2_sheep_tests/ -c outputs_GS/Cloitre_Gal1-2_A2_sheep_tests/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Cloitre_Gal1-2_A2_sheep_tests

	with background	without background
images
point3D.ply
mesh.obj text
mesh.obj gray
side view

conclusion: Sugar struggles to treat objects without background BUT the side part is better defined without the background

• Idea 3) Input the points3D.ply without background on Sugar but input the images with background (sort of mixed dataset)

folder: Cloitre_Gal1-2_A2_sheep_tests_2 command: python train.py -s data/Cloitre_Gal1-2_A2_sheep_tests_2/ -c outputs_GS/Cloitre_Gal1-2_A2_sheep_tests/ -r "sdf" --low_poly True --poisson_depth 8 --experiment_name Cloitre_Gal1-2_A2_sheep_tests_2

the object is behind the background

conclusion: does not work, Sugar adds the background in front of the object

[rebecca-lay3rs]

• Idea 4) Use GroudingDino + SAM to segment better the input dataset and put it in COLMAP using GDINO+SAM https://github.com/IDEA-Research/GroundingDINO https://github.com/facebookresearch/segment-anything

installation using Docker here: https://github.com/Lay3rss/Grounded-Segment-Anything

how to use mask_path COLMAP https://github.com/colmap/colmap/issues/1016 -> how to use mask_path https://github.com/colmap/colmap/pull/522 -> how to use mask_path https://colmap.github.io/faq.html#mask-image-regions https://github.com/colmap/colmap/issues/954 https://github.com/colmap/colmap/issues/416 -> Filter outlier reconstructed points? https://github.com/colmap/colmap/issues/1251 https://github.com/colmap/colmap/issues/1460 https://github.com/colmap/colmap/issues/208

• Idea 5) Use GroudingDino + SAM to segment better the input dataset and put it in Gaussian Splatting directly

render without background https://github.com/graphdeco-inria/gaussian-splatting/issues/542 https://github.com/graphdeco-inria/gaussian-splatting/issues/127 forked repo from GS with mask implementation: https://github.com/yzslab/gaussian-splatting

./output/coarse/Cloitre_Gal1-2_A2_sheep_masks/sugarcoarse_3Dgs7000_sdfestim05_sdfnorm02/15000.pt Mesh output path: ./output/coarse_mesh/Cloitre_Gal1-2_A2_sheep_masks Surface levels: [0.3] Decimation targets: [200000] Use custom bbox: False Use eval split: True GPU: 0 Use centers to extract mesh: False Use marching cubes: False Use vanilla 3DGS: False

Loading the initial 3DGS model from path outputs_GS/Cloitre_Gal1-2_A2_sheep_masks/... Warning: image extension .jgp not supported. Traceback (most recent call last): File "/root/SuGaR/train.py", line 163, in coarse_mesh_path = extract_mesh_from_coarse_sugar(coarse_mesh_args)[0] File "/root/SuGaR/sugar_extractors/coarse_mesh.py", line 129, in extract_mesh_from_coarse_sugar nerfmodel = GaussianSplattingWrapper( File "/root/SuGaR/sugar_scene/gs_model.py", line 111, in init cam_list = load_gs_cameras( File "/root/SuGaR/sugar_scene/cameras.py", line 126, in load_gs_cameras mask=resized_mask ,) #################### UnboundLocalError: local variable 'resized_mask' referenced before assignment