search

Jumpat / SegAnyGAussians

The official implementation of SAGA (Segment Any 3D GAussians)
Apache License 2.0
484 stars 36 forks source link

For rendering the segmentation results #36

Open Lizhinwafu opened 4 months ago

Lizhinwafu commented 4 months ago

python render.py -m <path to the pre-trained 3DGS model> --precomputed_mask --target scene --segment

I have saved multiple precomputed_masks, how can I segment these multiple targets at once?

Jumpat commented 3 months ago

I think the segmentation phase has been finished after you obtained the precomputed_masks. Do you want to render them together? If so, you should merge these precomputed_masks and use the merged mask for rendering.

Lizhinwafu commented 3 months ago

Thank you for your reply. I have always had a question. The current segmentation method of nerf or 3dgs is to first find the mask from 2D, then map the mask to 3D, and segment the 3D target.

This introduces occlusion issues in 2D images and is not directly segmented in 3D space like point clouds.

Many models now are a combination of 2D and 3D. I would like to ask why not perform segmentation directly in the generated 3D GS model? What's the difficulty?

Lizhinwafu commented 3 months ago

The generated 3D GS model retains both spatial information and high image-level fidelity. I think that achieving segmentation directly in 3D space truly reflects its advantages over point clouds.

Jumpat commented 3 months ago

Yeah, the computer vision community is always chasing an end to end method.

However, the key problem of directly segmenting 3D Gaussian is the lack of training data. In other words, there are lots of labeled point clouds but much less labeled 3D Gaussians.

Another fact is that this 2D to 3D paradigm is very suitable to 3D-GS, due to the existence of the differentiable rendering tech. It makes mapping 2D segments to 3D intuitive and reasonable.