Possible to use a pre-trained gaussian splatting and export the SAM extraction into the .ply file?

[abrahamezzeddine]

Hello,

Is it possible to load a pre-trained 3D Gaussian splatting (.ply) model from Nerfstudio and perform semantic segmentation on it? Specifically, I want to segment the entire model, regardless of the view, and export these segments back into the .ply file.

Can I extract the complete semantic segmentation of a Gaussian splatting model, ensuring that all objects in the scene (for example a room with all its objects) are segmented and these segments are stored as unique attributes within the .ply file? The goal is to avoid running SAM (Segment Anything Model) repeatedly on the same model because I need to store this information.

How can I export or append the segmented data into the .ply file for each respective point?

I hope this made sense, thank you! :)

[Jumpat]

Hi,

Is it possible to load a pre-trained 3D Gaussian splatting (.ply) model from Nerfstudio and perform semantic segmentation on it?

In fact I don't know the data structure of Splatfacto of NeRFStudio. SAGA definitely can be re-implemented with NeRFStudio. However, converting the splatfacto ply to the original GS form is another story.

Specifically, I want to segment the entire model, regardless of the view, and export these segments back into the .ply file.

This is not hard theoretically. You can use the binary mask we stored as xxx.pt in the segmentation_res folder (though the GUI or notebook) to extract the required Gaussians (and necessary attributes) for rendering the specific 3D object.

Can I extract the complete semantic segmentation of a Gaussian splatting model, ensuring that all objects in the scene (for example a room with all its objects) are segmented and these segments are stored as unique attributes within the .ply file? The goal is to avoid running SAM (Segment Anything Model) repeatedly on the same model because I need to store this information.

SAGA is not a semantic segmentation model, which conducts class-agnostic segmentation instead. You can check the notebook for the 3D clustering section, where you will learn how to generate 3D segmentation clusters automatically and you can store them as you like. Please note that in inference we do not run SAM repeatedly. SAM is only used to extract masks for multi-view images in data preparation.

How can I export or append the segmented data into the .ply file for each respective point?

Please refer to Q2.

[abrahamezzeddine]

Hi,

Is it possible to load a pre-trained 3D Gaussian splatting (.ply) model from Nerfstudio and perform semantic segmentation on it?

In fact I don't know the data structure of Splatfacto of NeRFStudio. SAGA definitely can be re-implemented with NeRFStudio. However, converting the splatfacto ply to the original GS form is another story.

Specifically, I want to segment the entire model, regardless of the view, and export these segments back into the .ply file.

This is not hard theoretically. You can use the binary mask we stored as xxx.pt in the segmentation_res folder (though the GUI or notebook) to extract the required Gaussians (and necessary attributes) for rendering the specific 3D object.

Can I extract the complete semantic segmentation of a Gaussian splatting model, ensuring that all objects in the scene (for example a room with all its objects) are segmented and these segments are stored as unique attributes within the .ply file? The goal is to avoid running SAM (Segment Anything Model) repeatedly on the same model because I need to store this information.

SAGA is not a semantic segmentation model, which conducts class-agnostic segmentation instead. You can check the notebook for the 3D clustering section, where you will learn how to generate 3D segmentation clusters automatically and you can store them as you like. Please note that in inference we do not run SAM repeatedly. SAM is only used to extract masks for multi-view images in data preparation.

How can I export or append the segmented data into the .ply file for each respective point?

Please refer to Q2.

Thank you for the quick answer,

This is not hard theoretically. You can use the binary mask we stored as xxx.pt in the segmentation_res folder (though the GUI or notebook) to extract the required Gaussians (and necessary attributes) for rendering the specific 3D object.

Can you provide more details on how the data is stored in that file so I can understand its structure.

[Jumpat]

In this file is a binary torch tensor with size N_G, where N_G denotes the number of 3D Gaussians in the 3D-GS model.