search

sharinka0715 / semantic-gaussians

Official implemetation of the paper "Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting".
MIT License
143 stars 11 forks source link

Does it support a pre-trained 3D Gaussian Splat? #6

Closed abrahamezzeddine closed 5 months ago

abrahamezzeddine commented 5 months ago

Hello,

I am wondering if this will support a pre-trained 3DGS from Nerfstudio for example and just append the masks/segmentation into the .ply file?

sharinka0715 commented 5 months ago

Hi, The format of our 3DGS follows the official repository of 3DGS. If you have .ply file trained from this repository, you can directly use them in our repo. I am not sure what format Nerfstudio uses.

abrahamezzeddine commented 5 months ago

@sharinka0715

The PLY have the same file format, same attributes in the header. However, it might be that the data is normalized because normalised data works better on tensors CUDA. But that I can fix.

Could you please explain how I could (if we assume the trained splat PLY is in the same format, can append the semantics into the ply file) with your implementation?

Assume I provide the colmap sparse folder, the images and the finished trained Gaussian Splat via nerfstudio.

Nerfstudio provides a good and clean splats compared to the official implementation.

Thank you very much. Nerfstudio is being widely implemented and if this works it would be great.

sharinka0715 commented 5 months ago

@abrahamezzeddine

I think you can directly load the .ply file of official 3DGS format in fusion.py.

fusion.py, which performs 2D versatile projection, receives the normal 3DGS as input, and it will create semantic features to create Semantic Gaussians. You can set the colmap folder (with images/, sparse/) in scene.scene_path, and set the .ply file in model.model_dir.