search

Anttwo / SuGaR

[CVPR 2024] Official PyTorch implementation of SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering
https://anttwo.github.io/sugar/
Other
2.36k stars 182 forks source link

How to use the Hybrid representation (Mesh + Gaussians on the surface) in 3d softwares #28

Open xiiliu opened 11 months ago

xiiliu commented 11 months ago

@Anttwo Thanks again for your great work. Have a question about how to use the Hybrid representation in 3d softwares like Blender, Unity, UE, etc. After I use 'train.py' to run the entire pipeline, I got the 'refined_mesh' folder with .obj and .png file, these are mesh and texture. I can import them into Blender, but the rendering effect is not satisfactory. I also get the 'refined' and 'coarse_mesh' folders with .pt files, but I don't know how to use them into Blender. Could you please give me guidance?

Anttwo commented 11 months ago

Hello xiiliu,

I just added a short notebook that shows how to render images with the hybrid representation in Python. However, it's true that there is currently no way to render in a dedicated 3D viewer or Blender the hybrid representation in real-time, as it is a non-trivial engineering task; But I'm working on it! The only limitation was that it was not possible to use the Gaussian Splatting rasterizer in 3D softwares.

However, I saw that there are a few Gaussian Splatting real-time viewers out there, and even Unity, Unreal Engines and Blender extensions. I'm currently taking a look at these, and I will try to propose some viewer or extension to visualize the hybrid representation.

For the presentation video, I first animated the meshes in Blender, then exported the animation files inside python to render with SuGaR. The scripts are inside the ./blender/ directory, and the dedicated class to load the blender camera/animation data inside SuGaR is in ./sugar_scene/sugar_compositor.py. But the script in the repo is not finished yet, since I'm still cleaning the dev code for that.

I will push the final version of this script and provide a tutorial as soon as possible.

xiiliu commented 11 months ago

Thank you for the reply and I got your point.

Yes, there are already some plugins specifically designed for Gaussian Splatting available in Unity or UE, and I have used them. With the help of these plugins, we can do more things in industry. Like this one for Unity: https://github.com/aras-p/UnityGaussianSplatting

But since the rendering way of SuGaR is different with Gaussian Splatting, so the plugins don't work with SuGaR. Hope SuGaR can has its own plugins in 3D softwares soon~

gchhablani commented 3 months ago

@Anttwo Is there an update on this one? Would be nice to be able to export a single file containing all the data!