search

autonomousvision / mip-splatting

[CVPR'24 Best Student Paper] Mip-Splatting: Alias-free 3D Gaussian Splatting
https://niujinshuchong.github.io/mip-splatting/
Other
1.01k stars 65 forks source link

Add Real-Time Rendering support using SIBR. #9

Open rahul-goel opened 9 months ago

rahul-goel commented 9 months ago

Hi.

I wanted to visualize these gaussians in real-time so I added the support for it. The following video shows a quick test I did on the synthetic ficus scene which went till 3000 iterations only.

Screencast from 04-12-23 02:03:04 AM IST.webm

Firstly, I've added another method to save the plyfile which is saved as point_cloud_sibr.ply. I apply the change to scaling and opacity and then save the values. Accordingly, I don't pass these values through the activation function in the real-time viewer. The real-time viewer will look for the point_cloud_sibr.ply file.

The Cuda Rasterizer files have been taken from your rasterizer implementation and modified according to the arguments that the real-time viewer passes. To the kernel_size parameter, a value of 0.2 is passed.

The instructions to install and run the SIBR viewer are the same as the one mentioned in the original gaussian splatting repository: https://github.com/graphdeco-inria/gaussian-splatting. Their viewer downloaded the cuda rasterization submodule. Instead, I've just put it directly in the directory. The instructions to launch the viewer are also the same.

I've tested this only on an Ubuntu Machine. I cannot test on Windows.

niujinshuchong commented 9 months ago

@rahul-goel Thank you so much for the PR. I think it would be better to create a fused ply that is compatible to the original 3DGS since in this case we just need to update the rasterisation. I just push the code for creating fused ply file which also consider activation function.

Btw, I think it would be good to have a script of this conversion since people can use it only when they need it otherwise it takes a lot storage if we save it always after training.

The kernel_size is 0.1 by default.

Would be great if you can update it accordingly to the latest commit and I will merge it then. Thank you very much again for the PR.

rahul-goel commented 9 months ago

Hi.

Cool idea to invert the activation functions. I've made changes to the SIBR accordingly. The kernel size is changed to 0.1.

To visualize, the filename should still be "point_cloud_sibr.ply" and be in the same directory as "point_cloud.ply". That's because SIBR expects this directory structure.

niujinshuchong commented 9 months ago

@rahul-goel Thanks for the update! Can you make it optional for saving the SIBR point cloud and default it to false?

rahul-goel commented 9 months ago

I don't understand what you mean by that.

niujinshuchong commented 9 months ago

Hi, I mean now the SIBR points are always saved along with the original point cloud and this doubles the storage. I think it would be better to have a flag that controls whether we want to save it or not.

rahul-goel commented 9 months ago

Ah. I missed removing that. Fixed.

niujinshuchong commented 9 months ago

Hi, thanks for the update. Since now only the SIBR project is update and there will be 2000+ files in this commit, I think it would be better to make it as an external module as done in the original 3DGS project. So could you make it as a standalone module in your own fork and i will link it and update the readme accordingly.

rahul-goel commented 9 months ago

Hi.

I also thought so. But I think it is better if you setup the module and host it yourself instead of me. Because if there is a change in your rasterizer, then I would have to make that change in the submodule also.

So I think it is best if you do it. Once done, you can close this PR without merging.

Thanks