search

graphdeco-inria / gaussian-splatting

Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Other
12.41k stars 1.54k forks source link

Which Gaussians are optimized? #837

Open robofar opened 3 weeks ago

robofar commented 3 weeks ago

Hello. I have one question regarding which Gaussians are optimized. IfI have set of N Gaussians, and I have one posed frame, in whose viewing frustum lies M Gaussians. Are only these M Gaussians that are in viewing frustum can be optimized, or there is some possibility that Gaussians outside viewing frustum (i.e. some of those N-M Gaussians) can be optimized as well?

So in short, are only Gaussians that are im viewing frustum and that contribute to rendering image are optimized, or not neccessary?

And how you define viewing frustum in your paper? As only front-cone or front and back cone?

hariharan1412 commented 3 weeks ago

Hi, The gaussian splatting code itself will clip those points that lie outside the viewing frame, i.e., it will only optimize the M gaussians. The clipping will happen based on the W and H of the images.

PanagiotisP commented 3 weeks ago

Always keep in mind that "lies" takes into account the projected splat. So If you have M centers inside the frustum, you might have more than M that are optimised. I haven't read the clipping procedure very carefully though, so I can't help with precise details. In any case, the visibility_filter shows you which ones are not clipped and receive a gradient

robofar commented 1 week ago

@PanagiotisP So, you are saying, if center of 3D Gaussian is not in vieweing frustum, but because of it's covariance matrix, that can contribute to rendering from certain viewpoint, it will be optimized as well? Thanks for notifying, makes sense.