Why use median depth instead of accumulated depth?

[XuyangBai]

Hi @JonathonLuiten , thanks for sharing your dynamic 3D gaussian work! May I ask the reason why you use median depth instead of the accumulated depth ( alpha blend the depth in the same way as color) to get the rendered depth image? I do find the former one achieves a cleaner depth map with less long tail effect, but I believe the latter is more friendly to back-propogate the gradient of depth to the gaussians, just curious about your motivation of making this decision choice for you dynamic gaussian splatting paper.

[JonathonLuiten]

It doesn’t matter too much. The depth is never used in training. It’s only used for: a) 2D tracking eval, to figure out which Gaussian best fits to a pixel. Here median is actually a more natural choice to choose a single Gaussian. b) Visualization, eg to put the 3D trajectories into the rendering with occlusion, and to guide the camera motion in interative camera mode when moved around with the mouse. Here from the rendered view it’s almost irrelevant which one you choose. However when you look at a depth map from a different view it becomes obvious that median has some advantages that it doesn’t bleed depth over edges which could be nice (eg for constructing a mesh or something like this).

Overall though, for how I use it, this is not a super critical decision.

Jonathon

[XuyangBai]

Hi @JonathonLuiten Thanks for the quick reply. Yeah I agree it does not matter too much in rendering. But may I ask your opinion about their effect in training? i.e. do you think the median depth is suitable for back-propogation if we have some kind of depth loss for regularization? I personally feel it may not have much effect on training because the gradient is non-zero only at one gaussian for each pixel, which means most of the gaussian does not have gradient from this depth supervision.

[JonathonLuiten]

For training it makes sense to use mean depth for the reason you said (but in my work I don't use depth for training)

[XuyangBai]

Thanks a lot!

[ingra14m]

Sorry to bother you in this closed issue. In my experiments, I think people exaggerate the effect of depth on 3D-GS. Depth loss may help geometry, but it has very little help with rendering. I also think that depth can only be used for visualization. Even if the accumulated depth is used, it is still not suitable for training.