about dL_dmean2D

[CanCanZeng]

Hi, thank you for sharing this reproduce work. Your notes are very helpful for 2DGS learners.

When I checked the code, I found that it seemed that dL_dmean2D was not calculated. If dL_mean2D was not implemented, how would the algorithm work? What I mean is that the later stage of densification depends on the gradient of the projected point.

I'm trying to implement AbsGS based on 2DGS, but since the projection points of 2DGS are different from 3DGS, obtaining the Abs gradient of the projected point is not trivial. Can you give me some help?

[will-zzy]

Hello! I'm very glad to be of help with your studies!

The two-dimensional gradients in 2DGS are not obtained by calculating gradients of the color with respect to the projected points; instead, they are derived from projecting the three-dimensional gradients of the splat centroids onto the image plane. This is mentioned in section 6.1, "Implementation," of the paper.

In fact, we also computed the gradients of the color with respect to the projected points, e.g., in backward.cu line 655, which is used for the gradient backpropagation of the filter. However, I found that the code I wrote for this part did not perform well, which led me to comment it out. But the path for gradient backpropagation is consistent with the official code, and you can further your study by combining my documentation with the official code.

Best wishes!

Original Email

Sender:"CanCanZeng"< @.*** >;

Sent Time:2024/6/17 1:22

To:"will-zzy/2dgs-non-official"< @.*** >;

Cc recipient:"Subscribed"< @.*** >;

Subject:[will-zzy/2dgs-non-official] about dL_dmean2D (Issue #1)

Hi, thank you for sharing this reproduce work. Your notes are very helpful for 2DGS learners.

When I checked the code, I found that it seemed that dL_dmean2D was not calculated. If dL_mean2D was not implemented, how would the algorithm work? What I mean is that the later stage of densification depends on the gradient of the projected point.

I'm trying to implement AbsGS based on 2DGS, but since the projection points of 2DGS are different from 3DGS, obtaining the Abs gradient of the projected point is not trivial. Can you give me some help?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[CanCanZeng]

Thank you for your reply. I think I made a mistake yesterday. I only noticed the variable dL_dmean2Dand didn't notice that the gradient of the image point you returned is p2_gradient.

[CanCanZeng]

Can I assume that only when dist3 is greater than dist2 does the loss have a gradient on the image point? Otherwise, the other case that the computational flow follows is independent of the image point. If my idea is correct, then this gradient does not seem to align with the intention of the 3DGS authors. The 3DGS authors observed that the over and under reconstruction areas had a greater gradient on the projected point, so they chose this indicator as the criterion for whether to clone or split the Gaussian.

[will-zzy]

Can I assume that only when dist3 is greater than dist2 does the loss have a gradient on the image point? Otherwise, the other case that the computational flow follows is independent of the image point. If my idea is correct, then this gradient does not seem to align with the intention of the 3DGS authors. The 3DGS authors observed that the over and under reconstruction areas had a greater gradient on the projected point, so they chose this indicator as the criterion for whether to clone or split the Gaussian.

You are correct in your understanding of the gradients for dist3 and dist2. However, the concept behind 2DGS is not to use the gradients of projection points on the image to represent whether the scene is under-reconstructed or over-reconstructed. Instead, it involves projecting the gradients of three-dimensional points onto the image. If the scene is either over-reconstructed or under-reconstructed, there will similarly be significant gradients of the three-dimensional points, which is consistent with 3DGS.

The reason 2DGS operates this way is that the projection transformation is not affine. Projecting the Gaussian centroids onto the image might not align with the center points of the two-dimensional Gaussian patterns. Therefore, they believe that projecting the gradients of the three-dimensional centroids onto the image plane more accurately reflects the "severity of the actual Gaussian movement." You can visualize this in the official code’s Python demo by plotting the projection points of the Gaussian centroids and the center points of the bounding boxes; you will notice a slight difference between the two.