question about the error analysis

[Zerg-Overmind]

Hi, thanks for the insightful work! My question is, even the projection error is minimized with the proposed optimal projection, whether the projection error of a 3D Gaussian is 0 only at the (3D Gaussian's) mean location (where mu=x0) of its associate tangent plane? If so, does projection error still exist when you do alpha-blending with a "ray" intersecting with some Gaussians' optimal tangent planes, which are not perpendicular to the ray?

[LetianHuang]

I’m glad to hear your praise and interest in this work!

Yes, the projection error still exists.

As shown in Figure 2(a) of our paper, the minimum value of the error function is greater than zero. This is because Gaussians are closed under affine transformations but not under projection transformations, resulting in the remainder term of the first-order Taylor expansion always being present. This is a trade-off made by original 3D Gaussian Splatting (3DGS) to achieve real-time performance, as the computation for projection is much cheaper compared to dense ray sampling and integration in NeRF.

However, our work achieves this minimum value by selecting the optimal projection plane for each Gaussian, as demonstrated in Figure 3 of the paper. Additionally, because our method combines projection and ray casting, it can achieve effects that are difficult to accomplish with rasterization but are more straightforward with NeRF's ray tracing, such as supporting various camera models as mentioned in the Figure 7 of our paper.

[Zerg-Overmind]

Thanks for the details! Though I agree that the expectation of the projection error of a single 3D Gaussian will be minimized through the projection to its tangent plane, I am worrying that the error might not be even minimized when you calculate the expectation of the projection error along a ray (not a single Gaussian). Once your ray-Gaussian intersection occurs at somewhere not the mean location of a 3D Gaussian. For example, in Fig.3 of the paper, if you are querying x' in ray-casting, the error is not 0 even mu (the projection of the 3D Gaussian's mean location) is the same as x0 (this is already the result after the optimal projection). Similarly, along the depth direction of a ray, the ray will intersect with many other Gaussian's optimal tangent planes but not necessarily at their mean locations, which causes error greater than the single Gaussian case. Please let me know or correct me if I am wrong!

[LetianHuang]

It is indeed true that the projection error of $x^{'}$ will always exist and is not zero, regardless of how the projection plane is adjusted. This is correct because the projection transformation of a Gaussian function does not necessarily result in another Gaussian function, meaning that Gaussian functions are not closed under projection transformations. However, when $x_0$ coincides with $\mu^{'}$, the expected error for the region covered by a single Gaussian function relative to $x^{'}$ is minimized. Note that this is a different issue from the case where $x^{'}$ coinciding with $\mu^{'}$ results in zero error, as rasterization is concerned with the entire region covered by the Gaussian function rather than just its center. Therefore, it is inevitable to encounter situations where $x^{'}$ and $\mu^{'}$ do not coincide. However, the alignment of $x_0$ with $\mu^{'}$ is controllable and can be achieved by selecting the optimal projection plane for each Gaussian function individually, as proposed in our paper.

Furthermore, we calculate the mathematical expectation of the error for a single Gaussian within the Gaussian coverage area, rather than the mathematical expectation for a single ray passing through multiple Gaussians. However, since we minimize the mathematical expectation of the error for each Gaussian through optimal projection, and since each planar Gaussian is optimally projected (note that minimal error does not imply zero error, as previously discussed), the alpha blending of these optimally projected planar Gaussians should also result in minimal error. This can be understood through mathematical induction.

I hope my explanation clarifies your doubts!

[Zerg-Overmind]

I see, thank you for the details! I just want to better understand the paper, there is no doubt at all 😂.