Fix several issues with the code and make it work nicely on large set of videos

[jb-ye]

Hi @wanmeihuali

I would like to start a PR effort on a few recent fixes we developed for Taichi Gaussian Splatting (Taichi-GS). I had branched out from this commit ( https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/commit/f7631e327d3e6e995324f1755bc3da25d603a584 ) and made a few fixes to make the current code work robustly on a wide range of examples (100+ scenes).

I tested the latest commit on main branch ( https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/commit/2cea5deb3940079efe917ad32a07c8390907507a ), which crashes on many videos, I don't really have clues.

Particularly, I made a set of changes fixes several issues of the current project ( https://github.com/jb-ye/taichi_3d_gaussian_splatting/tree/eames ): (1) It automatically scale the training image to no more than 1600 (if needed). Training higher resolution requires more fine-tuning of existing parameters, otherwise, you may observe undesirable outcomes like I mentioned here https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/issues/144.

(2) It addresses the root cause of https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/issues/119 and the issue was also reported by many other people on this forum who tried this repo.

The key idea to avoid numerical overflow is to always do a 2D convolution in pixel space after splatting 3D Gaussians. In official GS code here and here: the 2D Gaussian is convolved with an isotropic 2D Gaussian of σ=3​⋅I to simulate pixel integration (this is not in the paper). However, even official code is not done this in proper way. See also discussion here.

Let us say you have a 2D Gaussian with an opacity of 1 at the center. When doing a convolution with another 2D Gaussian, if the opacity is currently left unchanged the Gaussian will become larger while remaining opaque and may obscure Gaussians that are behind (we observed that on grid patterns).

When we implemented this strategy for Taichi GS (was not part of original implementation), we noticed in one example this issue becomes quite obvious:

https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/assets/132313008/076f2c3a-47f8-4c50-8b18-54ef8bdab068

When render camera moves from far to close (near the captured distance), we observe the color on the grid pattern of acoustic amplifier changes and creates aliasing like effect (though it is not aliasing). This issue is not usually reflected in standard metrics of validation images if those validation images are captured at similar distance as training images.

Instead, the opacity should be reduced so that the (2D) integrated opacity of the resulting Gaussian is the same as the original one. Thus in the 3DGS code the opacity should be multiplied by the factor as sqrt(∣Σ+0.3I∣/∣Σ∣​)​ . We implemented this rescaling factor into Taichi GS, and observed that the aliasing like effect is gone:

https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/assets/132313008/0f88ed9b-cc29-431d-aefa-2d70ab393b1d

In our internal evaluation, I found this change also improves standard metrics (PSNR, SSIM, LPIPS).

(3) We also reduce the parameter densification-view-space-position-gradients-threshold: 3e-6 by half, and observe nice improvements in standard metrics albeit not being too slow.

Besides those fixes, I also implemented a simple offline renderer for rendering images from a custom trajectory.

I noticed a few more recent interesting changes (e.g. camera optimization, and depth cov loss), but they are not of our primary interest at the moment. Is there a way to move them to a separate experimental branch before they becomes mature and tested.

[wanmeihuali]

@jb-ye Nice work done! You can go ahead and create the PR. I still need some time to read your code. I'll try to roll back the master branch because many people reports crash(I guess the two new features: camera pose optimization and depth covariance loss causes some bug).

[jb-ye]

Okay, once you roll back the main branch. I will create the PR.

[wanmeihuali]

Hi @jb-ye , I've already rolled back the current main branch. You can go ahead and create the PR. Thanks!

[jb-ye]

Code merged. close it.

[sakuratsuyu]

Hi, may I ask why the normalization factor is sqrt(∣Σ∣ / ∣Σ+0.3I∣​) instead of 1 / sqrt(∣Σ+0.3I∣)? What do sqrt(∣Σ∣) compensate for? I mean, to normalized the convolution result such that the integral of the result gaussian is 1, we only exactly need 1 / (2π * sqrt(∣Σ+0.3I∣)), where 1 / 2π can be omitted since it's a constant. @jb-ye