Xinjie-Q / GaussianImage

🏠[ECCV 2024] GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting
https://xingtongge.github.io/GaussianImage-page/
MIT License
180 stars 5 forks source link

When I permute the gaussians dimension, the rendered result will be noisy, why? #13

Open dongzhuoyao opened 2 weeks ago

dongzhuoyao commented 2 weeks ago

"This novel rasterization algorithm brings multiple benefits. First, our accu-mulated blending process is insensitive to the order of Gaussian points. Thisproperty allows us to avoid the impact of the random order of Gaussian pointson rendering"

I mainly permute the _xyz, _cholesky, _features_dc in GaussianImage_Cholesky, this is roughly the function(I only use your overfitting technique(fittign 8 parameters), I didn't use any compression technique):

image

with permute:

render_img2

without permute:

render_img_clean

djx99 commented 2 weeks ago

Yes, I've run into this problem as well. At the end of training, when I shuffle the Gaussians (get_xyz, get_cholesky_elements, get_features) and render the image, the same problem occurs. Here is my code and results. image image 11

I have found that when the number of num_points is small, there is no problem even if it is disrupted; however, when the number of num_points is large (downsampling by a factor of 2, kodak dataset, 50,000 points), the problem occurs.

I suspect it's a problem with the rasterize_forward_sum() function, and I don't think it should have done=true inside the loop for(int b = 0; b < num_batches; ++b), which would cause it not to be able to get into the loop for(int t = 0; (t < batch_size) && ! done; ++t) when b > 0. So some Gaussians aren't being trained. The only time donecan be set to trueis if next_T <= 1e-4occurs in 3D.

Please correct me if I'm wrong.

dongzhuoyao commented 2 weeks ago

For me, even when #GS=1000 in CIFAR10, I still got quite noisy image after permutation.

looking forward to the authors' reply, I need this permutation invariance featrure for my own project