Initial findings

[slimbuck]

Hello,

We swapped in rain-gs instead of the original on our generation backend and find that for the same scene:

• training is more than 2x slower (75 mins vs 33 mins)
• fewer splats are generated (2M vs 3M)
• there appear to be more 'floaters' in the scene compared to original
• missing detail on focus areas compared to original

Could you give any advice, presumably we are mis-configuring rain-gs?

Thanks!

[slimbuck]

You can download the output scenes at https://drive.google.com/file/d/1flYURakBs0hBQn2FTfRdqXj_a8ORgeKQ/view

This scene was generated from 94 images and 30k iterations.

[crepejung00]

Hi, we couldn't observe the specific downfalls you have mentioned in our experiments, so it seems as some kind of error to us in first glance. We will check it asap with the provided images.

Thanks for sharing this!

[crepejung00]

Hmm, it is quite weird. However, after visualizing the point clouds with the viewer, it seemed as if the two point clouds were defined in a different coordinate system. Could you also provide the images and poses you used to train?

[slimbuck]

HI @crepejung00 ,

I made a copy of the input dataset for both original and rain-gs. The reason they're different is rain-gs only compiled under cuda 12 for us, but colmap requires cuda 11. Therefore we disabled GPU acceleration in colmap. That might explain the difference.

The undistorted data is available at https://drive.google.com/file/d/143Q_fEisp8Fsu5LZNYCIerl2n6IL7Ko6/view

Thanks again!

[slimbuck]

To be sure, I ran rain-gs on the gpu-enabled colmap data, the results were very similar.

Here is the console output when training:

root@pipeline-gs-generation-7788577ff4-59rfk:/playcanvas/submodules/RAIN-GS# /playcanvas/pipeline/jobs/gs-generation/scripts/run-train.sh '--source_path' '/tmp/defender/undistorted' '--exp_name' '/tmp/defender/result' '--eval' '--ours' '--quiet' '--resolution' '1600' '--iterations' '30000' '--sh
_degree' '0' '--save_iterations' '30000'
Optimizing 
========= USING OUR INITIALISATION =========
args: Namespace(DSV=False, c2f=True, c2f_every_step=1000, c2f_max_lowpass=300, checkpoint_iterations=[], compute_cov3D_python=False, convert_SHs_python=False, data_device='cuda', debug=False, debug_from=-1, densification_interval=100, densify_from_iter=500, densify_grad_threshold=0.0002, densify_until_iter=15000, detect_anomaly=False, eval=True, exp_name='/tmp/defender/result', feature_lr=0.0025, images='images', ip='127.0.0.1', iterations=30000, lambda_dssim=0.2, model_path='', num_gaussians=10, opacity_lr=0.05, opacity_reset_interval=3000, ours=True, output_path='./output/', percent_dense=0.01, port=6009, position_lr_delay_mult=0.01, position_lr_final=1.6e-06, position_lr_init=0.00016, position_lr_max_steps=30000, project_name='gaussian-splatting', quiet=True, random_background=False, resolution=1600, rotation_lr=0.001, save_iterations=[30000, 30000], scaling_lr=0.005, sh_degree=0, source_path='/tmp/defender/undistorted', start_checkpoint=None, test_iterations=[7000, 30000], white_background=False, white_bg=False)
Failed to start GUI server, retrying with port 6010...
GUI server started at 127.0.0.1:6010
Output folder: /tmp/defender/result
Logging progress to Tensorboard at /tmp/defender/result
Set divide_ratio to 0.7
Reading camera 90/90
Generating OUR point cloud (10)...
The file /tmp/defender/undistorted/sparse/0/points3D.ply has been removed successfully.
Loading Training Cameras
Loading Test Cameras
Number of points at initialisation :  10
Training progress:   0%|                                                                                                                                                                                                                                                      | 0/30000 [00:00<?, ?it/s][ITER 1] Low pass filter : 300
Training progress:   3%|██████▌                                                                                                                                                                                                 | 990/30000 [05:40<2:33:11,  3.16it/s, Loss=0.2202263, num_gaussians=93][ITER 1000] Low pass filter : 300
Training progress:   7%|█████████████                                                                                                                                                                                        | 1990/30000 [09:51<1:46:40,  4.38it/s, Loss=0.1963298, num_gaussians=6614][ITER 2000] Low pass filter : 24.33285129339652
Training progress:  10%|███████████████████▌                                                                                                                                                                                | 2990/30000 [13:40<1:51:54,  4.02it/s, Loss=0.1719145, num_gaussians=69142][ITER 3000] Low pass filter : 2.327637014470576
Training progress:  13%|█████████████████████████▉                                                                                                                                                                         | 3990/30000 [17:40<1:42:26,  4.23it/s, Loss=0.1414735, num_gaussians=159615][ITER 4000] Low pass filter : 1.0082854271498578
Training progress:  17%|████████████████████████████████▍                                                                                                                                                                  | 4990/30000 [21:46<1:42:01,  4.09it/s, Loss=0.1120324, num_gaussians=378607][ITER 5000] Low pass filter : 0.42507792633132657
Training progress:  20%|██████████████████████████████████████▉                                                                                                                                                            | 5990/30000 [32:24<1:47:17,  3.73it/s, Loss=0.1052597, num_gaussians=648829][ITER 6000] Low pass filter : 0.3
Training progress:  23%|█████████████████████████████████████████████▍                                                                                                                                                     | 6990/30000 [37:28<1:57:32,  3.26it/s, Loss=0.1025546, num_gaussians=794899][ITER 7000] Low pass filter : 0.3
Training progress:  23%|█████████████████████████████████████████████▌                                                                                                                                                     | 7000/30000 [37:31<1:57:03,  3.27it/s, Loss=0.0855694, num_gaussians=794899]
[ITER 7000] Evaluating test: L1 0.06972833909094334 PSNR 20.14869260787964 LPIPS(vgg) 0.4820743997891744 SSIM 0.6587514877319336

[ITER 7000] Evaluating train: L1 0.04966195672750473 PSNR 22.617563629150393 LPIPS(vgg) 0.4640302419662476 SSIM 0.6794320344924927
Training progress:  23%|███████████████████████████████████████▍                                                                                                                                 | 7000/30000 [39:03<1:57:03,  3.27it/s, Loss=0.1103996, num_gaussians=823847]                Training progress:  26%|██████████████████████████████████████████████▊                                                                                                                                   | 7890/30000 [43:37<1:54:48,  3.21it/s, Loss=0.1111516, num_gaussians=1043578]      Training progress:  27%|█████████████████████████████████████████████████                                                                                                                                       | 7990/30000 [44:05<1:52:01,  3.27it/s, Loss=0.0912930, num_gaussians=1067968][ITER 8000] Low pass filter : 0.3
Training progress:  30%|███████████████████████████████████████████████████████████████████▋                                                                                                                                                              | 8990/30000 [49:23<1:51:48,  3.13it/s, Loss=0.0829622, num_gaussians=1316939][ITER 9000] Low pass filter : 0.3
Training progress:  33%|███████████████████████████████████████████████████████████████████████████▎                                                                                                                                                      | 9990/30000 [55:07<1:58:23,  2.82it/s, Loss=0.0646815, num_gaussians=1306469][ITER 10000] Low pass filter : 0.3
Training progress:  37%|█████████████████████████████████████████████████████████████████████████████████▋                                                                                                                                             | 10990/30000 [1:01:17<1:57:24,  2.70it/s, Loss=0.0824812, num_gaussians=1537313][ITER 11000] Low pass filter : 0.3
Training progress:  40%|█████████████████████████████████████████████████████████████████████████████████████████▏                                                                                                                                     | 11990/30000 [1:07:32<1:53:08,  2.65it/s, Loss=0.0834664, num_gaussians=1736927][ITER 12000] Low pass filter : 0.3
Training progress:  41%|████████████████████████████████████████████████████████████████████████████████▍                                                                                                                    | 12240/30000 [1:09:20<2:00:06,  2.46it/s, Loss=0.1027828, num_gaussians=1495563]Training progress:  43%|█████████████████████████████████████████████████████████████████████████████████████▎                                                                                                               | 12990/30000 [1:14:13<1:51:33,  2.54it/s, Loss=0.0813751, num_gaussians=1656843][ITER 13000] Low pass filter : 0.3
Training progress:  47%|███████████████████████████████████████████████████████████████████████████████████████████▊                                                                                                         | 13990/30000 [1:20:50<1:46:34,  2.50it/s, Loss=0.0797466, num_gaussians=1840377][ITER 14000] Low pass filter : 0.3
Training progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30000/30000 [2:35:06<00:00,  3.22it/s, Loss=0.0656560, num_gaussians=1995352]

[ITER 30000] Evaluating test: L1 0.05880104067424933 PSNR 21.4482634862264 LPIPS(vgg) 0.40298839410146076 SSIM 0.7030609846115112

[ITER 30000] Evaluating train: L1 0.0321400985121727 PSNR 26.172897338867188 LPIPS(vgg) 0.3390174925327301 SSIM 0.7882953882217407

[ITER 30000] Saving Gaussians

Training complete.

[crepejung00]

We just trained rain-gs on the default configuration with the provided images and compared to the original random initialization (DSV), RAIN-GS outperforms DSV in both training time (RAIN-GS : 33min, DSV : 40min) in a single RTX 3090 GPU, and rendering quality. We provide the trained point clouds on the google drive in the following link google drive .

One of my question is that is the defender_original.ply point cloud you first shared trained from SfM initialized point clouds? Our work aims to train 3DGS without any initialized point clouds, and we do not compare our performance with SfM initialized 3DGS. If you are asking that the original 3DGS trained from SfM point clouds is better than RAIN-GS that is correct. We show that we can achieve compelling results without any accurate initialized point clouds.

Thanks!

[slimbuck]

Hi @crepejung00,

Oh I see, yes that makes sense. I was comparing results against the original SfM point cloud version.

Seeing the two results you posted, it's obvious how much better rain-gs is compared to DSV.

Thanks for clear all your help!!