Lego dataset score worsens compared to 3DGS model

[Pari-singh]

Hi @jkulhanek , thanks a lot for making your code available to public and your excellent work.

I attempted to train wild-gaussians on the Lego (Blender) dataset using the default configuration (default.yml). However, I noticed that the performance was worse than what the 3DGS baseline paper claims. I suspect that I might have used incorrect configurations for this particular dataset. I was able to achieve similar numbers for phototourism datasets as mentioned in your paper. (I ran it on your code before your last update, and so had to make some changes to the nerfbaseline code to read from yml). However, for lego dataset, the best number I got for PSNR is 24.5 with 200K iterations. 3DGS code claims to achieve PSNR of 35. Was wondering if you have tried the experiment with lego dataset and what were the hyperparams you used? Would be great to see how wild gaussians perform on lego dataset as well.

Thanks a lot in advance.

[jkulhanek]

Hi @Pari-singh. I hope the last version works without the modifications. I wonder if you can perhaps share the error you were getting in case somebody else is having the same issue?

Regarding the blender dataset, I'm not sure if even the random point generation works (there are no 3D points, to begin with), I haven't tested the method on the dataset. Perhaps you can try disabling appearance and uncertainty modeling to see if you can get somewhere? However, the default hparams are more suitable for real-world captures, I assume they will not work that well for the blender scenes.

Thanks!

[Pari-singh]

Thanks for your prompt response @jkulhanek . The errors were mainly around passing arguments from yaml file. I installed nerfbaselines repo and added that part, as calling nerfbaselines train command was overlooking any train code in wildgaussians repo. However, I am certain that issue must have been resolved with the new updates, will try it out.

Yes, blender dataset doesn't have 3D points, so I generated random pointclouds hoping that pruning and densifying would eventually work. I did try it without appearance, sky points and uncertainty but the best results were with all features added.

I am curious to learn what aspect of wild gaussians (whether it is due to a certain loss added or Dinov2 comparison) that makes it not work on synthetic lego data. Since its most basic 3DGS code worked. Your insight will be very helpful! I will try some more hyperparam tuning to see if it works.

Thanks once again for your wonderful codebase!

[jkulhanek]

There are plenty of plausible causes for this. But perhaps if you start from the 3DGS default configuration (disabled sky, disabled app modelling, disabled uncertainty), the method reduces to Mip-Splatting + AbsGaussian (GOF) trick (the main difference in hparams is that COLMAP-estimated scene will have a different scale so we needed to adjust pos. lrs and grad thresholds to account for that, but you can take hparams of vanilla 3DGS). The results should definitely be >30. If not perhaps there is something wrong with how/where you sample the points? How do you load the dataset? Do you use nerfbaselines' blender dataset (external://blender/lego)? Do you sample the points uniformly in [-2, 2]^3?

[Pari-singh]

Do you use nerfbaselines' blender dataset (external://blender/lego) : I didn't download it using (external://blender/lego) as it was not passing 3D points since the dataset doesn't have one. I instead downloaded it locally but added random point generation in the load_blender_dataset function inside nerfbaselines' blender.py. This was taken from 3DGS code.

Do you sample the points uniformly in [-2, 2]^3: Since I used 3DGS random pointcloud generation, they take points between [-1.3, 3] instead. xyz = np.random.random((num_pts, 3)) * 2.6 - 1.3 shs = np.random.random((num_pts, 3)) / 255.0

And I return the xyz and rgb component of these points to the dataloader: points3D_xyz, points3D_rgb = (xyz).astype(np.float32), (SH2RGB(shs)*255).astype(np.uint8)

The experiment which gave 24.5 PSNR above used this config: "config_overrides": { "iterations": 200000, "densify_from_iter": 4000, "densify_until_iter": 100000, "densification_interval": 400, "opacity_reset_interval": 15000, "position_lr_max_steps": 200000, "position_lr_final": 1.6e-07, "position_lr_init": 1.6e-05, "scaling_lr": 0.0005, "sh_degree": 3, "uncertainty_warmup_start": 35000, "uncertainty_warmup_iters": 5000, "uncertainty_protected_iters": 1500, "uncertainty_regularizer_weight": 0.5, "uncertainty_mode": "dino", "uncertainty_preserve_sky": false, "appearance_enabled": true, "appearance_n_fourier_freqs": 4, "num_sky_gaussians": 100000 },

[jkulhanek]

Hi, sorry for the late response, I was on vacation. Did you figure out the issue?