negative value for gaussian.rgb

[antonyvan]

Hi, Thanks you very much for your very clean coding work. I am using your code for training some scenes. some of the scenes looks fine, but others just totally fails. I checked the saved final gaussians. gaussian.rgb have negative values, its range is not within 0-1 or 0-255. I borrowed come code to save it as ply file and tried also convert gaussian.rgb to sperical harmonics dc by rgb/C0 + 0.5, which doesn't give the correct color when display ply with some open source gaussian splatting viewer. Here is the code for saving gaussian to ply. The gaussians when rendered with viewer looks pale.

import os from os import makedirs from plyfile import PlyData, PlyElement def construct_list_of_attributes(_features_dc, _features_rest, _scaling, _rotation): l = ['x', 'y', 'z', 'nx', 'ny', 'nz']

All channels except the 3 DC

  for i in range(_features_dc.shape[1]):
      l.append('f_dc_{}'.format(i))
  for i in range(_features_rest.shape[1]):
      l.append('f_rest_{}'.format(i))
  l.append('opacity')
  for i in range(_scaling.shape[1]):
      l.append('scale_{}'.format(i))
  for i in range(_rotation.shape[1]):
      l.append('rot_{}'.format(i))
  return l

def save_ply(path, gaussian): makedirs(os.path.dirname(path), exist_ok= True) xyz = gaussian.xyz.detach().cpu().numpy() normals = np.zeros_like(xyz) rgb = gaussian.rgb.detach().cpu().numpy() # * 0.28209479177387814 + 0.5 sh = gaussian.sh.detach().flatten(start_dim=1).contiguous().cpu().numpy() opacities = gaussian.opacity.detach().cpu().numpy() scale = gaussian.scale.detach().cpu().numpy() rotation = gaussian.quaternion.detach().cpu().numpy() dtype_full = [(attribute, 'f4') for attribute in construct_list_of_attributes(rgb,sh, scale, rotation)] elements = np.empty(xyz.shape[0], dtype=dtype_full) attributes = np.concatenate((xyz, normals, rgb, sh, opacities, scale, rotation), axis=1) elements[:] = list(map(tuple, attributes)) el = PlyElement.describe(elements, 'vertex') PlyData([el]).write(path)

the gaussian splatting viewer I am using is https://github.com/limacv/GaussianSplattingViewer

The following is one of the results We got reasonably good result for this scene by official gaussian splatting. If you want the data to have a test. I could send it you.

[joeyan]

Hi Antony,

Thanks for the feedback. I haven't tried to view any of the splats in external viewer yet but that is definitely something that I want to support.

Could you answer these few questions to help narrow down the issue:

1. Does this color issue only occur in the gaussian splatting viewer or does this also happen in the saved training debug images?
2. Do you know if the official Inria implementation saves the opacity data with the sigmoid activation function applied? Perhaps that could be what is happening here?

[joeyan]

If you have some data, I can try to reproduce the issue. I probably won't have time this week but can get to it early next week.

Another thing I thought of - can you get the external viewer to work with one of the Mip-Nerf360 scenes?

[antonyvan]

Hi, Gaussian splatting viewer works fine with ply files produced by official gaussian splatting code.

The training with Mip-nerf360 scenes (your provided data) is pretty the same as your results , saved training debug images looks good, but gaussian splatting viewer with the saved ply file looks pale. (Probably its caused by minor difference of official rendering code and yours )

I will check how official version save opacities. (applied sigmoid or not)

[antonyvan]

I checked the opacity of official ply files. it doesn't apply sigmoid.

[antonyvan]

Here is the data https://drive.google.com/file/d/1k1FsiSPzVD6qR8UYwNEbYrcoY2dmJ8iR/view?usp=sharing Please download it. once it is done, please let me know I will stop the sharing.

By the way I modified your dataloader a little bit so that I don't need create multiple image downsample folders

[joeyan]

It seems like the issue with the GaussianSplattingViewer that you linked isn't unique to your dataset. I should be able to reproduce that one without your dataset.

Did you also have an issue training your custom scene? If so, can also share the settings that you used?

[antonyvan]

Actually the data I am sharing with you is my custom scene : ship_share. The image above is test result, which looks the training fails.

the setting I am using to train the ship_share scene is just your default settings . (by the way I turn off the tyro , and directly use SplatConfig, as tyro doesn't allow me to do step-by-step run of the code. -- Your code is really clean, I really enjoy it :))

config = tyro.cli(SplatConfigs,)

config = SplatConfig( num_iters=30000, adaptive_control_start=1500, adaptive_control_end=27500, adaptive_control_interval=300, reset_opacity_end=27500, use_background_end=28000)

[joeyan]

I won't have time to dig into this until next week. I have a few ideas that might help you debug in the mean time:

1) Can you try saving one of your train/test images after it's been loaded and converted to [0, 1] range? 2) Can you try with the spherical harmonics turned off? Maybe also try turning off all of the adaptive control steps too. 3) If you think the issue is RGB related, you can try using a sigmoid activation on RGB then dividing by C0. I think you also need apply inverse sigmoid and remove the divide by C0 in the data loader. Here is a good place to apply the activation.

[antonyvan]

Thanks, I will try these.