Export ply

[hecodeit]

Is it possible to export training results as a .ply file? If so, how can I do this?

[jkulhanek]

hi, here is the code that shows how to do it: https://github.com/nerfbaselines/nerfbaselines/compare/main...wg-demo But warning: ply (the format used in 3DGS) cannot store multiple appearances, so in the code I've sent you the scene is baked for a single appearance embedding.

[hecodeit]

@jkulhanek

Yes, it's working with nerfbaselines export-demo. The export-demo cli create a mkkellogg/GaussianSplats3D project, which is fantastic!

What did you mean "cannot store multiple appearances"?

1. After finished nerfbaselines train --method wild-gaussians --data {custom data}. Got output.zip and predictions-70000.tar two files.
2. Unzip output.zip got checkpoint, predictions and tensorboard folder. Within checkpoint folder there is a point_cloud.ply file. It's the training result flie? Or it is 'single appearance' file you mention above?
3. How to view the training result? Using nerfbaselines viewer --checkpoint {unziped checkpoint folder}, same with point_cloud.ply and exported mkkellogg/GaussianSplats3D project. So it's 'single appearance'?

Question:

1. How to export full ply (multiple appearances)?
2. How to view full scene (multiple appearances) with nerfbaselines viewer?

[jkulhanek]

What did you mean "cannot store multiple appearances"? WildGaussians uses MLP to produce SHs for different appearance embeddings. The MLPs cannot be stored in the PLY, therefore the PLY is baked for a single appearance embedding

2. The output.zip/checkpoint.tar.gz contain point_cloud.ply, which stores the Gaussians parameters including base SHs colors (before appearance color mapping is applied). This serves as a "mean appearance" representation. However, please do not use the PLY file stored in the checkpoint directly. It stores raw Gaussian parameters before postprocessing: e.g. Mip-Splatting's 3D filter is not applied

3. The viewer will enable you to visualize any appearance embedding. However, also pass the data option to the viewer nerfbaselines viewer --checkpoint {checkpoint folder} --data {data}. Note, that you don't need to unzip the checkpoint folder. You can simply use output.zip/checkpoint as the checkpoint argument. In the render tab in the viewer there is a dropdown menu that allows you to switch between embeddings. Also for each added trajectory keyframe, you can select a different embedding when you click on the camera handle and the popup appears.

Q1: The web demo (mkkellogg's project) does not support viewing different appearances. Therefore, the ply files used with the project need to be baked for a single appearance. However, you can bake few of these ply files and use a toggle to switch between them. This is used in nerfbaselines web platform for example if you look at phototourism results. If you take a look at the export-all-demos.sh script in nerfbaselines/scripts, the demo is created for 3 different embeddings. Ultimately, the nerfbaselines viewer provides the best viewing experience - what you see is exactly the output of the rendering and you can change the appearance embeddings freely.

Q2: I think I've already answered this. Just pass the --data option (dataset needs to be loaded in order for the viewer to know names of training images). Then there is the render tab in the viewer where you can switch between app. embeddings.

[hecodeit]

@jkulhanek Thank you for the detailed answer.

As you suggested, I found the 'Appearance from train image' dropdown in the viewer. So, is it correct to understand that the spherical harmonics of wild-gaussians store the colors of the input images after training, rather than the viewpoint colors corresponding to the original Gaussian Splatting?

Additionally, I have three more questions:

1. The mkkellogg/GaussianSplats3D project generated by nerfbaselines export-demo looks different in color compared to nerfbaselines viewer. I tried nerfbaselines export-demo --data {custom data} --output output or nerfbaselines export-demo --data {custom data} --output output --train-embedding 1. Both exported colors show overall reduced brightness and changed color saturation. How can I export the colors same with nerfbaselines viewer?
2. How can I increase the density of trained Splats? I'm using nerfbaselines train --method wild-gaussians --data {path to data} for training. The output point_cloud.ply has 171k splats. Since the scene is large (2800 images), I need a higher density for the scene.
3. The Skysphere (Skybox) is not displaying. Is this caused by the low density of Splats mentioned above?

[jkulhanek]

I don't understand what you mean. WG stores spherical harmonics of "base colors" and then (given an appearance embedding) applies an MLP to obtain new spherical harmonics corresponding to the appearance vector. Therefore, after baking the appearance embedding you get still view-dependent representation (unlike any other appearance-enabled 3DGS method, which actually requires you to evaluate the MLP for every viewing direction).

1) I've noticed the issue with mkkellogg's viewer as well. I was actively looking into it, which is why I didn't merge the export-demo function for WildGaussians yet. I believe it is perhaps caused by the imprecise sorting in mkkellogg's viewer, but not sure yet. 2) To increase density, change some hparams related to growth and pruning. In particular, you want to push the densification thresholds lower and increase pruning thresholds. 3) Could be multiple reasons. Not sure what is the case with your data. Depends a the position and orientation of the cameras. Are they forward-facing or object-centric?

In general, can you try running the method without uncertainty modeling (if you don't have a large amount of occlusions)? If you have so many images, it is likely not needed even if the amount of occlusion is slightly higher.

[hecodeit]

WG stores spherical harmonics of "base colors" and then (given an appearance embedding) applies an MLP to obtain new spherical harmonics corresponding to the appearance vector. Therefore, after baking the appearance embedding you get still view-dependent representation (unlike any other appearance-enabled 3DGS method, which actually requires you to evaluate the MLP for every viewing direction).

I understand now. This is why the term 'appearance embedding' is used, rather than Spherical Harmonics.

To increase density, change some hparams related to growth and pruning. In particular, you want to push the densification thresholds lower and increase pruning thresholds.

How can I setting parameters of growth and pruning? In --set KEY-VALUE or nb-info,json, and how?

Are they forward-facing or object-centric?

Yes, I'm training unbounded scene which camera facing center of the scene.

[jkulhanek]

How can I setting parameters of growth and pruning? In --set KEY-VALUE or nb-info,json, and how? Just add --set {parameter}={value} for a parameter you want to set. Take a look at the config to see the available parameters: https://github.com/jkulhanek/wild-gaussians/blob/eebbec0f3acd6a44c5e5e2e4e28740f99f6d22cd/wildgaussians/config.py#L14 You can try: --set uncertainty_mode=disabled --set densify_grad_threshold=0.0001 first.