search

yzslab / gaussian-splatting-lightning

A 3D Gaussian Splatting framework with various derived algorithms and an interactive web viewer
Other
431 stars 34 forks source link

readme

Gaussian Splatting PyTorch Lightning Implementation

# clone repository
git clone https://github.com/yzslab/gaussian-splatting-lightning.git
cd gaussian-splatting-lightning

1.2. Create virtual environment

# create virtual environment
conda create -yn gspl python=3.9 pip
conda activate gspl

1.3. Install PyTorch

1.4. Install requirements

pip install -e .

1.5. Install optional packages

2. Training

2.1. Basic command

python main.py fit \
    --data.path DATASET_PATH \
    -n EXPERIMENT_NAME

It can detect some dataset type automatically. You can also specify type with option --data.parser. Possible values are: Colmap, Blender, NSVF, Nerfies, MatrixCity, PhotoTourism, SegAnyColmap, Feature3DGSColmap.

[NOTE] By default, only checkpoint files will be produced on training end. If you need ply file in vanilla 3DGS's format (can be loaded by SIBR_viewer or some WebGL/GPU based viewer):

You can use utils/image_downsample.py to downsample your images, e.g. 4x downsample: python utils/image_downsample.py PATH_TO_DIRECTORY_THAT_STORE_IMAGES --factor 4

# it will load images from `images_4` directory
... fit \
  --data.parser Colmap \
  --data.parser.down_sample_factor 4 \
  ...

Rounding mode is specified by --data.parser.down_sample_rounding_mode. Available values are floor, round, round_half_up, ceil. Default is round.

cli_main(["fit", "--config", "/path/to/config.yaml", "--data.path", "/path/to/data", "args", ...])


### 2.3. Use <a href="https://github.com/nerfstudio-project/gsplat">nerfstudio-project/gsplat</a>
Make sure that command `which nvcc` can produce output, or gsplat will be disabled automatically.
```bash
python main.py fit \
    --config configs/gsplat.yaml \
    ...

2.4. Multi-GPU training (DDP)

[NOTE] Try New Multiple GPU training strategy, which can be enabled during densification.

[NOTE] Multi-GPU training with DDP strategy can only be enabled after densification. You can start a single GPU training at the beginning, and save a checkpoint after densification finishing. Then resume from this checkpoint and enable multi-GPU training.

You will get improved PSNR and SSIM with more GPUs: image

# Single GPU at the beginning
python main.py fit \
    --config ... \
    --data.path DATASET_PATH \
    --model.density.densify_until_iter 15000 \
    --max_steps 15000
# Then resume, and enable multi-GPU
python main.py fit \
    --config ... \
    --trainer configs/ddp.yaml \
    --data.path DATASET_PATH \
    --max_steps 30000 \
    --ckpt_path last  # find latest checkpoint automatically, or provide a path to checkpoint file

2.5. Deformable 3D Gaussians

python main.py fit \
    --config configs/deformable_blender.yaml \
    --data.path ...

2.6. Mip-Splatting

python main.py fit \
    --config configs/mip_splatting_gsplat.yaml \
    --data.path ...

2.7. LightGaussian

2.8. AbsGS / EfficientGS

... fit \
    --config configs/gsplat-absgrad.yaml \
    --data.path ...

2.9. 2D Gaussian Splatting

2.10. Segment Any 3D Gaussians

2.11. Large-scale scene reconstruction with partitioning and LoD

Baseline Partitioning
image image
image image

The implementation here references Block-NeRF, VastGaussian and CityGaussians.

There is no single script to finish the whole pipeline. Please refer to below contents about how to reconstruct a large scale scene.

(1) An example pipeline for the Rubble dataset from MegaNeRF

(2) Utilize multiple GPUs

2.12. Appearance Model

With appearance model, the reconstruction quality can be improved when your images have various appearance, such as different exposure, white balance, contrast and even day and night.

This model assign an extra feature vector $\boldsymbol{\ell}^{(g)}$ to each 3D Gaussian and an appearance embedding vector $\boldsymbol{\ell}^{(a)}$ to each appearance group. Both of them will be used as the input of a lightweight MLP to calculate the color.

$$ \mathbf{C} = f \left ( \boldsymbol{\ell}^{(g)}, \boldsymbol{\ell}^{(a)} \right ) $$

Please refer to internal/renderers/gsplat_appearance_embedding_renderer.py for more details.

Baseline New Model

2.13. 3DGS-MCMC

pip install submodules/mcmc_relocation
... fit \
    --config configs/gsplat-mcmc.yaml \
    --model.density.cap_max MAX_NUM_GAUSSIANS \
    ...

MAX_NUM_GAUSSIANS is the maximum number of Gaussians that will be used.

Refer to ubc-vision/3dgs-mcmc, internal/density_controllers/mcmc_density_controller.py and internal/metrics/mcmc_metrics.py for more details.

2.14. Feature distillation

Click me This comes from Feature 3DGS. But two stage optimization is adapted here, rather than jointly. * First, train a model using gsplat (see command above) * Then extract feature map from your dataset Theoretically, any feature is distillable. You need to implement your own feature map extractor. Here are instructions about extracting SAM and LSeg features. * SAM ```bash python utils/get_sam_embeddings.py data/Truck/images ``` With this command, feature maps will be saved to `data/Truck/semantic/sam_features`, and preview to `data/Truck/semantic/sam_feature_preview`, respectively. * LSeg: please use ShijieZhou-UCLA/feature-3dgs and follow its instruction to extra LSeg features (do not use this repo's virtual environment for it). * Then start distillation * SAM ```bash python main.py fit \ --config configs/feature_3dgs/sam-speedup.yaml \ --data.path data/Truck \ --data.parser.down_sample_factor 2 \ --model.initialize_from outputs/Truck/gsplat \ -n Truck -v feature_3dgs-sam ``` * LSeg [NOTE] In order to distill LSeg's high-dimensional features, you may need a GPU equipped with a large memory capacity ```bash python main.py fit \ --config configs/feature_3dgs/lseg-speedup.yaml \ ... ``` `--model.initialize_from` is the path to your trained model. Since rasterizing high dimension features is slow, `--data.parser.down_sample_factor` is used here to smaller the rendered feature map to speedup distillation. * After distillation finishing, you can use viewer to visualize the feature map rendered from 3D Gaussians ```bash python viewer.py outputs/Truck/feature_3dgs ``` CLIP is required if you are using LSeg feature: `pip install git+https://github.com/openai/CLIP.git` LSeg feature is used in this video.

2.15. In the wild
image image image image

Introduction

Based on the Appearance Model (2.12.) above, this model can produce a visibility map for every training view indicating whether a pixel belongs to transient objects or not.

The idea of the visibility map is a bit like Ha-NeRF, but rather than uses positional encoding for pixel coordinates, 2D dense grid encoding is used here in order to accelerate training.

Please refer to Ha-NeRF, internal/renderers/gsplat_appearance_embedding_visibility_map_renderer.py and internal/metrics/visibility_map_metrics.py for more details.

[NOTE] Though it shows the capability to distinguish the pixels of transient objects, may not be able to remove some artifats/floaters belong to transients. And may also treat under-reconstructed regions as transients.

Usage

Download PhotoTourism dataset from here and split file from the "Additional links" here. The split file should be placed at the same path as the dense directory of the PhotoTourism dataset, e.g.:

├──brandenburg_gate
  ├── dense  # colmap database
      ├── images
          ├── ...
      ├── sparse
      ...
  ├── brandenburg.tsv  # split file

[Optional] 2x downsize the images: python utils/image_downsample.py data/brandenburg_gate/dense/images --factor 2

python main.py fit \
    --config configs/appearance_embedding_visibility_map_renderer/view_independent-2x_ds.yaml \
    --data.path data/brandenburg_gate \
    -n brandenburg_gate

If you have not downsized images, remember to add a --data.parser.down_sample_factor 1 to the command above.

python main.py validate \
   --save_val \
   --val_train \
   --config outputs/brandenburg_gate/lightning_logs/version_0/config.yaml  # you may need to change this path

Then you can find the rendered masks and images in outputs/brandenburg_gate/val.

2.16. New Multiple GPU training strategy

Introduction

This is a bit like a simplified version of Scaling Up 3DGS.

In the implementation here, Gaussians are stored, projected and their colors are calculated in a distributed manner, and each GPU rasterizes a whole image for a different camera. No Pixel-wise Distribution currently.

This strategy works with densification enabled.

[NOTE]

Metrics of MipNeRF360 dataset One batch per GPU, 30K iterations, no other hyperparameters changed. * PSNR ![image](https://github.com/user-attachments/assets/1a7fa6ad-89cf-4a63-9c09-7d74a9e30103) * SSIM ![image](https://github.com/user-attachments/assets/f4c91a7c-745f-480f-bc06-27692ab09494) * LPIPS ![image](https://github.com/user-attachments/assets/ff1f98c5-c70e-4897-be25-2a74223c421f)

Usage

python utils/merge_distributed_ckpts.py outputs/TRAINED_MODEL_DIR
python viewer.py outputs/TRAINED_MODEL_DIR/checkpoints/MERGED_CHECKPOINT_FILE

2.17. SpotLessSplats

[NOTE] No utilization-based pruning (4.2.3 of the paper) and appearance modeling (4.2.4 of the paper)

2.18. Depth Regularization with Depth Anything V2

This is implemented with reference to Hierarchical 3DGS.

Baseline DepthReg DepthReg + AppearanceModel

3. Evaluation

Per-image metrics will be saved to TRAINING_OUTPUT/metrics as a csv file.

Evaluate on validation set

python main.py validate \
    --config outputs/lego/config.yaml

On test set

python main.py test \
    --config outputs/lego/config.yaml

On train set

python main.py validate \
    --config outputs/lego/config.yaml \
    --val_train

Save images that rendered during evaluation/test

python main.py <validate or test> \
    --config outputs/lego/config.yaml \
    --save_val

Then you can find the images in outputs/lego/<val or test>.

4. Web Viewer

Transform Camera Path Edit

4.1 Basic usage

4.3 Load model trained by other implementations

[NOTE] The commands in this section only design for third-party outputs

python viewer.py \
    Deformable-3D-Gaussians/outputs/lego \
    --vanilla_deformable \
    --reorient disable  # change to enable when loading real world scene

5. F.A.Q.

Q: The viewer shows my scene in unexpected orientation, how to rotate the camera, like the U and O key in the SIBR_viewer?

A: Check the Orientation Control on the right panel, rotate the camera frustum in the scene to the orientation you want, then click Apply Up Direction.


Besides: You can also click the 'Reset up direction' button. Then the viewer will use your current orientation as the reference.

Q: The web viewer is slow (or low fps, far from real-time).

A: This is expected because of the overhead of the image transfer over network. You can get around 10fps in 1080P resolution, which is enough for you to view the reconstruction quality.