DiffGS: Functional Gaussian Splatting Diffusion
(NeurIPS 2024)

Junsheng Zhou* · Weiqi Zhang* · Yu-Shen Liu

(* Equal Contribution)

Paper | Project Page

We release the code of the paper DiffGS: Functional Gaussian Splatting Diffusion in this repository.

Abstract

In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.

Method

Overview of DiffGS. (a) We disentangle the fitted 3DGS into three Gaussian Splatting Functions to model the Gaussian probability, colors and transforms. We then train a Gaussian VAE with a conditional latent diffusion model for generating these functions. (b) During generation, we first extract Gaussian geometry from the generated GauPF, followed by the GauCF and GauTF to obtain the Gaussian attributes.

Generation Results

Unconditional Generations

Visual Comparisons

Visualization Results

Text/Image-to-3D Generation

Gaussian Completion

Point-to-Gaussian Generation

Install

We recommend creating an anaconda environment using our provided environment.yml:

conda env create -f environment.yml
conda activate diffgs

Pretrained model

We first provide the pretrained models: Gaussian VAE and Gaussian LDM of the chair unconditional model. Please download the pretrained models from Google Drive.

Inference

To inference pretrained model of ShapeNet Chair, save the downloaded model checkpoint to config/stage1 and config/stage2. Additionally, you also need to adjust the checkpoint path in config/genetate/specs.json, then run the following command:

python test.py -e config/generate/

Training

The code and instructions for training and data preparation will be released soon.

Citation

If you find our code or paper useful, please consider citing

@inproceedings{diffgs,
    title={DiffGS: Functional Gaussian Splatting Diffusion},
    author={Zhou, Junsheng and Zhang, Weiqi and Liu, Yu-Shen},
    booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
    year={2024}
}