Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing (ECCV2024)

🌐Project Page | 🖨️ArXiv | 📰Paper

Jian Gao^1*, Chun Gu^2*, Youtian Lin¹, Hao Zhu¹, Xun Cao¹, Li Zhang², Yao Yao¹
¹Nanjing University ²Fudan University
*denotes Equally contributed.

This is official implement of Relightable 3D Gaussian for the paper Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing. Alt text

Installation

Clone this repo

git clone https://github.com/NJU-3DV/Relightable3DGaussian.git

Install dependencies

# install environment
conda env create --file environment.yml
conda activate r3dg

# install pytorch=1.12.1
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge

# install torch_scatter==2.1.1
pip install torch_scatter==2.1.1

# install kornia==0.6.12
pip install kornia==0.6.12

# install nvdiffrast=0.3.1
git clone https://github.com/NVlabs/nvdiffrast
pip install ./nvdiffrast

Install the pytorch extensions

We recommend that users compile the extension with CUDA 11.8 to avoid the potential problems mentioned in 3D Guassian Splatting.

# install knn-cuda
pip install ./submodules/simple-knn

# install bvh
pip install ./bvh

# install relightable 3D Gaussian
pip install ./r3dg-rasterization

Data preparation

NeRF Synthetic Dataset

Download the NeRF synthetic dataset from LINK provided by NeRF.

Pre-processed DTU

For real-world DTU data, we follow the Vis-MVSNet to get the depth maps and then filter the depth map through photometric and geometric check. We then convert the depth map to normal through kornia. And we get perfect masks from IDR. The pre-processed DTU data can be downloaded here.

Pre-processed Tanks and Temples

For real-world Tanks and Temples data, we also use Vis-MVSNet and kornia to get the filtered MVS depth maps and normal maps. The masks are from NSVF. The pre-processed Tanks and Temples data can be downloaded here.

Synthetic4Relight

Download the NeRF synthetic dataset from LINK provided by InvRender.

Data Structure

We organize the datasets like this:

Relightable3DGaussian
├── datasets
    ├── nerf_synthetic
    |   ├── chair
    |   ├── ...
    ├── neilfpp
    |   ├── data_dtu
    │   │   │── DTU_scan24
    |   |   |   |── inputs
    |   |   |   |   |── depths
    |   |   |   |   |── images
    |   |   |   |   |── model
    |   |   |   |   |── normals
    |   |   |   |   |── pmasks
    |   |   |   |   |── sfm_scene.json
    │   │   │── ...
    |   ├── data_tnt
    │   │   │── Barn
    |   |   |   |── inputs
    |   |   |   |   |── depths
    |   |   |   |   |── images
    |   |   |   |   |── model
    |   |   |   |   |── normals
    |   |   |   |   |── pmasks
    |   |   |   |   |── sfm_scene.json
    │   │   │── ...
    ├── Synthetic4Relight
    |   ├── air_baloons
    |   ├── ...

Ground Points for composition

For multi-object composition, we manually generate a ground plane with relightable 3D Gaussian representation, which can be downloaded here. We put the ground.ply in the folder ./point.

Running

We run the code in a single NVIDIA GeForce RTX 3090 GPU (24G). To reproduce the results in the paper, please run the following code. NeRF Synthetic dataset:

sh script/run_nerf.sh

DTU data:

sh script/run_dtu.sh

Tanks and Temples data:

sh script/run_tnt.sh

Synthetic4Relight data:

sh script/run_syn.sh

Evaluating

Run the following command to evaluate Novel View Synthesis:

# e.g. DTU dataset
# stage 1
python eval_nvs.py --eval \
    -m output/dtu/${i}/3dgs \
    -c output/dtu/${i}/3dgs/chkpnt30000.pth

# stage 2
python eval_nvs.py --eval \
    -m output/dtu/${i}/neilf \
    -c output/dtu/${i}/neilf/chkpnt50000.pth \
    -t neilf

Run the following command to evaluate Relighting (for Synthetic4Relight only):

# e.g.
python eval_relighting_syn4.py \
    -m output/Syn4Relight/hotdog/neilf \
    -c output/Syn4Relight/hotdog/neilf/chkpnt50000.pth \
    --sample_num 384

Composition and Relighting

Explicit point cloud representation facilitates composition. We recommend that users explore cloud compare to implement point cloud transformations such as scaling, translation, and rotation to composite a new scene. For multi-object composition and relighting result demonstrated in the paper, you could run:

python relighting.py \
-co configs/teaser \
-e "env_map/teaser.hdr" \
--output "output/relighting/teaser_trace" \
--sample 384

For multi-object composition and relighting video illustrated in the project page, you could run:

python relighting.py \
-e env_map/composition.hdr \
-co configs/nerf_syn \
--output "output/relighting/nerf_syn" \
--sample_num 384 \
--video 

python relighting.py \
-co configs/nerf_syn_light \
-e "env_map/composition.hdr" \
--output "output/relighting/nerf_syn_light" \
--sample_num 384 \
--video

For multi-scene composition video, you could run:

python relighting.py \
-co configs/tnt \
-e "env_map/ocean_from_horn.jpg" \
--output "output/relighting/tnt" \
--sample_num 384 \
--video

GUI

We also provide a GUI for visualization. You can utilize this GUI to supervise the training process by just adding the --gui option to the command for training. You can also visualize the model after training by running:

# for 3D Gaussian
python gui.py -m output/NeRF_Syn/lego/3dgs -t render 

# for relightable 3D Gaussian
python gui.py -m output/NeRF_Syn/lego/neilf -t neilf

Trying on your own data

We recommend that users reorganize their own data as neilfpp-like dataset and then optimize. Modified VisMVSNet and auxiliary scripts to prepare your own data will come soon.

Citation

If you find our work useful in your research, please be so kind to cite:

@article{R3DG2023,
    author    = {Gao, Jian and Gu, Chun and Lin, Youtian and Zhu, Hao and Cao, Xun and Zhang, Li and Yao, Yao},
    title     = {Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing},
    journal   = {arXiv:2311.16043},
    year      = {2023},
}

NJU-3DV / Relightable3DGaussian

readme