Joanna Waczyńska, Piotr Borycki, Sławomir Tadeja, Jacek Tabor, Przemysław Spurek
(* indicates equal contribution)
This repository contains the official authors implementation associated with the paper "GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting". Check also project page
Abstract: Recently, a range of neural network-based methods for image rendering have been introduced. One such widely-researched neural radiance field (NeRF) relies on a neural network to represent 3D scenes, allowing for realistic view synthesis from a small number of 2D images. However, most NeRF models are constrained by long training and inference times. In comparison, Gaussian Splatting (GS) is a novel, state-of-the-art technique for rendering points in a 3D scene by approximating their contribution to image pixels through Gaussian distributions, warranting fast training and swift, real-time rendering. A drawback of GS is the absence of a well-defined approach for its conditioning due to the necessity to condition several hundred thousand Gaussian components. To solve this, we introduce the Gaussian Mesh Splatting (GaMeS) model, which allows modification of Gaussian components in a similar way as meshes. We parameterize each Gaussian component by the vertices of the mesh face. Furthermore, our model needs mesh initialization on input or estimated mesh during training. We also define Gaussian splats solely based on their location on the mesh, allowing for automatic adjustments in position, scale, and rotation during animation. As a result, we obtain a real-time rendering of editable GS.
Check us if you want to make a flying hotdog:
@Article{waczynska2024games,
author = {Joanna Waczyńska and Piotr Borycki and Sławomir Tadeja and Jacek Tabor and Przemysław Spurek},
title = {GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting},
year = {2024},
eprint = {2402.01459},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
}
@Article{kerbl3Dgaussians,
author = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
title = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
journal = {ACM Transactions on Graphics},
number = {4},
volume = {42},
month = {July},
year = {2023},
url = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}
Multiple animation is possible like dancing ficus:
Since, the software is based on original Gaussian Splatting repository, for details regarding requirements, we kindly direct you to check 3DGS. Here we present the most important information.
# SSH
git clone git@github.com:waczjoan/gaussian-mesh-splatting.git --recursive
or
# HTTPS
git clone https://github.com/waczjoan/gaussian-mesh-splatting.git --recursive
To install the required Python packages we used 3.7 and 3.8 python and conda v. 24.1.0
conda env create --file environment.yml
conda gaussian_splatting_mesh
Common issues:
Our solution proposes several model types, which you set using the gs_type
flag.
You can run the following models in the repository:
**## Quick start
In this section we describe general information; please find below section Tutorial
for more details, or if you are here first time :)
Download dataset and put it in data
directory.
What scenario do you want check? To train a model in general use:
train.py --eval -s <path to data> -m <path to output> --gs_type <model_type> # use -w, if you want white background
if you don't have mesh (or you don't want use it):
train.py --eval -s /data/hotdog -m output/hotdog_flat --gs_type gs_flat -w
if you have mesh (in data/hotdog you should have mesh.obj
file):
train.py --eval -s /data/hotdog -m output/hotdog_gs_mesh --gs_type gs_mesh -w
train.py --eval -s /data/<id_face> -m output/<id_face> --gs_type gs_flame -w
To eval a model in general use:
python scripts/render.py -m <path to output> --gs_type <model_type> # Generate renderings
python metrics.py -m <path to output> --gs_type <model_type> # Compute error metrics on renderings
Tip: If you have trouble with imports running scrips/render.py
You should remember
export PYTHONPATH=/path/to/a/project
scripts/render.py -m output/hotdog_flat --gs_type gs_flat
or
scripts/render.py -m output/hotdog_flat --gs_type gs_points
then to calculate metrics:
python metrics.py -m <path to output> --gs_type <model_type> # Compute error metrics on renderings
scripts/render_points_time_animated.py -m output/hotdog_flat --skip_train
In this section we describe more details, and make step by step how to run GaMeS.
hotdog
dataset and put it in to data
directory. For example:<gaussian-mesh-splatting>
|---data
| |---<hotdog>
| |---<ship>
| |---...
|---train.py
|---metrics.py
|---...
hotdog.blend
. And save hotdog mesh: File -> Export -> Wavefront (.obj). File mesh.obj
has to be in the same dir as dataset:
<gaussian-mesh-splatting>
|---data
| |---<hotdog>
| | |---transforms_train.json
| | |---mesh.obj
|---train.py
|---metrics.py
|---...
rtx2070
it should take less than 15 minutes.
train.py --eval -s /data/hotdog -m output/hotdog_gs_mesh --gs_type gs_mesh
Tip: In default, 2 Gaussians per face in mesh is used, to change it use num_splats
. In fact, we highly recommend do it (in paper we used 5 or 10, check appendix), since it improves results, but training will take a bit longer, and in this tutorial, we would like it make it as easy it will be possible.
train.py --eval -s /data/hotdog -m output/hotdog_gs_mesh --gs_type gs_mesh --num_splats 5 -w
Tip2: If you would like to, you can manually subdivide bigger faces in blender app.
In output/hotdog_gs_mesh
you should find:
<gaussian-mesh-splatting>
|---data
| |---<hotdog>
| | |---transforms_train.json
| | |---mesh.obj
| | |---...
|---output
| |---<hotdog_gs_mesh>
| | |---point_cloud
| | |---xyz
| | |---cfg_args
| | |---...
|---train.py
|---metrics.py
|---...
During training you should get information:
Found transforms_train.json file, assuming Blender_Mesh data set!
Firstly let's check if our model correctly render files in init position. It should take less than 30 sec.
Tip: If you have trouble with imports running scrips/render.py
You should remember
export PYTHONPATH=/path/to/a/project
In this scenario let's run:
scripts/render.py -m output/hotdog_gs_mesh --gs_type gs_mesh
Use --skip_train
, if you would like to skip train dataset in render.
Then, let's calculate metrics (it takes around 3 minutes):
python metrics.py -m output/hotdog_gs_mesh --gs_type gs_mesh
In output/hotdog_gs_mesh
you should find:
<gaussian-mesh-splatting>
|---output
| |---<hotdog_gs_mesh>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs_mesh
| | |---results_gs_mesh.json
| | |---...
|---metrics.py
|---...
In fact since it is just init position, you can use gs
flag to render, and gets the same results.
Simply run:
scripts/render_time_animated.py -m output/hotdog_gs_mesh # --skip_train
Please find renders in time_animated
directory:
<gaussian-mesh-splatting>
|---output
| |---<hotdog_gs_mesh>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs_mesh
| | | |---time_animated
| | |---...
|---metrics.py
|---...
If you want more transformation, we recommend you check scripts/render_time_animated.py
file. Transformation transform_hotdog_fly
is default, but there is a few more. You can also create your own modification.
You can prepare your own more realistic transformation, for example an excavator lifting a shovel or spreading ficus branches, and save created mesh for example as ficus_animate.obj
.
Then you can use render_from_mesh_to_mesh.py
file.
hotdog
dataset and put it in to data
directory. For example:<gaussian-mesh-splatting>
|---data
| |---<hotdog>
| |---<ship>
| |---...
|---train.py
|---metrics.py
|---...
Here you don't need mesh.obj
. If you already download it, it is okey, it can be in directory, it will be just not used.
First step is train simple flat Gaussian Splatting, use gs_flat
flag. It should take around 10 minutes (using rtx2070).
train.py --eval -s /data/hotdog -m output/hotdog_gs_flat --gs_type gs_flat -w
In output/hotdog_flat
you should find:
<gaussian-mesh-splatting>
|---data
| |---<hotdog>
| | |---transforms_train.json
| | |---mesh.obj
| | |---...
|---output
| |---<hotdog_gs_flat>
| | |---point_cloud
| | |---xyz
| | |---cfg_args
| | |---...
|---train.py
|---metrics.py
|---...
During training you should get information:
Found transforms_train.json file, assuming Blender data set!
Firstly let's check you we can render Flat Gaussian Splatting:
scripts/render.py -m output/hotdog_gs_flat --gs_type gs_flat
Use --skip_train
, if you would like to skip train dataset in render.
Then, let's calculate metrics (it takes around 3 minutes):
python metrics.py -m output/hotdog_gs_flat --gs_type gs_flat
In output/hotdog_gs_flat
you should find:
<gaussian-mesh-splatting>
|---output
| |---<hotdog_gs_mesh>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs_flat
| | |---results_gs_flat.json
| | |---...
|---metrics.py
|---...
Since we would like to use parametrized Gaussians Splatting let's check renders after parametrization, use gs_points
flag:
scripts/render.py -m output/hotdog_gs_flat --gs_type gs_points #--skip_train
Then, let's calculate metrics (it takes around 3 minutes):
python metrics.py -m output/hotdog_gs_flat --gs_type gs_points
In output/hotdog_gs_flat
you should find:
<gaussian-mesh-splatting>
|---output
| |---<hotdog_gs_mesh>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs_flat
| | | |---renders_gs_points
| | |---results_gs_flat.json
| | |---results_gs_points.json
| | |---...
|---metrics.py
|---...
Please note, results_gs_flat
and results_gs_points
are differ slightly, this is due to numerical calculations.
Simply run:
scripts/render_points_time_animated.py -m output/hotdog_flat # --skip_train
Please find renders in time_animated
directory:
<gaussian-mesh-splatting>
|---output
| |---<hotdog_gs_mesh>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs_flat
| | | |---renders_gs_points
| | | |---time_animated_gs_points
| | |---...
|---metrics.py
|---...
face_f1036_A
dataset and put it in to data
directory. For example:<gaussian-mesh-splatting>
|---data
| |---<face_f1036_A>
| |---...
|---train.py
|---metrics.py
|---...
Here you don't need mesh.obj
. But... we use initial FLAME model. Hence:
Download FLAME model from official website. You need to sign up and agree to the model license for access to the model. Copy the downloaded models and put it in games\flame_splatting\FLAME\model
folder (for more details see games\flame_splatting\FLAME\config.py
file).
Train models with gs_flame
flag. It should take around 50 minutes (using rtx2070).
train.py --eval -s data/face_f1036_A -m output/face_f1036_A --gs_type gs_flame -w
In output/face_f1036_A
you should find:
<gaussian-mesh-splatting>
|---data
| |---<face_f1036_A>
| | |---transforms_train.json
| | |---...
|---output
| |---<face_f1036_A>
| | |---point_cloud
| | |---xyz
| | |---cfg_args
| | |---...
|---train.py
|---metrics.py
|---...
During training you should get information: "Found transforms_train.json file, assuming Flame Blender data set!"
Firstly let's check you we can render original Gaussian Splatting (since we save scaling, ration etc you can use gs
flag):
scripts/render.py -m output/face_f1036_A --gs_type gs
Use --skip_train
, if you would like to skip train dataset in render. You should see "assuming Blender data set" information.
Then, let's calculate metrics (it takes around 2 minutes):
python metrics.py -m output/face_f1036_A --gs_type gs
In output/face_f1036_A
you should find:
<gaussian-mesh-splatting>
|---output
| |---<face_f1036_A>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs
| | |---results_gs.json
| | |---...
|---metrics.py
|---...
Since we would like to use parametrized Flame Gaussians Splatting let's check renders after parametrization, use gs_flame
flag:
scripts/render_flame.py -m output/face_f1036_A #--skip_train
Please note, you will see "assuming Flame Blender data set" information.
In output/face_f1036_A
you should find:
<gaussian-mesh-splatting>
|---output
| |---<face_f1036_A>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---renders_gs_flame
| | |---...
|---metrics.py
|---...
Renders renders_gs_flame
and renders_gs
should correspond to each other, that is, give the same results (except numerical differences).
render_set_animated
function in scripts\redner_flame.py
-- you should manage how to animate! :))For render use animated
flag:
scripts/render_flame.py -m output/face_f1036_A --animated #--skip_train
In output/face_f1036_A
you should find:
<gaussian-mesh-splatting>
|---output
| |---<face_f1036_A>
| | |---point_cloud
| | |---cfg_args
| | |---test
| | |---<ours_iter>
| | | |---flame_animated
| | |---...
|---metrics.py
|---...
You will need to install a few dependencies before running the project setup.
# Dependencies
sudo apt install -y libglew-dev libassimp-dev libboost-all-dev libgtk-3-dev libopencv-dev libglfw3-dev libavdevice-dev libavcodec-dev libeigen3-dev libxxf86vm-dev libembree-dev
# Project setup
cd SIBR_viewers
cmake -Bbuild . -DCMAKE_BUILD_TYPE=Release # add -G Ninja to build faster
cmake --build build -j24 --target install
Common problem: