[CVPR 2024] GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting

Yiwen Chen^*1,2, Zilong Chen^*3, Chi Zhang², Feng Wang³, Xiaofeng Yang²,
Yikai Wang³, Zhongang Cai⁴ Lei Yang⁴ Huaping Liu³ Guosheng Lin^**1,2
*Equal contribution. ^**Corresponding author.
¹S-Lab, Nanyang Technological University,
²School of Computer Science and Engineering, Nanyang Technological University,
³Department of Computer Science and Technology, Tsinghua University, ⁴SenseTime Research,

## Demo Videos

Swift and controllable 3D editing with only 2-7 minutes.

https://github.com/buaacyw/GaussianEditor/assets/52091468/10740174-3208-4408-b519-23f58604339e https://github.com/buaacyw/GaussianEditor/assets/52091468/44797174-0242-4c82-a383-2d7b3d4fd693 https://github.com/buaacyw/GaussianEditor/assets/52091468/18dd3ef2-4066-428a-918d-c4fe673d0af8

## Release - [12/5] Docker support. Great thanks to [jhuangBU](https://github.com/jhuangBU). For windows, you can try [this guide](https://github.com/buaacyw/GaussianEditor/issues/9) and [this guide](https://github.com/buaacyw/GaussianEditor/issues/14). - [11/29] Release segmentation confidence score scaler. You can now scale the threshold of semantic tracing masks. - [11/27] 🔥 We released **GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting** and beta version of GaussianEditing WebUI. ## Contents - [Demo Videos](#demo-videos) - [Release](#release) - [Contents](#contents) - [Installation](#installation) - [WebUI Guide](#webui-guide) - [How to achieve better result](#how-to-achieve-better-result) - [Command Line](#command-line) - [TODO](#todo) - [FAQ](#faq) ## Installation Our environment has been tested on Ubuntu 22, CUDA 11.8 with 3090, A5000 and A6000. 1. Clone our repo and create conda environment ``` git clone https://github.com/buaacyw/GaussianEditor.git && cd GaussianEditor # (Option one) Install by conda conda env create -f environment.yaml # (Option two) You can also install by pip # CUDA version 11.7 pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117 # CUDA version 11.8 pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118 pip install -r requirements.txt # (Option three) If the below two options fail, please try this: # For CUDA 11.8 bash install.sh ``` 2. (Optional) Install our forked viser [Required by WebUI) ``` mkdir extern && cd extern git clone https://github.com/heheyas/viser pip install -e viser cd .. ``` 3. (Optional) Download Wonder3D checkpoints [Required by Add] ```bash sh download_wonder3d.sh ``` ## WebUI Guide Please be aware that our WebUI is currently in a beta version. Powered by [Viser](https://github.com/nerfstudio-project/viser/tree/main), you can use our WebUI even if you are limited to remote server. For details, please follow [WebUI Guide](https://github.com/buaacyw/GaussianEditor/blob/master/docs/webui.md). ## How to achieve better result The demand for 3D editing is very diverse. For instance, if you only want to change textures and materials or significantly modify geometry, it's clear that a one-size-fits-all hyperparameter won't work. Therefore, we cannot provide a default hyperparameter setting that works effectively in all scenarios. Therefore, if your results do not meet expectations, please refer to our [hyperparameter tuning](https://github.com/buaacyw/GaussianEditor/blob/master/docs/hyperparameter.md) document. In it, we detail the function of each hyperparameter and advise on which parameters to adjust when you encounter specific issues. ## Command Line We also provide a command line version of GaussianEditor. Like WebUI, you need to specify your path to the pretrained Gaussians and COLMAP outputs as mentioned in [here](https://github.com/buaacyw/GaussianEditor/blob/1fa96851c132258e0547ba73372f37cff83c92c3/docs/webui.md?plain=1#L20). Please check scripts in `sciprt` folder. Simply change `data.source` to your COLMAP output directory and `system.gs_source` to your pretrained Gaussians and run our demo scripts. ## TODO The repo is still being under construction, thanks for your patience. - [x] Tutorial for hyperparameter tuning. - [x] Step-by-step tutorial for WebUI . - [x] Realised WebUI beta version and GaussianEditor. ## FAQ - Bad result for Edit. We are using [InstructPix2Pix](https://github.com/timothybrooks/instruct-pix2pix) to generate edited 2D images as editing guidance. Unfortunately, InstructPix2Pix only works on limited prompts, please first try [here](https://huggingface.co/spaces/timbrooks/instruct-pix2pix) if you are not sure whether your text prompts work. - Bad result for Add. We use [ControlNet-Inpainting](https://github.com/lllyasviel/ControlNet) to first generate 2D inpainting and then transfer it into 3D. Also it doesn't work for bad prompts. Please try to enlarge your inpainting mask and try more seeds. - Bad result for Segmentation. Try scale the segmentation threshold, which changes the confidence score for segmentation. - Missing weights for DPT. Please read this [issue](https://github.com/buaacyw/GaussianEditor/issues/10) ## Acknowledgement Our code is based on these wonderful repos: * [Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting) * [Wonder3D](https://github.com/xxlong0/Wonder3D) * [Threestudio](https://github.com/threestudio-project/threestudio) * [Viser](https://github.com/nerfstudio-project/viser) * [InstructNerf2Nerf](https://github.com/ayaanzhaque/instruct-nerf2nerf) * [InstructPix2Pix](https://github.com/timothybrooks/instruct-pix2pix) * [Controlnet](https://github.com/lllyasviel/ControlNet)