😴 LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes 😴

https://github.com/luciddreamer-cvlab/LucidDreamer/assets/12259041/35004aaa-dffc-4133-b15a-05224e68b91e

LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

*Jaeyoung Chung, *Suyoung Lee, Hyeongjin Nam, Jaerin Lee, Kyoung Mu Lee

*Denotes equal contribution.

🤖 Install

Ubuntu

Prerequisite

• CUDA>=11.4 (higher version is OK).
• Python==3.9 (cannot use 3.10 due to open3d compatibility)

Installation script

conda create -n lucid python=3.9
conda activate lucid
pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d torch==2.0.1  torchvision==0.15.2 gradio omegaconf
# ZoeDepth
pip install timm==0.6.7
# Gaussian splatting
pip install plyfile==0.8.1

cd submodules/depth-diff-gaussian-rasterization-min
# sudo apt-get install libglm-dev # may be required for the compilation.
python setup.py install
cd ../simple-knn
python setup.py install
cd ../..

Windows (Experimental, Tested on Windows 11 with VS2022)

Checklist

• Make sure that the versions of your installed CUDA, cudatoolkit, and pytorch match. We have tested on CUDA==11.8.
• Make sure you download and install C++ (>=14) from the Visual Studio build tools.

Installation script

conda create -n lucid python=3.9
conda activate lucid
conda install pytorch=2.0.1 torchvision=0.15.2 torchaudio=2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d gradio omegaconf
# ZoeDepth
pip install timm==0.6.7
# Gaussian splatting
pip install plyfile==0.8.1

# There is an issue with whl file so please manually install the module now.
cd submodules\depth-diff-gaussian-rasterization-min\third_party
git clone https://github.com/g-truc/glm.git
cd ..\
python setup.py install
cd ..\simple-knn
python setup.py install
cd ..\..

⚡ Usage

We offer several ways to interact with LucidDreamer:

1. A demo is available on ironjr/LucidDreamer HuggingFace Space (including custom SD ckpt) and ironjr/LucidDreamer-mini HuggingFace Space (minimal features / try at here in case of the former is down) (We appreciate all the HF / Gradio team for their support).

https://github.com/luciddreamer-cvlab/LucidDreamer/assets/12259041/745bfc46-8215-4db2-80d5-4825e91316bc

2. Another demo is available on a Colab, implemented by @camenduru (We greatly thank @camenduru for the contribution).
3. You can use the gradio demo locally by running CUDA_VISIBLE_DEVICES=0 python app.py (full feature including huggingface model download, requires ~15GB) or CUDA_VISIBLE_DEVICES=0 python app_mini.py (minimum viable demo, uses only SD1.5).
4. You can also run this with command line interface as described below.

Run with your own samples

# Default Example
python run.py --image <path_to_image> --text <path_to_text_file> [Other options]

• Replace and with the paths to your image and text files.

Other options

• --image (-img): Specify the path to the input image for scene generation.
• --text (-t): Path to the text file containing the prompt that guides the scene generation.
• --neg_text (-nt): Optional. A negative text prompt to refine and constrain the scene generation.
• --campath_gen (-cg): Choose a camera path for scene generation (options: lookdown, lookaround, rotate360).
• --campath_render (-cr): Select a camera path for video rendering (options: back_and_forth, llff, headbanging).
• --model_name: Optional. Name of the inpainting model used for dreaming. Leave blank for default(SD 1.5).
• --seed: Set a seed value for reproducibility in the inpainting process.
• --diff_steps: Number of steps to perform in the inpainting process.
• --save_dir (-s): Directory to save the generated scenes and videos. Specify to organize outputs.

Guideline for the prompting / Troubleshoot

General guides

1. If your image is indoors with specific scene (and possible character in it), you can just put the most simplest representation of the scene first, like a cozy livingroom for christmas, or a dark garage, etc. Please avoid prompts like 1girl because it will generate many humans for each inpainting task.
2. If you want to start from already hard-engineered image from e.g., StableDiffusion model, or a photo taken from other sources, you can try using WD14 tagger (huggingface demo) to extract the danbooru tags from an image. Please ensure you remove some comma separated tags if you don't want them to appear multiple times. This include human-related objects, e.g., 1girl, white shirt, boots, smiling face, red eyes, etc. Make sure to specify the objects you want to have multiples of them.

Q. I generate unwanted objects everywhere, e.g., photo frames.

1. Manipulate negative prompts to set harder constraints for the frame object. You may try adding tags like twitter thumbnail, profile image, instagram image, watermark, text to the negative prompt. In fact, negative prompts are the best thing to try if you want some things not to be appeared in the resulting image.
2. Try using other custom checkpoint models, which employs different pipeline methods: LaMa inpainting -> ControlNet-inpaint guided image inpainting.

Visualize .ply files

There are multiple available viewers / editors for Gaussian splatting .ply files.

1. @playcanvas's Super-Splat project (Live demo). This is the viewer we have used for our debugging along with MeshLab.

2. @antimatter15's WebGL viewer for Gaussian splatting (Live demo).

3. @splinetool's web-based viewer for Gaussian splatting. This is the version we have used in our project page's demo.

🚩 Updates

• ✅ December 12, 2023: We have precompiled wheels for the CUDA-based submodules and put them in submodules/wheels. The Windows installation guide is revised accordingly!
• ✅ December 11, 2023: We have updated installation guides for Windows. Thank you @Maoku for your great contribution!
• ✅ December 8, 2023: HuggingFace Space demo is out. We deeply thank all the HF team for their support!
• ✅ December 7, 2023: Colab implementation is now available thanks to @camenduru!
• ✅ December 6, 2023: Code release!
• ✅ November 22, 2023: We have released our paper, LucidDreamer on arXiv.

🌏 Citation

Please cite us if you find our project useful!

@article{chung2023luciddreamer,
    title={LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes},
    author={Chung, Jaeyoung and Lee, Suyoung and Nam, Hyeongjin and Lee, Jaerin and Lee, Kyoung Mu},
    journal={arXiv preprint arXiv:2311.13384},
    year={2023}
}

🤗 Acknowledgement

We deeply appreciate ZoeDepth, [Stability AI](), and Runway for their models.

📧 Contact

If you have any questions, please email robot0321@snu.ac.kr, esw0116@snu.ac.kr, jarin.lee@gmail.com.

⭐ Star History