wyysf-98 / SweetDreamer

Apache License 2.0
232 stars 5 forks source link

SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D (ICLR 2024)

Weiyu Li, Rui Chen, Xuelin Chen, Ping Tan

Project Page | ArXiv | [Paper]() | [Video]()

All Code and Ckpt will be released in the next few days, sorry for the delay due some to some permission issues :( 🏗️ 🚧 🔨

Important: This repo. is under construction. Finally, I got time to sort it out :) really sorry for the dealy

Prerequisite

Setup environment (Install threestudio)

This part is the same as original threestudio. Skip it if you already have installed the environment.

See installation.md for additional information, including installation via Docker.

python3 -m virtualenv venv
. venv/bin/activate

# Newer pip versions, e.g. pip-23.x, can be much faster than old versions, e.g. pip-20.x.
# For instance, it caches the wheels of git packages to avoid unnecessarily rebuilding them later.
python3 -m pip install --upgrade pip
# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install ninja
pip install -r requirements.txt

Download the pretrained CCM model(TBD)

sh download.sh

Quick demo

python launch.py --config configs/sweetdreamer-stage1.yaml --train --gpu 0 \
                 system.prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 system.cmm_prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 tag=einstein

python launch.py --config configs/sweetdreamer-stage2.yaml --train --gpu 0 \
                 system.prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 system.cmm_prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 tag=einstein

Acknowledgement

This code is built on the amazing open-source projects:

We also thank Jianxiong Pan and Feipeng Tian for the help of the data and GPU server.

Citation

If you find our work useful for your research, please consider citing using the following BibTeX entry.

@article{sweetdreamer,
  author    = {Weiyu Li and Rui Chen and Xuelin Chen and Ping Tan},
  title     = {SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D},
  journal   = {arxiv:2310.02596},
  year      = {2023},
}