Official Implementation of DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling
This codebase is based on threestudio with minor changes to some files for parallel processing.
Python >= 3.11
.conda create -n dreampropeller python=3.11
pip install ninja
pip install -r requirements.txt
PyTorch == 2.1.0
. We have tested on torch2.1.0+cu121
.pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
huggingface-cli login
.We can compare the runtime for baseline and DreamPropeller by using launch_baseline.py
and launch_speedup.py
respectively. For DreamPropeller, one can tune speedup
parameters in .yaml
config files. The preset values in the files are the default values. We note their meaning below:
speedup.threshold
denotes the threshold for fixed-point error. Larger threshold gives greater speedup but lower generation quality.speedup.ema_decay
denotes the EMA rate of updating the threshold. This parameter makes it robust against variation in speedup for different prompts.speedup.P
denotes the window size, typically set to num_gpu-1
. e.g. 8 GPU cluster uses window size of 7.speedup.adaptivity_type
denotes the error aggregation function used to update speedup.ema_decay
. We only support median
and mean
and we use median
for all but ProlificDreamer. We find using median
makes quality more stable for tested frameworks but gives slower speedup than mean
.Other visualization options:
trainer.visualize_progress
sets whether to visualize the generation progress.trainer.display_time
sets whether to display current runtime for each displayed image.train_config.val_check_interval
denotes the wallclock time in seconds (instead of iteration number) after which the script will save a visualization.We can quickly test each of the frameworks presented in the paper below. Our method is tested on an 8-GPU A100-80G PCIe cluster.
You can freely switch between baseline and DreamPropeller by simply setting FILE
to launch_baseline.py
or launch_speedup.py
.
python $FILE --config configs/dreamfusion-if.yaml --train system.prompt_processor.prompt="an ice cream sundae" data.batch_size=16 trainer.visualize_progress=true
python $FILE --config configs/magic3d-coarse-if.yaml --train system.prompt_processor.prompt="a beautiful dress made out of fruit, on a mannequin. Studio lighting, high quality, high resolution" data.batch_size=16 trainer.visualize_progress=true
python $FILE --config configs/magic3d-refine-sd.yaml --train system.prompt_processor.prompt="a beautiful dress made out of fruit, on a mannequin. Studio lighting, high quality, high resolution" data.batch_size=16 trainer.visualize_progress=true system.geometry_convert_from=path/to/coarse/stage/trial/dir/ckpts/last.ckpt
python $FILE --config configs/textmesh-if.yaml --train system.prompt_processor.prompt="an old vintage car" data.batch_size=16 trainer.visualize_progress=true
python $FILE --config configs/prolificdreamer.yaml --train system.prompt_processor.prompt="a detailed Victorian era house" data.batch_size=[8,2] trainer.visualize_progress=true
python $FILE --config configs/prolificdreamer-geometry.yaml --train system.prompt_processor.prompt="a detailed Victorian era house" data.batch_size=8 trainer.visualize_progress=true system.geometry_convert_from=path/to/coarse/stage/trial/dir/ckpts/last.ckpt
python $FILE --config configs/prolificdreamer-texture.yaml --train system.prompt_processor.prompt="a detailed Victorian era house" data.batch_size=1 trainer.visualize_progress=true system.geometry_convert_from=path/to/geometry/stage/trial/dir/ckpts/last.ckpt
Since DreamGaussian is a separate standalone repo, we have copied it in the dreamgaussian/
subfolder with our dreampropeller implementation specifically tailored to this repo. Please find specific instructions to run DreamGaussian inside.
Our method can also be applied to Image-to-3D using score distillation. Luckily Zero 1-to-3 is implemented in the original threestudio code, so we can directly plug in our method.
Download pretrained Zero123XL weights into load/zero123
:
cd load/zero123
wget https://zero123.cs.columbia.edu/assets/zero123-xl.ckpt
To run,
python $FILE --config configs/zero123.yaml --train data.image_path=./load/images/grootplant_rgba.png data.random_camera.batch_size=[16,16,10] trainer.visualize_progress=true
@article{zhou2023dreampropeller,
title={DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling},
author={Zhou, Linqi and Shih, Andy and Meng, Chenlin and Ermon, Stefano},
journal={arXiv preprint arXiv:2311.17082},
year={2023}
}