Code for the paper ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation (ICLR 2023 spotlight) and DreamStone: Image as Stepping Stone for Text-Guided 3D Shape Generation (TPAMI).
Authors: Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, Chi-Wing Fu
conda env create -f environment.yaml
conda activate iss
python setup.py build_ext --inplace
wget https://s3.eu-central-1.amazonaws.com/avg-projects/differentiable_volumetric_rendering/data/ShapeNet.zip
Put the above two files to the same folder as "ISS-Image-as-Stepping-Stone-for-Text-Guided-3D-Shape-Generation"
(1) Pretrained Model
We provide pretrained models of stage 1 here.
(2) Training
cd stage1
python train.py configs/single_view_reconstruction/multi_view_supervision/ours_combined.yaml
Put the pretrained model of stage 1 to "stage2" folder.
(1) Training
cd ../stage2
python train.py configs/single_view_reconstruction/multi_view_supervision/ours_combined.yaml --text 'a red car'
(2) Inference
python generate.py configs/demo/demo_combined.yaml --text 'a red car' --it 20
"it" means the iteration of saved model.
Mesh and point cloud are saved in 'out/a red car/'
(1) First generate "a chair" in "stage2".
(2) Training
cd ../stage3_texture
python train.py configs/single_view_reconstruction/multi_view_supervision/ours_combined.yaml --text 'wooden' --model '../stage2/out/a chair/model20.pt'
The rendered images are saved in "./tmp" folder.
(3) Inference
python generate.py configs/demo/demo_combined.yaml --model 'out/single_view_reconstruction/multi_view_supervision/ours_combined/model.pt'
(1) First generate "a wooden boat" in "stage2". We may need more iterations to generate this shape, like "python train.py configs/single_view_reconstruction/multi_view_supervision/ours_combined.yaml --text 'a wooden boat' --iteration 30"
(2) Training
cd ../stage3_shape_texture
python train.py configs/single_view_reconstruction/multi_view_supervision/ours_combined.yaml --text 'tulip' --model '../stage2/out/a wooden boat/model20.pt'
The rendered images are saved in "./tmp" folder.
(3) Inference
python generate.py configs/demo/demo_combined.yaml --model 'out/single_view_reconstruction/multi_view_supervision/ours_combined/model.pt'
See ``get3d_release'' folder.
See ``DreamStone'' folder.
The code is built upon DVR and [Stable-Dreamfusion] (https://github.com/ashawkey/stable-dreamfusion)
If you have any questions or suggestions about this repo, please feel free to contact me (liuzhengzhelzz@gmail.com).