ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars

NEWS: our paper has been accepted to SIGGRAPH 2024!

Abstract

Real-world applications often require a large gallery of 3D assets that share a consistent theme. While remarkable advances have been made in general 3D content creation from text or image, synthesizing customized 3D assets following the shared theme of input 3D exemplars remains an open and challenging problem. In this work, we present ThemeStation, a novel approach for theme-aware 3D-to-3D generation. ThemeStation synthesizes customized 3D assets based on given few exemplars with two goals: 1) unity for generating 3D assets that thematically align with the given exemplars and 2) diversity for generating 3D assets with a high degree of variations. To this end, we design a two-stage framework that draws a concept image first, followed by a reference-informed 3D modeling stage. We propose a novel dual score distillation (DSD) loss to jointly leverage priors from both the input exemplars and the synthesized concept image. Extensive experiments and user studies confirm that ThemeStation surpasses prior works in producing diverse theme-aware 3D models with impressive quality. ThemeStation also enables various applications such as controllable 3D-to-3D generation.

Overview

Todo (Latest update: 2024/04/26)

[x] Release rendering code
[x] Release StageI code for concept image generation
[x] Release StageII code for reference-informed 3D assets modeling
[ ] Update our code to support more powerful image-to-3d models, such as InstantMesh/CRM to obtain the initial 3D

Installation

Setup:

conda create -n themestation python==3.10
conda activate themestation
pip install -r requirements.txt
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

We use Wonder3D to obtain the initial 3D model.
- Download the checkpoints into folder Wonder3D/ckpts.
- Make sure you have the following models.
```
Wonder3D
|-- ckpts
|-- unet
|-- scheduler.bin
...
```
(Optional, Recommended) Install ninja to speed up the compilation of CUDA extensions:
```
pip install ninja
```
(Optional) tiny-cuda-nn installation might require downgrading pip to 23.0.1
Stable Diffusion. We use diffusion prior from a pretrained 2D Stable Diffusion 2.1 model. To start with, you may need a huggingface token to access the model, or use huggingface-cli login command.

Preparation

Place reference 3D models under data/reference_models/models and install Blender3.2.2

Render images for generating concept images:

cd data/reference_models
sh render_script/render_single_concept.sh obj_file_name blender_path elevation
# For example:
# sh render_script/render_single_concept.sh 20 owl.glb ~/blender-3.2.2-linux-x64/blender

Render images for learning reference prior:

# RGB
cd data/reference_models
sh render_script/render_single_ref_rgb.sh obj_file_name blender_path

# Normal (optional)
cd data/reference_models
sh render_script/render_single_ref_normal.sh obj_file_name blender_path

# For example:
# sh render_script/render_single_ref_rgb.sh owl.glb ~/blender-3.2.2-linux-x64/blender

Make sure you have the following files.

ThemeStation
| -- data
| -- reference_models
    | -- renderings
        | -- elevation_n (used to generate concept images with elevation = n)
            | -- object_name
                | -- 000.png
                ...
        | -- elevation_rand (used to learn reference prior)
            | -- object_name
                | -- 000.png
                ...
                | -- normals_000_0001.png (optional)
                ...

Inference

Stage I: Concept image generation

For now, you can simply use Freepik-reimagine for concept image generation given the rendered front view of a given 3D exemplar.

We also show the steps as introduced in our paper below:

to generate concept images, run:

cd dreambooth
sh scripts/generate_concept.sh 20 # 20 is elevation, edit this template for you own data

# for controllable-3D-to-3D using text run:
sh scripts/generate_concept_control.sh 20
# results are saved to `dreambooth/outputs/...`

select the concept images you like and remove their background using Clipdrop or Photoshop and put the final concept images (with transparant background) under data/concept_images

Stage II: Reference-informed 3D asset modeling

Run ThemeStation to generate a final 3D model given the concept image and reference model.

run a specific concept image

sh scrips/run.sh # edit this template script for your own data

run a batch of concept images

sh scrips/run_batch.sh # edit this template script for your own data

export mesh

sh scripts/export_mesh.sh # edit this template script for your own data

Citation

If you find this code helpful for your research, please cite:

@article{wang2024themestation,
        title={ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars}, 
        author={Zhenwei Wang and Tengfei Wang and Gerhard Hancke and Ziwei Liu and Rynson W.H. Lau},
        booktitle={ACM SIGGRAPH},
        year={2024}
  }

Acknowledgments

We have intensively borrowed codes from the following repositories. Many thanks to the authors for sharing their codes.

3DTopia / ThemeStation

readme