Semantify offers a method to create and edit a 3D parametric model using semantically meaningful descriptors. Semantify is based on a self-supervised method that utilizes the semantic power of CLIP language-vision model to build a mapping between semantic descriptors to 3DMM model coefficients. This can be used in an interactive application defining a slider for each descriptor (a), or to fit a model to an image in a zero shot manner by feeding the image into CLIP and obtaining a vector of semantic scores that can be mapped to shape parameters (b).
git clone https://github.com/Omergral/Semantify.git
cd Semantify
conda env create -f requirements.yml
conda activate semantify
pip install -e .
bash ./get_mappers.sh
models_ckpts
├── flame
│ ├── expression
│ │ ├── flame_expression.ckpt
│ │ └── flame_expression_metadata.json
│ └── shape
│ ├── flame_shape.ckpt
│ └── flame_shape_metadata.json
├── smal
│ ├── smal.ckpt
│ └── smal_metadata.json
├── smpl
│ ├── smpl_female.ckpt
│ ├── smpl_female_metadata.json
│ ├── smpl_male.ckpt
│ ├── smpl_male_metadata.json
│ ├── smpl_neutral.ckpt
│ └── smpl_neutral_metadata.json
└── smplx
├── smplx_female.ckpt
├── smplx_female_metadata.json
├── smplx_male.ckpt
├── smplx_male_metadata.json
├── smplx_neutral.ckpt
└── smplx_neutral_metadata.json
FLAME [Li et al. 2017]
semantify/assets/flame
as follows:
├── flame
├── female_model.pkl
├── flame.py
├── flame_static_embedding_68.pkl
├── ft.npy
├── generic_model.pkl
├── male_model.pkl
├── mean.npy
└── vt.npy
SMPL [Loper et al. 2015]
semantify/assets/smpl
as follows:
├── smpl
├── SMPL_FEMALE.pkl
├── SMPL_MALE.pkl
├── SMPL_NEUTRAL.pkl
├── smpl_texture.png
└── smpl_uv.obj
SMPL-X [Pavlakos et al. 2019]
semantify/assets/smplx
as follows:
├── smplx
├── a_pose.npy
├── SMPLX_FEMALE.npz
├── SMPLX_MALE.npz
├── SMPLX_NEUTRAL.npz
└── smplx_texture.png
SMAL [Zuffi et al. 2017]
smal_CVPR2017.pkl
file and place it in semantify/assets/smal
├── smal
├── smal_CVPR2017.pkl
└── smal_layer.py
cd SPIN
./fetch_data.sh
SPIN/data/smpl_mean_params.npz
and SPIN/data/model_checkpoint.pt
to semantify/assets/spin
└── spin
├── model_checkpoint.pt
├── smpl_mean_params.npz
└── spin_model.py
python semantify/applications/sliders_demo_py3d.py --model_type <MODELTYPE> --specific <SPECIFIC>
Arguments Description:
model_type (str)
- "smplx" | "smpl" | "flame" | "smal"specific (str)
- "male" | "female" | "neutral" | "expression" | "shape"image_path (Optional[str])
- If you want to run the demo on an image, provide the path to the imagemapper_path (Optional[str])
- If you don't want to use Semantify's mappers, set the path to desired ckpt fileuse_raw_blendshapes (Optional)
- Use the raw parametric blendshapes of the modelout_dir (Optional[str])
- Path of directory to save outputs insemantify/applications/sliders_demo_py3d.py
python semantify/applications/image2shape.py --images_paths <IMAGES_PATHS> --model_type <TYPE> --specific <SPECIFIC> --output_path <PATH>
Arguments Description
images_paths (List[str])
- Paths to the images you wish to run the demo onmodel_type (str)
- "smplx" | "smpl" | "flame" | "smal"specific (str)
- "male" | "female" | "neutral" | "expression" | "shape"mapper_path (Optional[str])
- If you don't want to use Semantify's mappers, set the path to desired ckpt fileCreate Data
python semantify/data_management/creation/create_data.py --output_path <PATH> --model_type <TYPE> --specific <SPECIFIC> --use_tex --multiview --num_of_imgs <NUM>
This script will create as many images as you like, by randomly sample the parametric space of the given 3DMM. The output for a single sample will be a .png
file and .json
file that contains the sampled shape coefficients.
Arguments Description:
output_path (str)
- Path to a folder to store the datause_tex
- Use models textures (if any) - HIGHLY RECOMMENDEDmultiview
- Render the model from both frontal and side views instead of frontal onlynum_of_imgs (int)
- How many images to createGenerate CLIP's Ratings
python semantify/data_management/creation/clip_encoder.py --imgs_dir <PATH> --model_type <TYPE> --specific <SPECIFIC> --multiview
This script will run over the directory of images provided as input, along with a set of word descriptors and generate CLIP's ratings for each image against all descriptors. The output for a single sample will be a .json
file that contains the ratings for each descriptor.
Arguments Description:
imgs_dir (str)
- Path to the directory of created images from phase (1).descriptors (List[str])
- List of descriptors to use. We supply a default set, so this field is optional.First in semantify/config/train_mapper.yaml
fill the following fields:
output_path
- Where to store the mapperrun_name
- Name the rundataset.datadir
- Path to the data directorydataset.optimize_feature
- betas (SMPLX | SMPL) / beta (SMPL) / expression_params (FLAME) / shape_params (FLAME)dataset.labels_to_get
- On which descriptors to optimize.yaml
file you will find more configuration options.Then, to train the mapper run:
python semantify/train/train_mapper.py
If you make use of our work, please cite our paper:
@InProceedings{Gralnik_2023_ICCV,
author = {Gralnik, Omer and Gafni, Guy and Shamir, Ariel},
title = {Semantify: Simplifying the Control of 3D Morphable Models Using CLIP},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
pages = {14554-14564}
}