ADGAN

PyTorch | project page | paper

PyTorch implementation for controllable person image synthesis.

ADGAN: Controllable Person Image Synthesis with Attribute-Decomposed GAN,
Yifang Men, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: Proc. Computer Vision and Pattern Recognition (CVPR Oral), 2020 arXiv preprint (arXiv 2003.12267)

ADGAN++: Controllable Image Synthesis with Attribute-Decomposed GAN,
Guo Pu, Yifang Men, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. arXiv preprint (comming soon) Code

Updates

-(03/22/2022)ADGAN++, an extension vision with improved methods and more applications will be released soon.

Demo

Component Attribute Transfer

Pose Transfer

Requirement

python 3
pytorch(>=1.0)
torchvision
numpy
scipy
scikit-image
pillow
pandas
tqdm
dominate

Getting Started

You can directly download our generated images (in Deepfashion) from Google Drive.

Installation

Clone this repo:

git clone https://github.com/menyifang/ADGAN.git
cd ADGAN

Data Preperation

We use DeepFashion dataset and provide our dataset split files, extracted keypoints files and extracted segmentation files for convience.

The dataset structure is recommended as:

+—deepfashion
|   +—fashion_resize
|       +--train (files in 'train.lst')
|          +-- e.g. fashionMENDenimid0000008001_1front.jpg
|       +--test (files in 'test.lst')
|          +-- e.g. fashionMENDenimid0000056501_1front.jpg
|       +--trainK(keypoints of person images)
|          +-- e.g. fashionMENDenimid0000008001_1front.jpg.npy
|       +--testK
|          +-- e.g. fashionMENDenimid0000056501_1front.jpg.npy
|   +—semantic_merge
|   +—fashion-resize-pairs-train.csv
|   +—fashion-resize-pairs-test.csv
|   +—fashion-resize-annotation-pairs-train.csv
|   +—fashion-resize-annotation-pairs-test.csv
|   +—train.lst
|   +—test.lst
|   +—vgg19-dcbb9e9d.pth
|   +—vgg_conv.pth
...

Person images

Download person images from deep fasion dataset in-shop clothes retrival benchmark and download dataset split from Google Drive.
Crop the images. Split the raw images into the train split (fashion_resize/train) and the test split (fashion_resize/test). Launch
```
python tool/generate_fashion_datasets.py
```
Note: In our settings, we crop the images of DeepFashion into the resolution of 176x256 in a center-crop manner.

Keypoints files

Download train/test pairs and train/test key points annotations from Google Drive, including fashion-resize-pairs-train.csv, fashion-resize-pairs-test.csv, fashion-resize-annotation-train.csv, fashion-resize-annotation-train.csv. Put these four files under the deepfashion directory.

Generate the pose heatmaps. Launch

python tool/generate_pose_map_fashion.py

Segmentation files
- Extract human segmentation results from existing human parser (e.g. Look into Person) and merge into 8 categories. Our segmentation results are provided in Google Drive, including ‘semantic_merge2’ and ‘semantic_merge3’ in different merge manner. Put one of them under the deepfashion directory.

Optionally, you can also generate these files by yourself.

Keypoints files

We use OpenPose to generate keypoints.

Download pose estimator from Google Drive. Put it under the root folder ADGAN.
Change the paths input_folder and output_path in tool/compute_coordinates.py. And then launch
```
python2 compute_coordinates.py
```

Dataset split files

python2 tool/create_pairs_dataset.py

Train a model

bash ./scripts/train.sh

Test a model

Download our pretrained model from Google Drive. Modify your data path and launch

bash ./scripts/test.sh

Evaluation

We adopt SSIM, IS, DS, CX for evaluation. This part is finished by Yiming Mao.

1) SSIM

For evaluation, Tensorflow 1.4.1(python3) is required.

python tool/getMetrics_market.py

2) DS Score

Download pretrained on VOC 300x300 model and install propper caffe version SSD. Put it in the ssd_score forlder.

python compute_ssd_score_fashion.py --input_dir path/to/generated/images

3) CX (Contextual Score)

Refer to folder ‘cx’ to compute contextual score.

Citation

If you use this code for your research, please cite our paper:

@inproceedings{men2020controllable,
  title={Controllable Person Image Synthesis with Attribute-Decomposed GAN},
  author={Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2020 IEEE Conference on},
  year={2020}
}

@inproceedings{pu2022controllable,
  title={Controllable Image Synthesis with Attribute-Decomposed GAN},
  author={Pu, Guo and Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Pattern Analysis and Machine Intelligence (TPAMI), 2022 IEEE Transactions on},
  year={2022}
}

Acknowledgments

Our code is based on PATN and thanks for their great work.

menyifang / ADGAN

readme