Photoswap: Personalized Subject Swapping in Images

Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang

NeurIPS 2023

[Project Page] [Paper]

Model Architecture

TODO

• [ ] Release Benchmark
• [x] Release code

:fire: News

• [2023.07.08] We have released our code.

Concept Learning

Training a model with your own concept

The new subject will be learned as a new token in the diffusion model. Huggingface provides scripts for training. In detail, you could use Text Inversion, DreamBooth, Custom Diffusion, or any other concept learning model. Be sure to install the package in the corresponding requirements.txt

• More source images leads to a better learnt concept and therefore a better subject swap result. For example, more human face images during training leads to a better artistic figure transfer.
• For DreamBooth, finetuning the encoder leads to a better performance, especially for human face. That would also requires more memory.

Download PHOTPSWAP models

Please contact jgu110@ucsc.edu for checkpoints.

Attention Swap

Following the below steps to do subject swapping:

1. Put the trained Diffusion Model checkpoint in checkpoints folder.
2. Install package using requirements.txt by pip install -r requirements.txt. Note that the concept learning environment is not suitable for attention swap.
3. Running real-image-swap.ipynb for subject swapping.

Different learnt concepts could have different swap steps for successful subject swapping. Tuning the swapping step and the text prompt for better performance. A concept model that has its weights tuned will have a degration in its ability on general concept generation. To do Subject Swapping with Photoswap, a single GPU with 16 GB memory is required.

Citation

@misc{gu2023photoswap,
      title={Photoswap: Personalized Subject Swapping in Images}, 
      author={Jing Gu and Yilin Wang and Nanxuan Zhao and Tsu-Jui Fu and Wei Xiong and Qing Liu and Zhifei Zhang and He Zhang and Jianming Zhang and HyunJoon Jung and Xin Eric Wang},
      year={2023},
      journal={Advances in Neural Information Processing Systems}
}