Classifier-free guidance (CFG) is a fundamental tool in modern diffusion models for text-guided generation. Although effective, CFG requires high guidance scales, which has notable drawbacks:
We propose a simple fix to this seemingly inherent limitation and propose CFG++, which corrects the off-manifold problem of CFG. The following advantages are observed
Experimental results confirm that our method significantly enhances performance in text-to-image generation, DDIM inversion, editing, and solving inverse problems, suggesting a wide-ranging impact and potential applications in various fields that utilize text guidance.
First, create your environment. We recommend using the following comments.
git clone https://github.com/CFGpp-diffusion/CFGpp.git
cd CFGpp
conda env create -f environment.yaml
For reproducibility, using the same package version is necessary since some dependencies lead to significant differences (for instance, diffusers). Nonetheless, improvement induced by CFG++ will be observed regardless the dependency.
If you run one of the below examples, diffusers will automatically download checkpoints for SDv1.5 or SDXL.
CFG
python -m examples.text_to_img --prompt "a portrait of a dog" --method "ddim" --cfg_guidance 7.5
CFG ++
python -m examples.text_to_img --prompt "a portrait of a dog" --method "ddim_cfg++" --cfg_guidance 0.6
CFG
python -m examples.inversion --prompt "a photography of baby fox" --method "ddim_inversion" --cfg_guidance 7.5
CFG ++
python -m examples.inversion --prompt "a photography of baby fox" --method "ddim_inverion_cfg++" --cfg_guidance 0.6
[!tip] If you want to use SDXL, add
--model sdxl
.
We provide callback functionality to monitor intermediate samples during the diffusion reverse process. For now, the function could be called only at the end of each timestep, for the readability of scripts.
Currently, we provide two options (default: None).
Note that using callback may take more time due to file save. You can refer utils/callback_util.py for details.
If you find our method useful, please cite as below or leave a star to this repository.
@article{chung2024cfg++,
title={CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models},
author={Chung, Hyungjin and Kim, Jeongsol and Park, Geon Yeong and Nam, Hyelin and Ye, Jong Chul},
journal={arXiv preprint arXiv:2406.08070},
year={2024}
}
[!note] This work is currently in the preprint stage, and there may be some changes to the code.