Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching 🥯[Arxiv]
Xinyin Ma, Gongfan Fang, [Michael Bi Mi](), Xinchao Wang
Learning and Vision Lab, National University of Singapore, Huawei Technologies Ltd
We introduce a novel scheme, named Learning-to-Cache (L2C), that learns to conduct caching in a dynamic manner for diffusion transformers. A router is optimized to decide the layers to be cached.
Some takeaways:
A large proportion of layers in the diffusion transformer can be removed, without updating the model parameters.
L2C largely outperforms samplers such as DDIM and DPM-Solver.
Model | NFE | Checkpoint |
---|---|---|
DiT-XL/2 | 50 | link |
DiT-XL/2 | 20 | link |
U-ViT-H/2 | 50 | link |
U-ViT-H/2 | 20 | link |
We implement Learning-to-Cache on two basic structures: DiT and U-ViT. Check the instructions below:
@misc{ma2024learningtocache,
title={Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching},
author={Xinyin Ma and Gongfan Fang and Michael Bi Mi and Xinchao Wang},
year={2024},
eprint={2406.01733},
archivePrefix={arXiv},
primaryClass={cs.LG}
}