About SVD Distillation - Githubissues

cloneofsimo / lora

Using Low-rank adaptation to quickly fine-tune diffusion models.

https://arxiv.org/abs/2106.09685

Apache License 2.0

6.99k stars 481 forks source link

About SVD Distillation #219

Open zhihongp opened 1 year ago

zhihongp commented 1 year ago

Great work. A pity that I am late on this so this might be a naïve question but it will help me greatly.

What is the SVD distillation (as mentioned in README) for? As I understand, with LoRA, the dimensionality reduction is already done in the form of low rank matrices A & B. What is SVD (singular value decomposition) distillation applied to?

cloudhan commented 1 year ago

See this discussion: https://github.com/cloneofsimo/lora/discussions/56#discussioncomment-4510380