search

kohya-ss / sd-scripts

Apache License 2.0
5.02k stars 843 forks source link

[Feat] Dora extraction #1251

Open 311-code opened 5 months ago

311-code commented 5 months ago

I use the Lora extraction option from a dreambooth or finetuned checkpoint all the time as it seems to produce better results than just training a Lora. Would love to see a way to extract Dora.

If it's just a parameter change in Lora to implement Dora I don't think it would be too difficult?

kohya-ss commented 5 months ago

In my understanding, extracting DoRA may be possible. It needs the same preprocessing of DoRA (extracting magnitude vector), and SVD. However, I also think it doesn't improve the extracted DoRA quality from LoRA. Because SVD for DoRA and SVD for LoRA may have similar precision.

I'm not a math person, so please let me know if DoRA extraction has an advantage from LoRA.