a Lora version to add to any Community Models?

tianweiy / DMD2

(NeurIPS 2024 Oral 🔥) Improved Distribution Matching Distillation for Fast Image Synthesis

Other

491 stars 27 forks source link

a Lora version to add to any Community Models? #5

Closed xiaoleexiang closed 3 months ago

xiaoleexiang commented 5 months ago

Could a Lora containing the difference between Base XL and This SDXL (4-steps) Checkpoint successfully transfer the 4-steps inference capability to any of the finetunes of Base SD-XL?

ThereforeGames commented 5 months ago

+1. It would be interesting to see how a DMD2 LoRA would fare against Hyper-SD or Lightning. Not to mention, a LoRA would allow Automatic WebUI users to try this tech!

tianweiy commented 5 months ago

we just released one. you can find the usage at https://huggingface.co/tianweiy/DMD2

First time using LORA for me so could you try and let me know if the format is correct?

Thanks

voodoohop commented 5 months ago

Will test later on today too applying it to a fine-tune of sdxl and report back.

tianweiy commented 5 months ago

I tried it with some finetuned models. It still seems to work the best with the original sdxl one :( . Probably there are some lora parameters that I haven't done right. I also noticed that we might need to use a smaller lora_scale during inference for the finetuned model to have less artifacts.

pipe.fuse_lora(lora_scale=0.75)

tianweiy commented 5 months ago

left is original sdxl + lora, right is Juggernaut-X-v10+lora

voodoohop commented 4 months ago

lora_scale 0.5 seems to be working ok for me with the 4step lora and this model:

https://civitai.com/models/413466/boltning-realistic-lightning-hyper

it's currently powering the image generation on http://pollinations.ai