search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.3k stars 378 forks source link

[Feature]: Lora support for Hunyuan, PixArt #3235

Open congdm opened 3 weeks ago

congdm commented 3 weeks ago

Feature description

Hi! Both https://github.com/Tencent/HunyuanDiT and https://github.com/PixArt-alpha/PixArt-sigma have their training code released. I think in the future there will be many new promising models, so it would be nice to support Lora for them.

Both HunyuanDiT and PixArt-sigma Lora are applied to the transformer:

# LoRA model
transformer = Transformer2DModel.from_pretrained(MODEL_ID, subfolder="transformer", torch_dtype=torch.float16)
transformer = PeftModel.from_pretrained(transformer, "Your-LoRA-Model-Path")

# Pipeline
pipe = PixArtAlphaPipeline.from_pretrained(MODEL_ID, transformer=transformer, torch_dtype=torch.float16)
del transformer

Version Platform Description

No response

vladmandic commented 3 weeks ago

valid, I'll look into adding support soon. please provide links to loras to test with for both models - without lora to test, i cannot just "release the code".

congdm commented 3 weeks ago

PixArt-alpha: https://huggingface.co/jasperai/flash-pixart Hunyuan: https://huggingface.co/Tencent-Hunyuan/HYDiT-LoRA (not for diffusers)

The Hunyuan official lora examples are actually for their ckpt model (its has different modules' names from the one in diffusers pipeline). Therefore, I have done my own lora training on diffusers model (attached file Test.zip )

I think there should be checking of base model class (in adapter_config.json), so users wouldn't load wrong lora by mistake.