Any way to apply Lora result of training script to Diffusers model?

Tencent / HunyuanDiT

Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

https://dit.hunyuan.tencent.com/

Other

2.58k stars 179 forks source link

Any way to apply Lora result of training script to Diffusers model? #95

Open congdm opened 1 week ago

congdm commented 1 week ago

Hi, at the moment I noticed that the transformer model in huggingface-cli Tencent-Hunyuan/HunyuanDiT has different structure than the transformer model in Diffusers pipeline. This makes me cannot apply lora such as links to Diffusers pipeline.

For example, the transformer in Tencent-Hunyuan/HunyuanDiT has fused qkv layer while the one in diffusers is unfused. They also different modules naming too. In long run, having to train lora separately for both types of transformer is very impractical.

xljh0520 commented 1 week ago

Thank you for your attention to our work. Due to certain compatibility adjustments and modifications made by the diffuser, there may be issues with the model state_dict keys not matching. We have provided a code example in lora README that demonstrates how to merge LoRA weights through keyword matching. We hope this can help you.