PEFT LoHa to kohya-ss LoHa conversion scripts

[Xynonners]

Feature request

To have a conversion script from peft (lycoris) to kohya-ss especially for SDXL.

Motivation

Currently, it's possible to use peft to centralize training of multiple different adapters (LoRA, LoCon, LoHa, LoKr) for Stable Diffusion, however the peft format is not accepted in some webuis. Having a good conversion script would allow better compatibility between peft and other webuis.

Your contribution

I have tried adapting the current peft to kohya_ss lora script to work for SDXL LoHa, however, there seems to be an extra number after the block (eg. lora_unet_input_blocks_X_1_transformer... VS base_model.model.down.blocks.attentions.0.transformer...) (sorry, writing this off the top of my head so may be slightly inaccurate) which I cannot find the meaning of or equivalent in peft, even when training SDXL adapters.

Additionally, the current script leaves an extra . at the end of the key kohya_key.count(".") - 2) while looking at the kohya-ss format there is only one. It would be great to have some insight to this.

[BenjaminBossan]

Thanks for looking into this. I assume that you're basing your work on this script:

https://github.com/huggingface/peft/blob/main/examples/lora_dreambooth/convert_peft_sd_lora_to_kohya_ss.py

Note that there has recently been an updated script that converts from kohya to PEFT, maybe that can work as a reference for reversing the process:

https://github.com/huggingface/peft/blob/main/examples/stable_diffusion/convert_sd_adapter_to_peft.py

I'm not very knowledgeable about kohya, but perhaps @kovalexal can give more insight into your questions.

In the mid to long run, it would be great to add regression tests for the conversion scripts to ensure that we can convert back and forth without error.

[kovalexal]

@Xynonners, hi!

Thanks for working on that.

I agree that there is still a lot of space for improving the PEFT -> kohya/LyCORIS conversion script to incorporate recently added LoHa and LoKr adapters and adapting it for SDXL compatibility.

Unfortunately, I don't have a lot of time to work on this right now, probably I will have some at the end of the week.

Meanwhile, @BenjaminBossan is right - the main idea is to reverse the kohya/LyCORIS -> PEFT conversion script to incorporate other adapters. There need to be performed the correct mapping between PEFT state_dict keys and kohya state_dict keys + adding alpha as an additional tensor.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.