Open pin-lpt-heatherwick opened 8 months ago
The TransparentVAEDecoder is compatible with Diffusers, though the LoRA models may take some effort in translating keys.
@arpitsahni04 Specifically for the core LoRA model (not the others which change layer sizes):
Diffusers have some support for converting SGM/Automatic/Kohya format loras to diffusers format. This lora seemed to be a slightly different format which I handled with a few string replacements.
Specifically:
k = k.replace("diffusion_model", "lora_unet")
k = k.replace(".", "_")
k = k.replace("_weight::lora::0", ".lora_up.weight")
k = k.replace("_weight::lora::1", ".lora_down.weight")
@lachlan-nicholson I don't think it's going to work. For example, despite having similar naming conventions, the first middle block layer has the name and shape of: diffusion_model.middle_block.1.transformer_blocks.0.attn1.to_k.weight::lora::0 torch.Size([1280, 256])
and the expected name and size of: unet.mid_block.attentions.0.transformer_blocks.0.attn1.processor.to_k_lora.down.weight torch.Size([4, 1280])
am I mistaken or is there some way to convert them?
@pin-lpt-heatherwick I can confirm it does work.
Those shapes you shared are okay - the first is an up weight and the second is a down weight, so there shapes are transposed.
Lora-down-weights have shape [rank, dim]
in this case [r, 1280].
As long as the up and down weights have the same rank, there shouldn't be any concerns.
@lachlan-nicholson Hello, sorry to bother you. May I ask if you know what the preprocessing of the input and output data in the TransparentVAEDecoder looks like, and whether the range of latents and images is -1 to 1 or 0-1 or something like that. I am not good at using WebUI
@arpitsahni04 Specifically for the core LoRA model (not the others which change layer sizes):
Diffusers have some support for converting SGM/Automatic/Kohya format loras to diffusers format. This lora seemed to be a slightly different format which I handled with a few string replacements.
Specifically:
k = k.replace("diffusion_model", "lora_unet") k = k.replace(".", "_") k = k.replace("_weight::lora::0", ".lora_up.weight") k = k.replace("_weight::lora::1", ".lora_down.weight")
Do you mind sharing your code? I just wonder how you solve this issue. Thanks!
@arpitsahni04 Specifically for the core LoRA model (not the others which change layer sizes):
Diffusers have some support for converting SGM/Automatic/Kohya format loras to diffusers format. This lora seemed to be a slightly different format which I handled with a few string replacements.
Specifically:
k = k.replace("diffusion_model", "lora_unet") k = k.replace(".", "_") k = k.replace("_weight::lora::0", ".lora_up.weight") k = k.replace("_weight::lora::1", ".lora_down.weight")
Hey @lachlan-nicholson did you try applying the lora weights on the SD1.5 variant as well(layer_sd15_transparent_attn.safetensors)?
Will there be a version for diffusers?