[Feature]: LyCORIS OFT Support

[v0xie]

Feature description

LyCORIS now has support for OFT networks in the dev branch, (https://github.com/KohakuBlueleaf/LyCORIS/blob/dev/lycoris/modules/diag_oft.py) so I thought I'd open a feature request here to give a heads-up.

I've got an open pull request on A1111 with support for LyCORIS OFT: https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/13692/files

The major differences between kohya-ss and LyCORIS OFT:

• The weight is named _"oftdiag", and there is no "alpha" parameter
• The variables _block_num, blocksize are reversed from kohya-ss (correct size of blocks from paper is [d, d/r, d/r])
• The PR uses LyCORIS calc_updown() which works for both kohya-ss and LyCORIS OFT
  • It's much faster than merging weights like in kohya-ss

Thank you!

Version Platform Description

n/a

[vladmandic]

we already support lora oft, but i this is just slightly different. is there a trained lora that can be used for dev/test?

[v0xie]

Here's a diag-oft network and sample image. https://huggingface.co/v0xie/sdxl-oft_monster_toy/blob/main/monster_toy_diag-oft-000060.safetensors

[vladmandic]

i took a quick look at integrating this, but i think it needs more work... i'm getting dtype mismatch in def calc_updown_kb and scenarios where out_dim is not set at all since block is none of the items in if/elif block in class NetworkModuleOFT

[v0xie]

Maybe the dtype error is related to the R = R * multiplier + torch.eye(self.block_size, device=orig_weight.device) line? That's the only line in the function without a dtype specifier.

You can probably set out_dim to num_blocks * block_size or self.oft_blocks.shape[0] * self.oft_blocks.shape[1] since those are necessarily divisible by the module's out_dim.

[vladmandic]

Maybe the dtype error is related to the R = R * multiplier + torch.eye(self.block_size, device=orig_weight.device) line? That's the only line in the function without a dtype specifier.

yes, thats the line and i've worked around it by moving cast from couple of lines below to above the block.

You can probably set out_dim to num_blocks * block_size or self.oft_blocks.shape[0] * self.oft_blocks.shape[1] since those are necessarily divisible by the module's out_dim.

i tried that and it passes, but it was still not producing results that i expected. need to dive deeper into this, but it will have to wait.

[vladmandic]

done in dev branch.