Training for SDXL Finetunes?

[Kinglord]

Hello,

This isn't a bug but a question / feature request. I'm excited by BLoRAs but in my testing they have trouble working on SDXL finetunes, especially when the blocks have been significantly retrained.

Is it possible to use this to train on a checkpoint other than the base SDXL without a massive amount of effort? I saw where I think the blocks are being targeted, so I assume that can be adjusted as long as we know which blocks are the important ones for the finetune (not exactly easy but should be doable).

[ThereforeGames]

Hi @Kinglord,

Thanks for reaching out.

If you know which unet blocks you want to target for training, it is relatively easy to update the code. You would need to do so in two places:

1. The unet_target_name in the Lycoris preset. These are Regex strings that are parsed with re.match(). Right now, it targets blocks containing output_blocks.0.1 (content) and output_blocks.1.1 (style), excluding layers with the terms ff.net and proj - this effectively limits it to the same range as the original B-LoRA trainer.
2. The CONTENT_BLOCKS and STYLE_BLOCKS maps in blora_slicer.py. The syntax is a little different, but the idea is the same as the Lycoris preset.

You can also uncomment line 22 in the slicer and it will print every block name to your console. This is useful for getting a quick look at what's inside a given LoRA.

Of course, changing these values means you're not really performing the B-LoRA technique anymore, but I suppose that's alright. 😛

---

Having said all that, I have tested B-LoRA with the default SDXL checkpoint as well as Godiva. The latter is a realistic, PonyDiffusion-based model--a significant departure from original SDXL--and the B-LoRA content and style blocks still line up perfectly. May I ask which finetunes are yielding poor results with the method?

[ThereforeGames]

Quick update:

The new v0.1.0 of B-LoRA Slicer now reads from blora_traits.json! You can easily set up new rules there, instead of modifying the slicer script directly.

Example use:

    "my_custom_blocks":
    {
        "whitelist": ["lora_unet_some_blocks_here"],
        "blacklist": ["ff_net", "proj_in", "proj_out", "alpha"]
    },

python blora_slicer.py --loras C:/path/to/lora.safetensors --traits content my_custom_blocks --debug

And don't forget to set up the Lycoris preset accordingly.