search

ogkalu2 / Merge-Stable-Diffusion-models-without-distortion

Adaptation of the merging method described in the paper - Git Re-Basin: Merging Models modulo Permutation Symmetries (https://arxiv.org/abs/2209.04836) for Stable Diffusion
MIT License
139 stars 21 forks source link

Writing a PermutationSpec for SDXL #44

Open AI-Casanova opened 9 months ago

AI-Casanova commented 9 months ago

Do you have any plans to Develop a PermutationSpec for SDXL?

If not, would you be so kind as to explain a bit more how you arrived at the one for 1.5?

I read through your conversation over on the original repo, but I'm having trouble following your work compared to a raw model.

ogkalu2 commented 9 months ago

I plan to write for XL. I'm just a bit swamped at the moment.

AI-Casanova commented 9 months ago

I think I got the hang of it.

I'm working up a PermutationSpec for SDXL without VAE keys (I find no reason to ever merge VAE, simply replace if necessary) if I get it to work, I'll link it here for your use.

AI-Casanova commented 9 months ago

https://github.com/vladmandic/automatic/blob/dev/modules/merging/merge_PermSpec_SDXL.py

SDXL PermutationSpec

ogkalu2 commented 9 months ago

Nicely done! Did you run a test ? Do you know what layers change when you attempt a merge ?

AI-Casanova commented 9 months ago

It seems to work fine (my code is based on @s1dlx code which is based on yours) but I haven't delved deeper.

AI-Casanova commented 9 months ago

xyz_grid-xyz_grid-0000-basic50-a_8k_high_definition_photo_of_a_beautiful

6DammK9 commented 4 months ago

Finally got it done. https://github.com/6DammK9/Merge-Stable-Diffusion-models-without-distortion/blob/main/merge_PermSpec_SDXL.py

You need to "sew" it properly, the permutation loop somehow iterlate the whole model in sequence, and checking tensor sizes. No wonder why it is left untouched for months.

Sample image in my fork, with comparasion.

AI-Casanova commented 4 months ago

Amazing! I got it as far as I could, glad someone picked it up from there.