search

nihedon / sd-webui-lora-block-weight

4 stars 0 forks source link

Flux support #2

Open diodiogod opened 3 weeks ago

diodiogod commented 3 weeks ago

I see you are trying to add Flux support and the new Forge support (I imagine that means a more current commit?). How is it going? I tried updating and it was not showing. Is it still in development? Thanks a lot for this! I really appreciate you efforts!

nihedon commented 2 weeks ago

@diodiogod Actually, I’ve added support for Flux LoRA weights, but I’m still uncertain if the implementation is correct since it’s based on my own interpretation. As you know, the previous StableDiffusion models' LoRA consisted of 25 blocks (26 if you include Base), but Flux LoRA uses 19 blocks for both single and double. This makes it incompatible with previous LoRA models, and since I couldn’t find any documentation, my implementation is entirely based on my own understanding.

For now, you should be able to run Flux LBW by using the following format: <lora:some_lora:1:lbw=1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1>

I’ve also implemented this in the XYZ plot, where you can check it under "LoRA Block Weight" -> "XYZ plot" in the txt2img tab.

Additionally, I’ve created a branch called "flux" for Weight Helper, where a Flux-compatible LoRA popup should appear.

I’m curious about the issue you mentioned where it was not showing. Was it not working correctly?

diodiogod commented 2 weeks ago

I tried changing branches but I did not update the helper (I think I completely forgot they are two extensions 😂) that is why I was not seeing anything related to flux showing up. I'll test as soon as I can and get back to you. Thanks for the clarification. I know (but have not had thte time to test it yet), that there is a node on comfyui that does flux block weights as well, maybe that can help you in further developments (if needed)

Also, does it still works only on the 8e4a6fc commit of Forge, or did you manage to update it? Thanks!

diodiogod commented 2 weeks ago

Wow I'm using it and it is GREAT! I'm baffled by the quality of the results I'm getting by messing around with flux lora blocks. I really can't thank you enough!! You should announce this (and the helper) on Reddit, I bet more people would love it!

The only thing I'm missing now, that would be perfect, is a way to actually save the Lora with its weight blocks set... I used to do it with Supermerger extension using "SAME TO STRENGTH", also by @hako-mikan. But I don't know of any other solution for Flux. It would be amazing to share LoRas on Civitai, for example, with a block analysis already set for people who don't know how to use block weighs.

nihedon commented 2 weeks ago

@diodiogod Thank you for testing it out! I'm glad to hear that it's working correctly! (In my environment, generating images with Flux takes about 5 minutes per image, so testing was quite challenging...)

While it would be nice to share this with more people, I'm content with programming quietly on my own. As long as even one person is satisfied, that's enough for me ☺.

My fork of LoRA Block Weight is compatible with the latest version of Forge. It should work not only with 8e4a6fc, but also with the most recent versions.

As for Supermerger, I’ve managed to get it displayed in the WebUI, but none of the buttons are functional yet. I’ll try to address this slowly, but I do plan on working on it.

diodiogod commented 1 week ago

@nihedon, from what I've seen in som comfyui nodes implementations, Flux Loras have a total of 57 blocks. 19 (from 0 to 18) from the double layers and 38 (from 0 to 37) from the single layers. Some nodes like the "Lora Lader (Block Weight)" from Inspire-Pack allow us to set these values (but they have 58 values... I imagine that the first is the TE?? I should probably ask them, not you).

Anyway. I wonder, how does it translate from how you implemented it. Your format allow 19 values. So if I remove the third block for a lora as an example, does it mean that I removed: "double_blocks.2.": 0.0 -> the third from the 19 (0-18) double blocks "single_blocks.2.": 0.0 -> The third from the first 19 (0-18) single blocks "single_blocks.21.": 0.0 -> the third block after 19+3 of the single blocks

Is that correct?

So, your lbw=1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 would translate to a 57 blocks format to this? 1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1

Thanks!!

nihedon commented 1 week ago

@diodiogod Yes, your interpretation is mostly correct. However, the way the 19 blocks are distributed differs slightly from the LBW specifications, so let me clarify.

The focus of the issue is the distribution to Single layers. In LBW, for instance, if you overwrite the third block:

are all subject to modification. The fifth and sixth positions might seem a bit strange, but this represents the blocks "3 x 2 - 1" and "3 x 2."

It's difficult to explain in words, so I summarized it in a table. image

As shown above, there are two interpretations for the distribution to Single layers. LBW adopts "Single (Interpretation 1)," but in reality, "Single (Interpretation 2)" might be the correct one...

I could have provided 57 blocks, but that would make the prompt input form quite cluttered. Additionally, having too many sliders in the Weight Helper I’m developing would just make it confusing, so I consolidated it to 19 blocks 😅. However, if Double and Single serve entirely different roles, I think it would be more appropriate to provide all blocks, similar to ComfyUI's Inspire-Pack.

To conclude, if we express the above in LBW with 57 blocks, it would be 1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1.

If you find any mistakes, I would appreciate it if you could let me know!

diodiogod commented 1 week ago

Ok got it! Fantastic. This is merely a suggestion, but would you to consider accepting the complete 57 block format as well? Maybe the 19 blocks could be the standard, but also have an option on the helper to switch to the 57 sliders interface. I know it would be cluttered but it would help test all these blocks to see if they actually have an effect. I can try on comfyui right now, but it takes so much time to find, edit and run... so many numbers.... and your helper is so good with this.... anyway. Thanks!

Also, may I ask some advice? I'm developing (not really me, but ChatGPT), a tool to adjust and save the LoRA with the desired changed block weighs. I also used your explanation to convert 19 to 57 blocks on the code, if that was the input. So far I've managed to make it work, but I'm not sure if it is correct, here it is "my" code: https://github.com/diodiogod/Flux-Block-Weight-Remerger I'm not sure if I'm implementing this right. I'm targeting all the all "lora_B" blocks because it's what gave me the closest results to what I get with your extension. But still, I have no idea if this is the correct target layers. Do you know if this is correct? I should really be targeting only the b and not the A blocks? Should I maybe target only atnn or other layers? Also I was afraid of removing the tensor and breaking it, so I instead changed the weight to zero.

nihedon commented 1 week ago

@diodiogod That's a good idea. I've added support for specifying both 19 blocks and 57 blocks 870c1a3fd6d42d4c01f0b2e7dbc985ac53d59e9a. Please note the following when using XYZ:

I've also added support for "19ALL" and "57ALL." I wanted to conduct more tests, but the image generation with Flux in my environment is too slow, so my testing might be insufficient.

Additionally, regarding the Weight Helper, I think the implementation will take longer since it requires a redesign of the UI.

Also, I've checked out the Flux-Block-Weight-Remerger. You're creating a very interesting extension! I don't quite understand the keywords "lora_A" and "lora_B," and upon investigating why I'm unfamiliar with them, it seems that by the time the LoRA models are loaded and usable in the web UI, the names "lora_A" and "lora_B" are stripped away. Therefore, those key names don't need to be considered in LoRA Block Weight. I believe that internally, "lora_A" and "lora_B" are appropriately distributed to "something." I will analyze this further.