search

Kosinkadink / ComfyUI-Advanced-ControlNet

ControlNet scheduling and masking nodes with sliding context support
GNU General Public License v3.0
545 stars 54 forks source link

[Feature request] Flux regional conditioning support #153

Open tristan22mc opened 1 month ago

tristan22mc commented 1 month ago

The new controlnets for flux are not the best quality however I would love to be able to use the controlnets regionally so that we can apply the controlnet only to the subject we are trying to control for, as the current controlnet implementations crush the backgrounds quite badly. I used to do this with SDXL before we got good controlnets. It helps to let the model still generate the backgrounds freely without influence from the controlnet. Essentially Im asking if we can implement Flux with regional conditioning support

mikebilly commented 1 month ago

Yes, please implement it, I cannot use controlnet without attention mask

Kosinkadink commented 1 month ago

Since flux controlnet is supported via vanilla comfyui as of a day ago, I should be able to make it work with all the Advanced-ControlNet features relatively easy.

I haven't messed with Flux at all yet as I have a lot on my plate, but in the next few days I should have it ready. I'll update this issue when it's pushed out.

tristan22mc commented 4 weeks ago

Amazing! Best news today!

mikebilly commented 3 weeks ago

Is this implemented yet? @Kosinkadink

Kosinkadink commented 3 weeks ago

Not yet, been working on AnimateDiff-Evolved stuff this week. I should be able to get to it sometime this coming week.

tristan22mc commented 2 weeks ago

Just checking in! Waiting patiently hah

Kosinkadink commented 2 weeks ago

In develop branch, last night I pushed up support for most ACN features for flux (scheduling, soft scaled weights, etc).

Masking will still need work, as unlike SD, Flux's tensors while sampling don't have simple x,y dimensions - the dims get combined at some point, so I'd need to dig through the code to figure out exactly how it gets combined so I can do the same to my own mask tensor.

General cond masking that ComfyUI supports does work currently, but I will try to make the ACN attn masking work before I do any merging into main.

tristan22mc commented 2 weeks ago

Ahh super interesting. So flux is down sampling or combining spatial dimensions between layers? Jealous of your knowledge, been trying to gain a deeper understanding of how things work lately. Looking forward to seeing how you tackle that!

tristan22mc commented 2 weeks ago

By the way when you mentioned that Comfyui supports cond masking atm did you mean with controlnets too? Went and tried to test and couldnt get it working

alex-mitov commented 2 weeks ago

I've tried the develop branch and it works with the InstantX controlnet Union model, but only with the alpha version of that model. The final release produces poor quality results with the tile controlnet. Is this a problem with the controlnet or is it because Flux support is still in WIP stage? Thanks for the great work so far! :)

mikebilly commented 2 weeks ago

@Kosinkadink any updates?

Kosinkadink commented 1 week ago

@mikebilly In develop branch, all ACN features besides masking work for the xlabs flux models and the instantx depth model. flux is very odd with how it deals with encoding width and height into images, thought I figured it out but the results are not as expected. I'll give it another go today, but either way I will probably merge things into main branch regardless of if I can get masking working prior to initial release.

@tristan22mc yes, cond masking works for controlnets too, assuming everything's plugged in correctly. the attention-level masking is not currently supported in dev branch for flux controlnets through the Apply Advanced ControlNet nodes. I also added an Apply Advanced ControlNet(1) node that only receives one cond to simplify connecting for certain workflows.

As for the flux union model, comfy added the initial support for it in ComfyUI, so I can piggy back off of that and make it possible to modify the type soon.

Kosinkadink commented 1 week ago

I pushed what I have so far into main branch, i'll keep the thread posted with progress on masking.

tristan22mc commented 1 week ago

Awesome thanks for the update! Also Im currently working with Xinsir to help him find compute for Flux controlnets. Wanted to see if you knew of any possible resources or people in your network that would be interested in getting involved in the project. Also happy to connect you guys if it would be of any value