add support for InstantX/FLUX.1-dev-Controlnet-Union

[geroldmeisinger]

Feature Idea

https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union-alpha

Existing Solutions

No response

Other

No response

[geroldmeisinger]

https://www.reddit.com/r/StableDiffusion/comments/1euaeum

[hartmark]

Awesome, it seems each day in AI is like a year in other technologies.

Looking forward to this.

I tried using an image with low denoise with flux and sometimes the results were good but it was after tweaking the prompt and denoise value for each photo I wanted to generate.

[EeroHeikkinen]

Now that #4444 has been merged you can load InstantX's union controlnet using this custom loader node: https://github.com/EeroHeikkinen/ComfyUI-eesahesNodes Workflow You can find more detailed instructions here: https://www.reddit.com/r/StableDiffusion/comments/1euz2a9/union_flux_controlnet_running_on_comfyui_workflow/

[geroldmeisinger]

it works great, thank you so much! memory requirements are quite high though. I wasn't able to create a 1024x1024 image with all flux and t5 in fp8 and even turning off all desktop managers and browser to free VRAM. I guess we need fp8 controlnets too.

[hartmark]

it works great, thank you so much! memory requirements are quite high though. I wasn't able to create a 1024x1024 image with all flux and t5 in fp8 and even turning off all desktop managers and browser to free VRAM. I guess we need fp8 controlnets too.

How much vram did you have?

[geroldmeisinger]

16 VRAM (about 200M occupied by DM), 64 SRAM, 4 swap

[geroldmeisinger]

https://github.com/comfyanonymous/ComfyUI/commit/10a79e989869f8878e27a8f373d85aef31822415 https://github.com/comfyanonymous/ComfyUI/commit/ea3f39bd6906dd455c867198d4d94152e76ad074 https://github.com/comfyanonymous/ComfyUI/commit/b33cd610703213dbe73baa6aaa3fdc2c61a84adc

[rootliuat]

The downloaded instantX flux union controller loader model should be placed in that directory