search

shiimizu / ComfyUI-TiledDiffusion

Tiled Diffusion, MultiDiffusion, Mixture of Diffusers, and optimized VAE
285 stars 19 forks source link

Flux support #36

Open alex-mitov opened 3 weeks ago

alex-mitov commented 3 weeks ago

Hello,

I've been using the Tiled Diffusion node in ComfyUI with really good results. I was wondering if there are any plans to support the new Flux model by Black Forest Labs.

Thanks in advance!

Adreitz commented 3 weeks ago

Not the dev, but have you confirmed that it does not work? I haven't tried it yet with Flux dev, but I had tried it with SD3. It seemed to work, but failed when I introduced the SD3 tile controlnet (it applied the whole initial image to each tile, rather than just the corresponding portion of the initial image). Since there is no Flux tile controlnet yet to my knowledge, and Flux seems to hallucinate very little when naively tiling anyway, this might not be a big problem.

alex-mitov commented 3 weeks ago

I meant to say that the problem happens with the tile controlnet. Sorry for not being clear. Just like you described, it applies the initial image to each tile. :(

There already is a tile control net for Flux as part of the Union controlnet model by Instant X.

Adreitz commented 3 weeks ago

Thanks for the quick confirmation! While getting the node working correctly with tile is important in the long run, I'd recommend trying it without the controlnet with Flux. Up to now I've just been using Ultimate SD Upscale node with Flux with barely any hallucinations or tile discontinuities (mostly in very smooth backgrounds where one tile might not have any detail to speak of, so Flux might introduce speckles or posterization in the gradations in the second pass), so hopefully it will work well with this node, too.

Example with no controlnets: ComfyUI_00013_ ComfyUI_00014_

alex-mitov commented 3 weeks ago

Thanks for the quick confirmation! While getting the node working correctly with tile is important in the long run, I'd recommend trying it without the controlnet with Flux. Up to now I've just been using Ultimate SD Upscale node with Flux with barely any hallucinations or tile discontinuities (mostly in very smooth backgrounds where one tile might not have any detail to speak of, so Flux might introduce speckles or posterization in the gradations in the second pass), so hopefully it will work well with this node, too.

Example with no controlnets:

Looks good! Have you been able to creatively resample or/and upscale with Flux? I'm getting much more creative results when resampling/upscaling images with SD 1.5 models and the SD 1.5 tile controlnet.

Adreitz commented 3 weeks ago

I haven't tried. My personal opinion is that the creative aspect should happen at the initial low-res generation, where the quality will be the best, and that upscaling should simply be a matter of increasing the image detail with as few compositional changes as possible. Flux is working pretty well for me in that regard, but it means I don't really have experience with other ways of working.

Vigilence commented 2 weeks ago

This node doesn't support the GGUF version but does the fp8 version. I look forward to the gguf version being supported in the future.

shiimizu commented 2 weeks ago

Flux fp & gguf should be okay now.

alex-mitov commented 2 weeks ago

Hi, @shiimizu . I was very excited to try the new update to Tiled Diffusion, but it still doesn't work for me. I'm still getting the same image in each tile. Am I doing something wrong? Please see the screenshot below..

I tried both Flux FP16 and Flux FP8, and I also tried Multi Diffusion in both Mixture of Diffusers and MultiDiffusion modes. My ComfyUI is updated to the latest version - 2639 [b643ea] (2024-09-01).

tiled_diffusion_flux_not_working

Vigilence commented 2 weeks ago

Does anyone have a working workflow for flux? I keep getting:

The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0

alex-mitov commented 2 weeks ago

Does anyone have a working workflow for flux? I keep getting:

The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0

Make sure the widths and heights of your latents and images are always divisible by 16. Flux doesn't like dimensions that are not divisible by 16 and will often give errors if you don't follow this rule.

shiimizu commented 2 weeks ago

@alex-mitov Instantx/union flux controlnets aren't working at the moment. I'll try to fix it. The other depth & canny ones work fine though.

Vigilence commented 2 weeks ago

Does anyone have a working workflow for flux? I keep getting: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 0

Make sure the widths and heights of your latents and images are always divisible by 16. Flux doesn't like dimensions that are not divisible by 16 and will often give errors if you don't follow this rule.

I just checked and they are divisible by 16.

shiimizu commented 2 weeks ago

FLUX Union ControlNets should work now.

@Vigilence Is that related to TiledDiffusion? If so, open a new issue with a workflow and full error stacktrace.

alex-mitov commented 2 weeks ago

I can confirm that Tiled Diffusion now works as expected with Flux!

hakbodi commented 1 week ago

I can confirm that Tiled Diffusion now works as expected with Flux!

Can you share a workflow for that please?

alex-mitov commented 1 week ago

I can confirm that Tiled Diffusion now works as expected with Flux!

Can you share a workflow for that please?

Just connect the Tiled Diffusion node to the "model" input of your Ksampler. Nothing complicated. :)

uzitotte commented 11 hours ago

I also need a workflow for this..