[Feature Request]: Flux

[StableLlama]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Today https://blackforestlabs.ai/ (from the same people that created Stable Diffusion) introduced their first new modes: FLUX

Weights: https://huggingface.co/black-forest-labs/FLUX.1-dev Code to run: https://github.com/black-forest-labs/flux

It would be great to use it with the stable-diffusion-webui as well

Proposed workflow

1. Open WebUI
2. Select FLUX
3. Be happy

Additional information

No response

[mr-lab]

much needed

[Desm0nt]

Completely agree. Technicaly it is the real next SD model (instead of SD 3) from SD authors, and it's deserve to be run on really comfortable UI (instead of Comfy)

[micodev]

thumbs up

[BrechtCorbeel]

Lets hope this one doesn't get banned, but what we really need is more video models and better opensource video generators!!!

The people producing these are beating image models to death with unnecessary amounts of them, we need video and 3D!!

[cccyberwolke]

What is needed to get this done? Am i wrong in assuming it is not too big of a task seeing how quickly comfy supported it?

[levicki]

What is needed to get this done? Am i wrong in assuming it is not too big of a task seeing how quickly comfy supported it?

Here's what the authors say:

• Upgrade diffusers to latest
• Use FluxPipeline from diffusers

Example usage:

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    output_type="pil",
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image.save("flux-dev.png")

[Desm0nt]

What is needed to get this done? Am i wrong in assuming it is not too big of a task seeing how quickly comfy supported it?

It's not a big task for comfy or sd.next because they use diffusers (that already adapted for flux).

But SD webui use own custom implementation for SD, so it is need to be particularly rewritten and adapted for a new model architecture.

[colotoon]

it works in comfy in 8bit on 3090, its pretty fast in distilled mode, but loading times... oh man. img2img does not work that well, up to 0.8 denoise it just upscales original, past 0,81 it generates totally different pic. But it has best anatomy coherence from all local models, can generate people upside down like pony, no other models can do this.

please show ur node list, i dont understand how to make this works, especially node Load Diffusion model

[wizz13150]

please show ur node list, i dont understand how to make this works, especially node Load Diffusion model

My node was stuck too. The flux checkpoint actually go in the 'unet' folder, not in the 'checkpoint' folder. Don't ask me why.... it's comfy lol

Cheers 🥂

[colotoon]

please show ur node list, i dont understand how to make this works, especially node Load Diffusion model

My node was stuck too. The flux checkpoint actually go in the 'unet' folder, not in the 'checkpoint' folder. Don't ask me why.... it's comfy lol

Cheers 🥂

aaand, how select flux model? its not shows in the load checkpoint node

[rkfg]

its not shows in the load checkpoint node

You need to use a different node:

I suppose FLUX needs to be reassembled into just one .safetensors file first. It consists of UNET, VAE and two CLIP encoders, all separate. A1111 expects one file per model.

[colotoon]

its not shows in the load checkpoint node

You need to use a different node:

I suppose FLUX needs to be reassembled into just one .safetensors file first. It consists of UNET, VAE and two CLIP encoders, all separate. A1111 expects one file per model.

https://prnt.sc/MzIljv3j-tc9

it ask clip but in load diff model node i cant see clip connectors..

can u share ur node list plz IDK how it shoud be ((

[wizz13150]

can u share ur node list plz IDK how it shoud be ((

To end this discussion, obviously not about A1111, just use the default workflows ? : https://comfyanonymous.github.io/ComfyUI_examples/flux/

[AcademiaSD]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

[colotoon]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

[bews]

did u manage to make it work?

[uhd-ai]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

How did you use it? I was not able to run it on A1111

[colotoon]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

How did you use it? I was not able to run it on A1111

just use this weights: https://huggingface.co/Comfy-Org/flux1-dev/tree/main

and place to a1111 folder with SD models and thats all)

PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

[overgodofchaos]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

How did you use it? I was not able to run it on A1111

just use this weights: https://huggingface.co/Comfy-Org/flux1-dev/tree/main

and place to a1111 folder with SD models and thats all)

PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

The model tries to work if everything is done in the way described. But for some reason it tries to consume more than 32GB of video memory (I have a 16GB card + another 16GB is used as VRAM swapping from RAM, judging by the task manager) and crashes with an error.

This is strange considering that for ComfyUI this model uses about 14GB of video memory.

[Desm0nt]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

How did you use it? I was not able to run it on A1111

just use this weights: https://huggingface.co/Comfy-Org/flux1-dev/tree/main

and place to a1111 folder with SD models and thats all)

PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

I try to do as you say, it loads and try to sample, but blue noise in preview and grey square as result.

[AcademiaSD]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

How did you use it? I was not able to run it on A1111

just use this weights: https://huggingface.co/Comfy-Org/flux1-dev/tree/main and place to a1111 folder with SD models and thats all) PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

I try to do as you say, it loads and try to sample, but blue noise in preview and grey square as result.

same

[ZeroHackz]

Just bumping this request up!

[AcademiaSD]

This if FLUX1-dev checkpoint with included VAE, maybe it can help https://huggingface.co/Comfy-Org/flux1-dev/tree/main

Waaau!! Great thank to u bro! Now i can use it in a1111 yaahoo!!!

How did you use it? I was not able to run it on A1111

just use this weights: https://huggingface.co/Comfy-Org/flux1-dev/tree/main

and place to a1111 folder with SD models and thats all)

PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

Have you activated the fp8 mode? Have you checked the T5 mode for SD3? Have you updated any libraries? Show screenshot with the generation parameters please!

[bews]

PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

Uuuum... It uses previous model to generate the images, not Flux.

[ollobrains]

still not working sorry folks

[protector131090]

Please add FLux SUpport.

[colotoon]

PS: flux model not working if u use it and close a1111 and then start again (it most load by default) BUT, u need load any other model first and then u can load flux model again for work.

Uuuum... It uses previous model to generate the images, not Flux.

you were right, I ran tests, chose an Asian model, generated a girl, then changed the model to flux and clicking generate several times I got only Asian girls...

It's funny, but I don't understand why it works like this...

there are no errors in the console...

what a pity (((

[dhwz]

Please add FLux SUpport.

this is not gonna happen unless diffusers is supported (which previously was decided against it)

[zdoek001]

Please add FLux SUpport.

this is not gonna happen unless diffusers is supported (which previously was decided against it)

Does it mean that flux will not arrive on the webui?

[nitinmukesh]

Please add FLux SUpport.

this is not gonna happen unless diffusers is supported (which previously was decided against it)

Does it mean that flux will not arrive on the webui?

Looks like it will not be supported for now.

[ollobrains]

what would it take to reverse that position in terms of no support, if it becomes more popular perhaps

[mr-lab]

Please add FLux SUpport.

this is not gonna happen unless diffusers is supported (which previously was decided against it)

you are saying unless diffusers can run Flux there is no support for flux ? Well ask no more diffusers v0.30.0 is out with Flux support and much more , Let's Go https://github.com/huggingface/diffusers/releases/tag/v0.30.0

[dhwz]

Please add FLux SUpport.

this is not gonna happen unless diffusers is supported (which previously was decided against it)

you are saying unless diffusers can run Flux there is no support for flux ? Well ask no more diffusers v0.30.0 is out with Flux support and much more , Let's Go https://github.com/huggingface/diffusers/releases/tag/v0.30.0

No, I said stable-diffusion-webui doesn't use diffusers at all, it uses the official reference implementation of Stability AI.

[mr-lab]

dang it , what do we need to add support then ?

[ollobrains]

Please add FLux SUpport.

this is not gonna happen unless diffusers is supported (which previously was decided against it)

you are saying unless diffusers can run Flux there is no support for flux ? Well ask no more diffusers v0.30.0 is out with Flux support and much more , Let's Go https://github.com/huggingface/diffusers/releases/tag/v0.30.0

it does give another option should any automatic1111 like project decide to intergrate them, it would be good for auto1111 to do so tbh

[ollobrains]

dang it , what do we need to add support then ?

a popular push, community interest and perhaps some coders on board.

[mr-lab]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

[colotoon]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

How to use it?

[micodev]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

still not implemented yet.

[ollobrains]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

does that mean the coding implementation has begun ?

[ollobrains]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

still not implemented yet.

has the coding process started ?

[zethfoxster]

these are beating image models to death with unnecessary amounts of them, we need video and 3D!!

the tech is out there! go pioneer one yourself! shock us!

[ollobrains]

these are beating image models to death with unnecessary amounts of them, we need video and 3D!!

the tech is out there! go pioneer one yourself! shock us!

one step at a time but good sentiment

[Ahmet-Dedeler]

yo when is this going to be added?

[Gingertrout]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

still not implemented yet.

has the coding process started ?

No idea how this works (github and whot ) but I added a couple of lines to "backend/attention.py" around line 319:

if mask is not None:
    pad = 8 - q.shape[1] % 8
    mask_out = torch.empty([q.shape[0], q.shape[1], q.shape[1] + pad], dtype=q.dtype, device=q.device)

    mask_out = mask_out.expand([heads, -1, -1])

# mask_out[:, :, :mask.shape[-1]] = mask

    q = q.to(torch.float16)
    k = k.to(torch.float16)
    v = v.to(torch.float16)
    mask = mask.to(torch.float16).contiguous()

out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)

and very ugly brute forced a flux1.s and flux1.d model, but only generated 256x256 images, because I only have 24gb VRAM and my memory management is trash :') but could someone that knows what they are doing build on this, or get this to the right place, if it can help??

[ollobrains]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

still not implemented yet.

has the coding process started ?

No idea how this works (github and whot ) but I added a couple of lines to "backend/attention.py" around line 319:

if mask is not None:
    pad = 8 - q.shape[1] % 8
    mask_out = torch.empty([q.shape[0], q.shape[1], q.shape[1] + pad], dtype=q.dtype, device=q.device)

    mask_out = mask_out.expand([heads, -1, -1])

mask_out[:, :, :mask.shape[-1]] = mask

    q = q.to(torch.float16)
    k = k.to(torch.float16)
    v = v.to(torch.float16)
    mask = mask.to(torch.float16).contiguous()

out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)

and very ugly brute forced a flux1.s and flux1.d model, but only generated 256x256 images, because I only have 24gb VRAM and my memory management is trash :') but could someone that knows what they are doing build on this, or get this to the right place, if it can help??

So it proves with some coding it can be done, hopefully this starts a process of getting auto 1111 = flux support

[dhwz]

https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/backend/diffusion_engine/flux.py

still not implemented yet.

has the coding process started ?

No idea how this works (github and whot ) but I added a couple of lines to "backend/attention.py" around line 319:

if mask is not None:
    pad = 8 - q.shape[1] % 8
    mask_out = torch.empty([q.shape[0], q.shape[1], q.shape[1] + pad], dtype=q.dtype, device=q.device)

    mask_out = mask_out.expand([heads, -1, -1])

mask_out[:, :, :mask.shape[-1]] = mask

    q = q.to(torch.float16)
    k = k.to(torch.float16)
    v = v.to(torch.float16)
    mask = mask.to(torch.float16).contiguous()

out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)

and very ugly brute forced a flux1.s and flux1.d model, but only generated 256x256 images, because I only have 24gb VRAM and my memory management is trash :') but could someone that knows what they are doing build on this, or get this to the right place, if it can help??

So it proves with some coding it can be done, hopefully this starts a process of getting auto 1111 = flux support

Seems you still don't got it it's not gonna happen

[StableLlama]

Forge can run Flux now, so it shouldn't be too hard to add it to Auto1111, should it? https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/981

[mr-lab]

Forge can run Flux now, so it shouldn't be too hard to add it to Auto1111, should it? lllyasviel/stable-diffusion-webui-forge#981 well it happened
not only that 1.87it/s 14GBVRAM rtx 4090 some even ran it on rtx 2070

[Ahmet-Dedeler]

Forge can run Flux now, so it shouldn't be too hard to add it to Auto1111, should it? lllyasviel/stable-diffusion-webui-forge#981

is this being added now? I agree we should be getting it very soon, please devs...