How can I use flux model?

[K-O-N-B]

My comfyui and webui shared all the checkpoints. Krita can found my SD5 and SDXL model. But flux con't. Local server mode.

[mvstrawn]

Two things: One, the directory recently changed in ComfyUI from "unet" to "diffusion_models" so you should change the name of your folder. Two, you need to modify the file .../ComfyUI/extra_model_paths.yaml like so,

    checkpoints: |
         models/checkpoints
         models/diffusion_models

[yecai2024]

Two things: One, the directory recently changed in ComfyUI from "unet" to "diffusion_models" so you should change the name of your folder. Two, you need to modify the file .../ComfyUI/extra_model_paths.yaml like so,

    checkpoints: |
         models/checkpoints
         models/diffusion_models

Thank you so much for providing the solution! Could you please let me know which line I should place these code on? I tried putting it in the two positions indicated by the arrow, but my ComfyUI wouldn't start after that. Could I see a screenshot of your file?

[creeduk]

Can the extension stay as sft or do we need a symlink from sft to safetensors?

[mvstrawn]

@yecai2024 Well I used A1111 before CUI, so my file will look a little different if you didn't do the same, but here is how mine is done:

#config for a1111 ui
#all you have to do is change the base_path to where yours is installed
a1111:
    base_path: /opt/stable-diffusion-web-ui/

    checkpoints: |
         /games/.sd-models/Stable-diffusion
         /games/.sd-models/diffusion_models
    diffusion_models:
         /games/.sd-models/diffusion_models
    configs: /games/.sd-models/Stable-diffusion
    vae: /games/.sd-models/VAE
    vae_approx: /games/.sd-models/VAE-approx
    loras: |
         /games/.sd-models/Lora
         /games/.sd-models/LyCORIS
    upscale_models: |
                  /games/.sd-models/ESRGAN
                  /games/.sd-models/RealESRGAN
                  /games/.sd-models/SwinIR
    embeddings: /games/.sd-models/embeddings
    hypernetworks: /games/.sd-models/hypernetworks
    controlnet: /games/.sd-models/ControlNet
    clip: /games/.sd-models/clip

Reading over this again it didn't occur to me that I had a reference to the diffusion_models folder in two different places, as well as that the diffusion_models entry is a custom one. Apologies for that, but for what it's worth this does work for me.

@creeduk I think the process of detecting valid Flux models is more complex than that. I made a comment about it in another issue here but I don't recall where. Regardless, I have multiple Flux models in my directory, some of which are picked up by the plugin and some of which are not (all of them are usable in CUI however), so I'm not sure how to handle that.

[creeduk]

Thank you.

I also used A111 previously so mine looks similar without the added diffusion which I might have to add. I was able to get my clips sorted (still used name CLIP-ViT-bigG-14-laion2B-39B-b160k) by mklink, I had installed my flux only in comfy as that is all I use now (I really should just port all files over one day). Also don't do what I did and forget to restart comfy a sit is comfy that reads the yaml not krita plug (yeah I had a few restarts before I realized what I had done :) )

no flux showing have an error in api.py - in inspect_diffusion_model base_model = model_detection.model_config_from_unet_config(unet_config)

Maybe too many linked and it is messing it up so I am going to dial back and start over, yaml in place and see if I can get it to show.

[mvstrawn]

Oh right that reminds me. I get this error, which is probably related, when I start up the server and when I refresh my model list. I think it's an error with ComfyUI though and not with this plugin. I suspect it's the reason the plugin doesn't display some of the models.

Traceback (most recent call last):
  File "/opt/ComfyUI/custom_nodes/comfyui-tooling-nodes/api.py", line 78, in inspect_diffusion_model
    base_model = model_detection.model_config_from_unet_config(unet_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/model_detection.py", line 277, in model_config_from_unet_config
    if model_config.matches(unet_config, state_dict):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/supported_models_base.py", line 56, in matches
    if k not in unet_config or s.unet_config[k] != unet_config[k]:
       ^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/opt/ComfyUI/custom_nodes/comfyui-tooling-nodes/api.py", line 78, in inspect_diffusion_model
    base_model = model_detection.model_config_from_unet_config(unet_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/model_detection.py", line 277, in model_config_from_unet_config
    if model_config.matches(unet_config, state_dict):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/supported_models_base.py", line 56, in matches
    if k not in unet_config or s.unet_config[k] != unet_config[k]:
       ^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/opt/ComfyUI/custom_nodes/comfyui-tooling-nodes/api.py", line 78, in inspect_diffusion_model
    base_model = model_detection.model_config_from_unet_config(unet_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/model_detection.py", line 277, in model_config_from_unet_config
    if model_config.matches(unet_config, state_dict):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ComfyUI/comfy/supported_models_base.py", line 56, in matches
    if k not in unet_config or s.unet_config[k] != unet_config[k]:
       ^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

[creeduk]

yes that is what I am getting, one of the diffusion, I have the dev1 flux I wonder if it does not like that one

[creeduk]

Yeah I removed references to diffusion (so only SDXL and SD1.5 loading) and that error is gone. Will add a new folder with only 1 link at a time see how it goes.

Ah well it only seems to support flux1-dev-fp8.safetensors or flux1-schnell-fp8.safetensors that will be it, so it probably se's your other model and throws that error. I also found it ignores .sft

[mvstrawn]

For me it recognizes the speedFP8 models (https://civitai.com/models/629858?modelVersionId=488645) as well as flux1DevHyperNF4Flux1DevBNB_flux1DevHyperNF4.safetensors (https://civitai.com/models/638187?modelVersionId=819165) but I also have the default flux1-schnell.sft from their huggingface.co repo, which is not listed in the plugin (I renamed the .sft extension to .safetensors to no avail). So yeah I have no idea what the criteria is.

[Acly]

diffusion_models are not checkpoints, don't mix those folders

the following folders are equivalent: unet = diffusion_model both are for files which contain the diffusion model only

checkpoints are files which contain diffusion model along with image encoder (vae) and text encoder (t5, clip)

[Acly]

https://github.com/Acly/krita-ai-diffusion/discussions/1176

[creeduk]

I will try the update as I have flux1-dev.sft in my diffusion_model folder (was unet and symlinked) which works fine in comfyui (using for about 6 weeks). MAin interest would be the inpainting being able to leverage the text and fix text from generated images.