search

comfyanonymous / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.
https://www.comfy.org/
GNU General Public License v3.0
52.45k stars 5.54k forks source link

[question] UNet Models #866

Open MoonMoon82 opened 1 year ago

MoonMoon82 commented 1 year ago

grafik What can I do with them? Where do I get them? SDXL related?

MaddyAurora commented 1 year ago

A model checkpoint that usually ends in ckpt or safetensors that we all usually use, like those you can download from civitai or the oficial SD 1.5, all are comprised of 3 actual models.

The model contains a Unet model, a CLIP model and a VAE model.

If you separate them, you can load that individual Unet model similarly how you can load a separate VAE model.

The Unet is the neural network model that generates the image in the latent space.

MoonMoon82 commented 1 year ago

@MaddyAurora Thank you for your detailed feedback ! I already noticed that some checkpoints come with more or less usefull VAEs, so I tended to use a separate VAE file independent of the checkpoint.

But I'm still not sure why there are separate loaders for Unet and Clip models. I never saw a "pure" Unet or Clip modelfile to download on ie civitai or huggingface. And for which purpose would it be useful to get Unet and Clip models separated from each other?

comfyanonymous commented 1 year ago

UNET Loader is for loading these files (unet files in diffusers format): https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main/unet

CLIP Loader is for loading these files: https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main/text_encoder

They are in advanced so you should only use them if you really need to.

samiede commented 1 year ago

Does it support sdxl unets as well?

comfyanonymous commented 1 year ago

Yes, it should support both SDXL and SDXL refiner unets.

Feihei commented 10 months ago

Is the unet loader only looking inside the models/unet path? Is it possible to specify another path for unet models in extra_model_paths.yaml ?

joshgura commented 2 months ago

unet loader is not launching a file dialog in order to select the unet. should it? same for the Load CLIP node. maybe the files are supposed to be in a default location, if so i apologize.

ltdrdata commented 2 months ago

unet loader is not launching a file dialog in order to select the unet. should it? same for the Load CLIP node. maybe the files are supposed to be in a default location, if so i apologize.

Fundamentally, ComfyUI operates as a server-based structure. Except for cases like image uploads from your computer, it uses files that are already stored on a remote server. Therefore, it only allows access to files in specific paths within the server