search

luosiallen / latent-consistency-model

Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
MIT License
4.39k stars 231 forks source link

readme

Latent Consistency Models

Official Repository of the paper: Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference.

Official Repository of the paper: LCM-LoRA: A Universal Stable-Diffusion Acceleration Module.

Project Page: https://latent-consistency-models.github.io

Try our Demos:

🤗 Hugging Face Demo: Hugging Face Spaces 🔥🔥🔥

Replicate Demo: Replicate

OpenXLab Demo: Open in OpenXLab

LCM Community: Join our LCM discord channels for discussions. Coders are welcome to contribute.

Breaking News 🔥🔥!!

News

🔥 Image2Image Demos (Image-to-Image):

We support Img2Img now! Try the impressive img2img demos here: Replicate, SD-webui, ComfyUI, Colab

Local gradio for img2img is on the way!

🔥 Local gradio Demos (Text-to-Image):

To run the model locally, you can download the "local_gradio" folder:

  1. Install Pytorch (CUDA). MacOS system can download the "MPS" version of Pytorch. Please refer to: https://pytorch.org. Install Intel Extension for Pytorch as well if you're using Intel GPUs.
  2. Install the main library: 
    pip install diffusers transformers accelerate gradio==3.48.0
  3. Launch the gradio: (For MacOS users, need to set the device="mps" in app.py; For Intel GPU users, set device="xpu" in app.py) 
    python app.py

Demos & Models Released

Ours Hugging Face Demo and Model are released ! Latent Consistency Models are supported in 🧨 diffusers.

LCM Model Download: LCM_Dreamshaper_v7

LCM模型已上传到始智AI(wisemodel) 中文用户可在此下载，下载链接.

For Chinese users, download LCM here: (中文用户可以在此下载LCM模型) Open in OpenXLab

Hugging Face Demo: Hugging Face Spaces

Replicate Demo: Replicate

OpenXLab Demo: Open in OpenXLab

Tungsten Demo: Tungsten

Novita.AI Demo: Novita.AI Latent Consistency Playground

By distilling classifier-free guidance into the model's input, LCM can generate high-quality images in very short inference time. We compare the inference time at the setting of 768 x 768 resolution, CFG scale w=8, batchsize=4, using a A800 GPU.

Usage

We have official LCM Pipeline and LCM Scheduler in 🧨 Diffusers library now! The older usages will be deprecated.

You can try out Latency Consistency Models directly on: Hugging Face Spaces

To run the model yourself, you can leverage the 🧨 Diffusers library:

  1. Install the library:

    pip install --upgrade diffusers  # make sure to use at least diffusers >= 0.22
pip install transformers accelerate

  2. Run the model:

    
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7")

To save GPU memory, torch.float16 can be used, but it may compromise image quality.

pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.

num_inference_steps = 4

images = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, lcm_origin_steps=50, output_type="pil").images


For more information, please have a look at the official docs:
👉 https://huggingface.co/docs/diffusers/api/pipelines/latent_consistency_models#latent-consistency-models

## Usage (Deprecated)
We have official [**LCM Pipeline**](https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/latent_consistency_models) and [**LCM Scheduler**](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_lcm.py) in 🧨 Diffusers library now! The older usages will be deprecated. But you can still use the older usages by adding ```revision="fb9c5d1"``` from ```from_pretrained(...)``` 

To run the model yourself, you can leverage the 🧨 Diffusers library:
1. Install the library:

pip install diffusers transformers accelerate


2. Run the model:
```py
from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main", revision="fb9c5d")

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 4 

images = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, lcm_origin_steps=50, output_type="pil").images

Our Contributors :

BibTeX

LCM:
@misc{luo2023latent,
      title={Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference}, 
      author={Simian Luo and Yiqin Tan and Longbo Huang and Jian Li and Hang Zhao},
      year={2023},
      eprint={2310.04378},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

LCM-LoRA:
@article{luo2023lcm,
  title={LCM-LoRA: A Universal Stable-Diffusion Acceleration Module},
  author={Luo, Simian and Tan, Yiqin and Patil, Suraj and Gu, Daniel and von Platen, Patrick and Passos, Apolin{\'a}rio and Huang, Longbo and Li, Jian and Zhao, Hang},
  journal={arXiv preprint arXiv:2311.05556},
  year={2023}
}