🔥Diffusers++🔥 is built on top of HuggingFace Diffusers, ensuring the inclusion of the latest state-of-the-art models related to image and video generation. 🔥Diffusers++🔥 is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, 🔥Diffusers++🔥 is a modular toolbox that supports both. Our library is designed with a focus on [usability over performance](https://huggingface.co/docs/diffusers/conceptual/philosophy#usability-over-performance), [simple over easy](https://huggingface.co/docs/diffusers/conceptual/philosophy#simple-over-easy), and [customizability over abstractions](https://huggingface.co/docs/diffusers/conceptual/philosophy#tweakable-contributorfriendly-over-abstraction). Diffusers++ offers three core components: - **Plus Pipelines** and **Plus Models** include the latest advancements such as CHAMP, ELLA, and FIFO-Diffusion. We strive to incorporate the latest advances in the image and audio fields to ensure our library remains cutting-edge. Additionally, we offer state-of-the-art [diffusion pipelines](https://huggingface.co/docs/diffusers/api/pipelines/overview) that can be run in inference with just a few lines of code. - Interchangeable noise [schedulers](https://huggingface.co/docs/diffusers/api/schedulers/overview) for different diffusion speeds and output quality. - Pretrained [models](https://huggingface.co/docs/diffusers/api/models/overview) that can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems. Additionally, **Plus Models** replicate some of the latest advancements, including CHAMP and ELLA, to provide state-of-the-art performance. ## Installation We recommend installing Diffusers++ in a virtual environment from PyPI or Conda. For more details about installing [PyTorch](https://pytorch.org/get-started/locally/), please refer to their official documentation. ### Diffusers++ Currently, Diffusers++ can be installed through cloning the repository: ```sh git clone https://github.com/ModelsLab/diffusers_plus_plus.git cd diffusers_plus_plus python -m pip install -e ``` ### Apple Silicon (M1/M2) support Please refer to the [How to use Stable Diffusion in Apple Silicon](https://huggingface.co/docs/diffusers/optimization/mps) guide. ## Quickstart Generating outputs is super easy with Diffusers++. To generate an image from text, use the `from_pretrained` method to load any pretrained diffusion model (browse the [Hub](https://huggingface.co/models?library=diffusers&sort=downloads) for 25.000+ checkpoints): ```python from diffusers import DiffusionPipeline import torch pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16) pipeline.to("cuda") pipeline("An image of a squirrel in Picasso style").images[0] ``` You can also dig into the models and schedulers toolbox to build your own diffusion system: ```python from diffusers import DDPMScheduler, UNet2DModel from PIL import Image import torch scheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256") model = UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda") scheduler.set_timesteps(50) sample_size = model.config.sample_size noise = torch.randn((1, 3, sample_size, sample_size), device="cuda") input = noise for t in scheduler.timesteps: with torch.no_grad(): noisy_residual = model(input, t).sample prev_noisy_sample = scheduler.step(noisy_residual, t, input).prev_sample input = prev_noisy_sample image = (input / 2 + 0.5).clamp(0, 1) image = image.cpu().permute(0, 2, 3, 1).numpy()[0] image = Image.fromarray((image * 255).round().astype("uint8")) image ``` Check out the [Quickstart](https://huggingface.co/docs/diffusers/quicktour) to launch your diffusion journey today! ## How to navigate the documentation | **Documentation** | **What can I learn?** | | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Tutorial](https://huggingface.co/docs/diffusers/tutorials/tutorial_overview) | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model. | | [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading_overview) | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. | | [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/pipeline_overview) | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. | | [Optimization](https://huggingface.co/docs/diffusers/optimization/opt_overview) | Guides for how to optimize your diffusion model to run faster and consume less memory. | | [Training](https://huggingface.co/docs/diffusers/training/overview) | Guides for how to train a diffusion model for different tasks with different training techniques. | ## Contribution We ❤️ contributions from the open-source community! If you want to contribute to this library, please check out our [Contribution guide](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md). You can look out for [issues](https://github.com/ModelsLab/diffusers_plus_plus/issues) you'd like to tackle to contribute to the library. - See [Good first issues](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) for general opportunities to contribute. - See [New model/pipeline](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+pipeline%2Fmodel%22) to contribute exciting new diffusion models/diffusion pipelines. - See [New scheduler](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+scheduler%22). - See [Bug](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22bug%22) if something isn't working. - See [Documentation](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22documentation%22) for improvements or additions to documentation. - See [Duplicate](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22duplicate%22) if this issue or pull request already exists. - See [Enhancement](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22enhancement%22) for new feature or request. - See [Help wanted](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) if extra attention is needed. - See [Invalid](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22invalid%22) if something doesn't seem right. - See [Question](https://github.com/ModelsLab/diffusers_plus_plus/issues?q=is%3Aopen+is%3Aissue+label%3A%22question%22) if further information is requested. Also, say 👋 in our public Discord channel
Task | Pipeline | 🔥Diffusers++🔥 Hub |
---|---|---|
Generating Infinite Videos from Text(upcoming) | FIFO-Diffusion | dummy/dummy-pipeline |
Text-to-Image | ELLA | dummy/dummy-pipeline |
Parametric 3D Human Animation via Latent Diffusion | CHAMP | dummy/dummy-pipeline |