We provides an easy-to-use LoaderMixin API to load adapter weights
Textual inversion is very similar to DreamBooth and it can also personalize a diffusion model to generate certain concepts (styles, objects) from just a few images. Now you can load the textual inversion embeddings with the load_textual_inversion() method and generate some images. Let’s load sd-concepts-library/gta5-artwork embeddings and you’ll need to include the special word in your prompt to trigger it:
from mindone.diffusers import StableDiffusionPipeline
pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
pipeline.load_textual_inversion("gta5-artwork/")
prompt = "A cute brown bear eating a slice of pizza, stunning color scheme, masterpiece, illustration, <gta5-artwork> style"
image = pipeline(prompt=prompt)[0][0]
image.save("textual_gta5.png")
IP-Adapter is a lightweight adapter that enables image prompting for any diffusion model. To start, load a Stable Diffusion checkpoint，then load the IP-Adapter h94/IP-Adapter weights and add it to the pipeline with theload_ip_adapter() method. Once loaded, you can use the pipeline with an image and text prompt to guide the image generation process:
from mindone.diffusers import StableDiffusionPipeline
from mindone.diffusers.utils import load_image
pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.safetensors")
image=load_image("https://huggingface.co/datasets/huggingface/documentationimages/resolve/main/diffusers/load_neg_embed.png")
image = pipeline(
prompt='best quality, high quality, wearing sunglasses',
ip_adapter_image=image,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
num_inference_steps=50,
)[0][0]
image.save("ipadapter.png")
Try it out!
Fixes # (issue)
Adds # (feature)
Before submitting
[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
documentation guidelines
[ ] Did you build and run the code without any errors?
[ ] Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
[ ] Did you write any new necessary tests?
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
What does this PR do?
add loaders: autoencoders, controlnet, single_file, ip_adapter, textual_inversion.
We provides an easy-to-use LoaderMixin API to load adapter weights
Textual inversion is very similar to DreamBooth and it can also personalize a diffusion model to generate certain concepts (styles, objects) from just a few images. Now you can load the textual inversion embeddings with the in your prompt to trigger it:
load_textual_inversion()method and generate some images. Let’s load sd-concepts-library/gta5-artwork embeddings and you’ll need to include the special wordIP-Adapter is a lightweight adapter that enables image prompting for any diffusion model. To start, load a Stable Diffusion checkpoint，then load the IP-Adapter h94/IP-Adapter weights and add it to the pipeline with the
load_ip_adapter()method. Once loaded, you can use the pipeline with an image and text prompt to guide the image generation process:Try it out!
Fixes # (issue)
Adds # (feature)
Before submitting
What's New. Here are the documentation guidelinesWho can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
@xxx