Closed ilisparrow closed 11 months ago
sayakpaul
commented
11 months ago The issue is a bit convoluted. Can you provide a shorter code snippet and help me understand what's the problem?
https://github.com/huggingface/diffusers/issues/4664 is already solved. See here: https://github.com/huggingface/diffusers/issues/4664#issuecomment-1689339390.
ilisparrow
commented
11 months ago Hello, Sorry, indeed it was, because of the code markdown. I fixed it. Here is the code to reproduce it. You would need any Lora for SDXL inside a ./lora folder :
import os
import datetime
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler, AutoencoderKL
from diffusers.utils import load_image, make_image_grid
from controlnet_aux.lineart import LineartDetector
import torch
# load adapter
adapter = T2IAdapter.from_pretrained(
"TencentARC/t2i-adapter-lineart-sdxl-1.0", torch_dtype=torch.float16, varient="fp16"
).to("cuda")
# load euler_a scheduler
model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
euler_a = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
model_id, vae=vae, adapter=adapter, scheduler=euler_a, torch_dtype=torch.float16, variant="fp16",
).to("cuda")
pipe.load_lora_weights("lora/Samaritan 3d Cartoon SDXL.safetensors")
pipe.enable_xformers_memory_efficient_attention()
line_detector = LineartDetector.from_pretrained("lllyasviel/Annotators").to("cuda")
#url = "https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/org_lin.jpg"
url = "https://images.pexels.com/photos/18710356/pexels-photo-18710356/free-photo-of-woman-in-eyeglasses-in-black-and-white.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1"
image = load_image(url)
image = line_detector(
image, detect_resolution=384, image_resolution=1024
)
prompt = "Painting of a woman, oil painting"#Standard
prompt = "a woman, Chinese ink painting,liujiyou"#"lora/liujiyou-SDXL.safetensors"
prompt = "a portrait painting of a woman, v0ng44g p0rtr14t"#"lora/v0ng44g, p0rtr14t.safetensors"
prompt = "a portrait of a woman wearing glasses, 3D, cartoon, pixar, disney "#"lora/Samaritan 3d Cartoon SDXL.safetensors"
negative_prompt = "text, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured"
gen_images = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
image=image,
num_inference_steps=30,
adapter_conditioning_scale=0.8,
guidance_scale=7.5,
).images[0]
gen_images.save("example.jpg")
Best regards,
sayakpaul
commented
11 months ago I can reproduce it here: https://colab.research.google.com/gist/sayakpaul/1989dcb2428719e6be5850ade1c23eb6/scratchpad.ipynb
Feel free to open a PR :-)
Here is our contribution guidelines: https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md
Describe the bug
Hello,
Thank you for this useful library. I have a small problem, I managed to use the code to generate images using SDXL with t2i, then an Image with a Lora. But for some reasons not with both. When I load the lora using the StableDiffusionXLPipeline It does work but not when I do so with this : StablediffusionXLPipeline I already had a similar problem with controlnet and Lora. Here is the issue : #4664. I followed somehow similar steps, and managed to fix it on my local install. I have added this bit of code at the end of pipeline_stable_diffusion_xl_adapter (Without forgetting the import os) :
I would love to open a PR and contribute to your code, the files don't look exactly the same (the one I have installed, and the one I cloned).
And I hope that this will help improve the library :).
Reproduction
Logs
System Info
diffusersversion: 0.21.0.dev0Who can help?
@sayakpaul