Closed sunhaozhepy closed 7 months ago
Even if the reproducible code snippet doesn't matter, consider providing a more complete and minimal reproducible snippet. In your case, there are multiple unknowns such as: get_landmarks()
, ./lkw-lora-sd-v1-5
, etc.
So, I recommend providing something we can reproduce from our official code examples such as the one from here: https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet#diffusers.StableDiffusionControlNetImg2ImgPipeline
Cc: @yiyixuxu
I see. Why not try this official pipeline, since I managed to reproduce my error on it:
from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch
from torchvision.transforms import Resize
import cv2
from PIL import Image
# download an image
image = load_image(
"https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
transform = Resize((631, 512))
image = transform(image)
np_image = np.array(image)
# get canny image
np_image = cv2.Canny(np_image, 100, 200)
np_image = np_image[:, :, None]
np_image = np.concatenate([np_image, np_image, np_image], axis=2)
canny_image = Image.fromarray(np_image)
# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
# generate image
generator = torch.manual_seed(0)
image = pipe(
"futuristic-looking woman",
num_inference_steps=20,
generator=generator,
image=image,
control_image=canny_image,
).images[0]
# for me output is (512, 624)
print(image.size)
Hmm from a quick glance seems like we fix the height and width here:
Hi! Is there any update on this?
By the way, I've tested on the default pipeline and found that even if we specify the height and the width, it'll still be ignored:
from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch
from torchvision.transforms import Resize
import cv2
from PIL import Image
# download an image
image = load_image(
"https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
transform = Resize((631, 512))
image = transform(image)
np_image = np.array(image)
# get canny image
np_image = cv2.Canny(np_image, 100, 200)
np_image = np_image[:, :, None]
np_image = np.concatenate([np_image, np_image, np_image], axis=2)
canny_image = Image.fromarray(np_image)
# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
# generate image
generator = torch.manual_seed(0)
image = pipe(
"futuristic-looking woman",
num_inference_steps=20,
generator=generator,
image=image,
control_image=canny_image,
height=631,
width=512
).images[0]
# for me output is still (512, 624)
print(image.size)
@sunhaozhepy correct me if I'm wrong, but width and height of images generated with Stable Diffusion should always be multiples of 8. Your height of 631 is invalid, therefore I assume it will fall back to 624.
Please follow this link to understand better what is saying about @andypotato: solution
That makes sense! Thank you all for your help, @andypotato @hi-sushanta @sayakpaul, I'll probably change the resolution of my original images.
Describe the bug
I wish to turn an image of resolution 512x631 into a new image of the same resolution. It is vital that they should have the same resolution, otherwise I can't use them in downstream tasks. And the StableDiffusionControlNetImg2ImgPipeline keeps returning images of resolution 512x624, regardless of whether I tell the model to output 512x631 when calling the pipeline.
Reproduction
Logs
No response
System Info
diffusers
version: 0.24.0Who can help?
@sayakpaul @patrickvonplaten