Open cadaeix opened 1 year ago
Thanks for the issue @cadaeix,
I can reproduce it. Here a simple repro based on your example:
import requests
import numpy as np
from PIL import Image
from io import BytesIO
from diffusers import StableDiffusionImg2ImgPipeline, DPMSolverSinglestepScheduler
device = "cuda"
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
).to(device)
pipe.scheduler = DPMSolverSinglestepScheduler.from_config(pipe.scheduler.config)
url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg"
response = requests.get(url)
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((768, 512))
strength = 0.15
images = pipe(
prompt="a fantasy landscape",
negative_prompt=None,
image=init_image,
strength=strength,
num_inference_steps=15,
guidance_scale=10,
num_images_per_prompt=1
).images
image = images[0].save(f"fantasy_landscape_{strength}.png")
This looks definitely like a bug and we should try to fix it. Also cc @LuChengTHU in case you have any idea what might be going on here.
@cadaeix just a quick tip, it's not recommend to do:
pipe.scheduler = DPMSolverSinglestepScheduler()
as this will load the wrong config instead you should do:
pipe.scheduler = DPMSolverSinglestepScheduler.from_config(pipe.scheduler.config)
I'll try to allocate time to solve this soon except @LuChengTHU beats me to it :-)
Haven't found time yet to look into it. Will try to do so soon.
Still on my TODO - @williamberman could this be related to: https://github.com/huggingface/diffusers/pull/2969 ?
@patrickvonplaten
I look at code and found some moments:
1) Wrong variable name in set_timesteps
:
self.orders = self.get_order_list(num_inference_steps)
correct:
self.order_list = self.get_order_list(num_inference_steps)
2) When we select timesteps for img2img we skip some
https://github.com/huggingface/diffusers/blob/v0.16.0/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L522
Now first called timestep are not first scheduler timestep, due to this order
for this timestep could be any value. But to work properly orders need to be called in order from 1 to self.config.solver_order
.
I not sure how properly it should be fixed, so just leave info here: 1) limiting order by lastly executed order
order = min(order, self.max_prev_order + 1)
self.max_prev_order = max(self.max_prev_order, order)
or 2) by calculating order dynamically like:
def get_next_order(self, step_index):
self.current_order += 1
if self.config.lower_order_final and step_index == len(self.timesteps) - 1:
# make it looks like [..., 3, 1, 2, 1]
if self.config.solver_order == 3 and self.current_order == 3:
self.current_order = 1
if self.current_order > self.config.solver_order:
self.current_order = 1
return self.current_order
Upd: after some thoughts i think second option should be correct, so if no other thoughts will be here create a PR tomorrow
Thanks a lot @StAlKeR7779, good catch!
Fixing the bug here: https://github.com/huggingface/diffusers/pull/3413
Posting pictures of #3413 here:
0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
This looks good to me now!
Describe the bug
When using
StableDiffusionImg2ImgPipeline
, theDPMSolverSinglestepScheduler
schedule, with certain img2img strength values, produces an error resulting from previous model outputs not being available.Settings:
Inference steps: 15 Strength: 0.1, 0.3, 0.4, 0.45, 0.55, 0.7, 0.8, 0.95
Excerpt from the stack trace:
Reproduction
Logs
Note: First two strength values tested, 0.0 and 0.05, fail because of a different issue (#1867)
System Info
diffusers
version: 0.11.1