openvinotoolkit / openvino_notebooks

📚 Jupyter notebook tutorials for OpenVINO™
Apache License 2.0
2.3k stars 791 forks source link

Incorrect results for pix2pix model #819

Closed adrianboguszewski closed 1 year ago

adrianboguszewski commented 1 year ago

It looks like the pix2pix model doesn't give me any meaningful results. Whatever I prompt, I get a result that isn't even close to what I asked for. Look at the below: Screenshot from 2023-02-08 18-25-10 Screenshot from 2023-02-08 18-23-34 image

I tried different seeds, a number of steps and strengths. @eaidova Is this expected? It doesn't look like depicted in README.

eaidova commented 1 year ago

@adrianboguszewski do you mean 231 or image2image in 225? jugging mention of strengths, it is stable diffusion. in image2image feature prompt used as additional detalization of image. It can not replace some particular details because it does not know image semantic,image used only for skping first step with random generation. Model uses image geometry and colors for preserving shapes, it does not know which area should be kept and which modified. Also problem in prompt what you provide, as it does not know image content, it can not change or turn something. Instead instruction what it should to do, it expects description how final image should looks like e.g. Turn him to cyborg -> a photo of cyborg

image

it is the result for strengths 0.8

but your prompt is valid for 231, this model was fine-tuned to understand image modification instructions and understand image semantic: Turn him into chicken

adrianboguszewski commented 1 year ago

@eaidova Oh yeah, you're right. I mixed up 2 notebooks. I was using 225, thinking that it was 231. When I tried the same prompt with 231, it worked.

raymondlo84 commented 1 year ago

image This is nightmare inducing...