search

kohya-ss / sd-scripts

Apache License 2.0
4.74k stars 798 forks source link

Inpaint not working on SDXL using sdxl_gen_img.py #735

Open FelixDeMan opened 1 year ago

FelixDeMan commented 1 year ago

When I try to generate images using: "python sdxl_gen_img.py --outdir "outputs" --ckpt stabilityai/stable-diffusion-xl-base-1.0 --steps 40 --strength 0.8 --prompt "A picture of a chair" --network_module networks.lora --network_weights test_output/test_output-step00001500.safetensors --image_path "inpaint-images/input/input0.jpeg" --mask_path "inpaint-images/input/input0-mask.jpg"" My images get weirdly distorted all over. --fp16 returns black images, but that apparently has something to do with the VAE. using --bf16 or the standard float32 as above, does not resolve the issue. I will add the mask and the init_image below, as well as the generated image.

im_20230808125017_000_57285212 input0-mask input0

kohya-ss commented 1 year ago

Thank you for reporting. Unfortunately, inpainting doesn't work with LoRA models. Because the mask image is used for regional LoRA if LoRA models are specified. I will update to be able to use both inpainting mask and reginal mask same time.

Currently, please merge the LoRA models to the base model in advance, and use it for generation.

FelixDeMan commented 1 year ago

Okay, thanks! I went back to rewriting the Diffusers training code for it to implement more functionality.

On another note; I would love to get in touch with you regarding a business query! You can reach out to me at felix(at)neo-locus.com, much appreciated!