Stability-AI / generative-models

Generative Models by Stability AI
MIT License
24.55k stars 2.73k forks source link

Question: Inpainting model for SDXL #45

Open Markus-Pobitzer opened 1 year ago

Markus-Pobitzer commented 1 year ago

Dear Stability AI Team

Thanks for an other great release!

Do you plan on releasing an inpainting model for SDXL too?

aleksmirosh commented 1 year ago

I will much appreciate any news about inpainting too!

mlconnor commented 1 year ago

+1

jiqizaisikao commented 1 year ago

+1

wangqyqq commented 1 year ago

Waiting for the inpainting model for SDXL, too!

bghira commented 1 year ago

it will be v-prediction and zero terminal SNR, most likely. i think that will come with 1.1.

yang-fei commented 1 year ago

when to release the inpainting model? there will be much use of the inpainting model

jaidevshriram commented 1 year ago

Thanks @bghira! Is there a timeline for this? Also, is the SDXL-base model v-prediction too or is that also in the 1.1 release?

Markus-Pobitzer commented 1 year ago

SDXL can already be used for inpainting, see: https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#inpainting

To me it seems like they are using the base model for inpainting.

Will there be a fine tuned version like stable-diffusion-2-inpainting?

landmann commented 1 year ago

Anyone notice inpainting with a mask alters the image the mask says it should not be altering? Is this expected?

Markus-Pobitzer commented 1 year ago

@landmann If you are referring to small changes, than it is most likely due to the encoding/decoding step of the pipeline. We bring the image in a latent space (containing less information than the original image) and after the inpainting we decode it back to an actual image but in this process we are losing some information (the encoder is lossy as mentioned by the authors). This is expected behavior.

Workaround can be found here: https://github.com/huggingface/diffusers/issues/3514#issuecomment-1565221994

landmann commented 1 year ago

Haha good ol ctrl+c ctrl+v - love it.

shreyassks commented 1 year ago

Is it possible to finetune SDXL 1.0 base model for inpainting?

I tried to do that but I get a error from UNET that it expects the input to have 4 channels but it got 9 channels (4 - input image, 4 - masked image, 1 - mask (B/W))

Am I doing anything wrong or should I wait for inpainting model release?

bghira commented 1 year ago

if you're feeling adventurous you can modify the pix2pix finetuner from Diffusers project.

it already uses 8 channel input.

nasy commented 1 year ago

+1

JohanHuynh0130 commented 1 year ago

+1 im waiting for inpainting model of SDXL

JohanHuynh0130 commented 1 year ago

SDXL INPAINTING MODEL has been out!

yjhong89 commented 1 year ago

Is there any in-paint configs for SDXL 1.0 (base)?

gel-crabs commented 10 months ago

A yaml file for SDXL-inpainting is desperately needed.

kalle07 commented 10 months ago

yes pls !

kalle07 commented 10 months ago

new inpaint model https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors/blob/main/sd_xl_base_1.0_inpainting_0.1.safetensors

its also HALF so 16GB RTX nvidia doesnt work !

wangqyqq commented 10 months ago

as i mentioned in webui post, this sdxl-inpaint model can work in 16-bit float and you don't need to appy --no-half-vae or --no-half. you can download the model, merge the code in pull request https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14390 and try it!