search

lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
29.13k stars 2.63k forks source link

What's the "hint" image used for training the inpaint controlnet model? #424

Open zjintheroom opened 1 year ago

zjintheroom commented 1 year ago

In the tutorial, it is mentioned that a "hint" image is used when training controlnet models. but for the inpainting process, there's a original image and a binary mask image. I would like to know that which image is used as "hint" image when training the inpainting controlnet model? Thanks in advance!

geroldmeisinger commented 10 months ago

I'm also interested for more information on this please! What I found out so far is you can use images with alpha channels for training. I believe the inpainting model was trained with RGB+A. also see here https://github.com/lllyasviel/ControlNet/issues/271

donggoing commented 8 months ago

Anythings new? I would like to finetune the SD2.1 for multicontrol (Inpainting + xx). But i am not sure how the inpainting been trained, if i just need to change the hint "image" to give the masked-image and mask info? Or it there something more i need to do?

geroldmeisinger commented 8 months ago

also see here https://github.com/lllyasviel/ControlNet-v1-1-nightly/issues/89