search

mikonvergence / ControlNetInpaint

Inpaint images with ControlNet
MIT License
344 stars 29 forks source link

Did you retrain the ControlNet for the SD-inpainting backbone? #3

Closed ThibaultGROUEIX closed 1 year ago

ThibaultGROUEIX commented 1 year ago

Hi! Thank you for this repo.

I did not understand if you retrained ControlNet using the SD-inpainting backbone, or if you copied over the weight that were trained for the regular SD-backbone by ControlNot authors, and those weights somehow work on the SD-inpainting backbone as well?

Thank you very much, Thibault

mikonvergence commented 1 year ago

Hello @ThibaultGROUEIX! Thank you!

No new models, just the regular ControlNet + SD-inpainting backbone, and the weights indeed somehow work. Naturally, the SD-inpainting and regular SD have some "shared history", but it seems that these dynamics, especially in the context of modulating the features with ControlNet have not yet been thoroughly studied.

As for the "somehow", I have empirically found the scale of the ControlNet injections to be an important factor in achieving successful results, as demonstrated in this example shared on my twitter: https://twitter.com/mikonvergence/status/1635581057494917120