search

TencentARC / BrushNet

[ECCV 2024] The official implementation of paper "BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion"
https://tencentarc.github.io/BrushNet/
Other
1.44k stars 120 forks source link

Does the Unet have inpainting ability? #55

Open CharlesGong12 opened 4 months ago

CharlesGong12 commented 4 months ago

Hi, I am reading your excellent work and have a question. Is the unet you use a classical T2I unet or an inpainting unet?

RunpuWei commented 4 months ago

It seems to load the sd-base weights

CharlesGong12 commented 4 months ago

@RunpuWei Thanks a lot but actually I mean whether the unet branch has inpainting ability before being equipped with Brushnet branch.

RunpuWei commented 4 months ago

@RunpuWei Thanks a lot but actually I mean whether the unet branch has inpainting ability before being equipped with Brushnet branch.

I believe the sd-base model has the capability for inpainting (e.g., Repaint: https://github.com/huggingface/diffusers/blob/bbd2f9d4e9ae70b04fedf65903fd1fb035437db4/examples/community/stable_diffusion_repaint.py), although the results were relatively poor. Additionally, you can enhance it through any fine-tuning method, such as ControlNet, BrushNet, or adding 5 additional input channels (sd-inpaint model) like runwayml.