Can Differential Diffusion help improve the inpainting model?

[Sanster]

Thank you for your work and open-source code! I would like to ask a question, Can Differential Diffusion help improve the inpainting model? e.g runwayml/stable-diffusion-inpainting

[exx8]

It is expected it should. So despite both of the techniques seem to overlap there might be use cases it was beneficial to use an inpaint checkpoint with diff diff. The previous inpaint inference algorithms did not allow specifying gradual change; passing non-binary maps to the model also seems to fail. This is problematic, because a sharp transition in an image, might look unrealistic, especially for merging different contexts. We showed this problem of sharp transition, in the paper while we tested the inpaint checkpoint of SD2. While in the paper, we used the general checkpoint for our "soft inpainting" technique, one might prefer to use the inpaint checkpoint. The general checkpoint seems to prefer greater change, while usually inpaint checkpoints put an emphasis on creating a more cohesive image. Implementation wise: For most inpaint checkpoints both the mask and a censored image must be passed, but beside this the algorithm doesn't change. The key idea remains the same, a binarized mask is calculated at each step which is dependent on the current time step. So now the inpaint model accepts beside the previous inputs, a different censored image at each timestep, and a different mask (which the general model doesn't accept as an input).

[Sanster]

Thank you for your detailed reply!

[loboere]

Can you share the pipeline to use it with the runway inpaint model please?

[exx8]

Can you share the pipeline to use it with the runway inpaint model please?

Sorry for the late response, I need to think about a release plan. In the original paper we did not investigate the usage of inpaint models, as it seems redundant for its applications. However, I do recognize that there are use cases that can benefit from inpaint models.