Details of Diffusion Model Training based Paint By Example

[EricShow]

Diffusion Model 1、尝试将warped-cloth与Image-agnostic制作成您文中的样子进行训练，对应于Reconstruction Branch，遇到了一些问题： a. Paint-By-Example. Pretrained Model VAE编解码会导致脸部变形的问题 期待您的回复

[Limbor]

Hi @EricShow, thank you for your interest in our work. The problem of facial deformation caused by VAE does exist, but we did not pay too much attention to it. When inference, we chose to paste back the area outside the inpainting mask directly. Of course, there are also some ways to solve this problem. Simply, we can increase the image resolution, and the loss caused by VAE at 1024 resolution will basically not affect the image quality. You can also refer to Asymmetric_VQGAN to retrain the decoder separately to solve this problem.