BUG in stable_diffusion.image_encoder.ImageEncoder creates edge padding on encoded image.

[bobqywei]

This was found when debugging: https://github.com/keras-team/keras-cv/issues/1158 and is likely the root cause.

The pretrained image encoder adds zero padding to the top and left edges of the image when encoding. This can be reproduced and seen as follows:

The original image is transformed via out = decoder(encoder(img)) resulting in an output image that is visually identical other than the gray border at the top and left edges. Furthermore, we can likely assume that there's no problem with the decoder since that is being used in the standard StableDiffusion.text_to_image function which does not have this issue visually.

[LukeWood]

I've confirmed that the decoder does in fact create an 8px padding on the top left which is incorrect:

https://colab.research.google.com/drive/1XR7Rs-mLZBeQI3_9c7p3lr_LJ65bJjJD?usp=sharing

[LukeWood]

more info at https://github.com/keras-team/keras-cv/issues/1158

[LukeWood]

Notes for Ian/Francois/Myself:

• we should definitely iron this out, it creates an 8px offset on the top-left of EVERY image passed into the model
• decoder(encoder(image)) has the 8px boundary
• images diffused from noise I don't think have this, so I think it must be coming from the ImageEncoder

[ianstenbit]

Re-opening as I rolled back the fix, which caused a separate issue with inpainting.

[LukeWood]

Asked in https://github.com/divamgupta/stable-diffusion-tensorflow/issues/71

[LukeWood]

No borders in https://colab.research.google.com/drive/1Bf-bNmAdtQhPcYNyC-guu0uTu9MYYfLu?usp=sharing