Borders missing in edited images

[NicolasWinckler]

Hi!

Thank you for sharing your work. I have an issue in the generation step. When using Instruct pix2pix, there are some parts of the original image that are missing in the edited image: the top/left/right/bottom border have been removed. It looks like image has been cropped or zoomed in or have some kind of padding effect. Is it something that you have observed and have taken into account in your code when blending? Or is it some parameters of instructpix2pix that i have missed and need to set?

Below is an example of input image (from BDD100k) and edited image with "add some fog" prompt.

input image:

edited image:

We can see for example the second tree on the right of the original image is missing, and same on the top/left corner: a part of the road sign is cut off.

I have used the following command for the generation:

python edit_cli.py --input $INPUT \
                    --output $OUTPUT \
                    --steps 100 --resolution 512 \
                    --edit "add some fog"\
                    --ckpt $CKPT \
                    --cfg-text 7.5 --cfg-image 1.5 --seed 58912

where edit_cli.py is slightly modified to rescale back the edited image to the original resolution with:

edited_image = ImageOps.fit(edited_image, (W, H), method=Image.Resampling.LANCZOS)

With W and H the width and height of the original input image. I have the same issue without rescaling the edited image, and on all examples.

Thanks for your help

EDIT: to reproduce my problem i also give the name of the BDD100k image: 0000f77c-6257be58.jpg (from trainset)

[NicolasWinckler]

Hi,

It turned out that preprocessing the input image with:

input_image = ImageOps.fit(input_image, (width, height), method=Image.Resampling.LANCZOS)

actually performs some cropping when the input does not have squared resolution, even with parameter bleed=0.0. I could handle this problem by padding the original image this way:

def imageOps_fit(im, width, height):
    im.thumbnail((width, height), Image.Resampling.LANCZOS)
    delta_w = width - im.size[0]
    delta_h = height - im.size[1]
    padding = (delta_w // 2, delta_h // 2, delta_w - (delta_w // 2), delta_h - (delta_h // 2))
    # Add padding to the image
    im = ImageOps.expand(im, padding)
    return im

and could get back the borders: