search

invoke-ai / InvokeAI

InvokeAI is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, supports terminal use through a CLI, and serves as the foundation for multiple commercial products.
https://invoke-ai.github.io/InvokeAI/
Apache License 2.0
22.77k stars 2.35k forks source link

Image2Image fit option blends pixels when scaling down which ruins mask edges #625

Closed mavispuford closed 1 year ago

mavispuford commented 1 year ago

Describe your environment

Describe the bug When passing a masked image with "fit" enabled, it scales down the image and blends the pixels in the process. This causes artifacts near the edges in the output. Maybe there should be a "nearest neighbor" scaling option?

To Reproduce Images: Source image: person-girl-woman-hair-photography-portrait-108386-pxhere com (1) Large masked image (2496x1664): Mask-637989379517936043 Small masked image (960x640): Mask-637989379593202687

Steps to reproduce the behavior:

  1. Enter a prompt ("a frowning woman") using the large masked image above, default settings (50 steps, strength .75) but with width at 960 and height at 640
  2. Notice the seams around the masked area
  3. Now try it with the small masked image
  4. Notice that it is seamless

Expected behavior There should be no seams/artifacts in the masked area. The resulting image should be seamless

Screenshots Large image output: 000374 3113900562-artifacts

Small image output: 000375 820978995-seamless

Additional context Could you add a nearest neighbor/non-filtered scaling option so the pixels don't get blurred? I like the "fit" option because I don't have to resize images on my own, so it would be nice if it handled masked images well.

lstein commented 1 year ago

I see the problem. Would the PIL.Image.Resampling.NEAREST resampling filter be the best choice here?

mavispuford commented 1 year ago

@lstein Seems like that would do it. There's definitely a quality difference when using nearest neighbor vs the other methods, though, so it'd probably be best as a parameter. Either that or I suppose you could switch to nearest neighbor when transparent pixels/inpainting is detected.

lstein commented 1 year ago

I'll give it a try as soon as I get access to my development environment again. (It's an HPC cluster down for maintenance over the weekend). In the meantime, if you'd like to swap the nearest resampling in, just open the file ldm/dream/image_util.py, look for the line resized_image = self.image.resize((rw,rh),resample=Image.Resampling.LANCZOS) and change the resampling method to Image.Resampling.NEAREST

mavispuford commented 1 year ago

Seems to work for me. Here's my output using the large masked image with 960x960 chosen as the size: a_sad_woman-1228974340-637991001671014705-0

lstein commented 1 year ago

This should be fixed now.