AbdullahAlfaraj / Auto-Photoshop-StableDiffusion-Plugin

A user-friendly plug-in that makes it easy to generate stable diffusion images inside Photoshop using either Automatic or ComfyUI as a backend.
MIT License
6.77k stars 508 forks source link

Hard edge in lasso mode inpainting? #374

Open Jonseed opened 1 year ago

Jonseed commented 1 year ago

Loving the plugin. But I'm having an issue with lasso mode inpainting. I always get a hard edge on the rectangular area surrounding the masked area, no matter what I set the mask blur, mask expansion, or inpaint padding. The lasso mask seems to run into the hard rectangular edge surrounding it. This isn't a problem in regular inpainting mode, as you can set the rectangular area to be relatively bigger than the masked area. Is there a way to control how much bigger the rectangular area is than the lasso mask?

cdmusic2019 commented 1 year ago

You can design fuzziness and expansion values on the plugin panel. Moreover, the "inpainting mode" cannot use the XL model; it is recommended to try the "inpainting" model instead.

Jonseed commented 1 year ago

It doesn't matter what I set the mask blur and expansion to, it still shows a hard edge where the rectangle meets the edge of the lasso mask area. I'm not using an XL model, and I am using an inpainting model.

Jonseed commented 1 year ago

Screenshot 2023-09-18 091629 Screenshot 2023-09-18 091517-2 For example, you can see here that I lasso'd the cat's face to turn it into a lion, and I get hard edges at the top/bottom and left/right sides where the lasso meets up with the rectangular boundary. I've tried changing the mask expansion, blur, and inpaint padding, and none of those seem to fix it. The rectangular boundary around the lasso seems to just be too close, and I don't know how to avoid that.

Jonseed commented 1 year ago

Increasing the mask expansion would just make the hard edge worse. This is what the mask looks like after increasing expansion to 30. Screenshot 2023-09-18 093330

Jonseed commented 1 year ago

It seems the plugin is placing a rectangular bounding box directly around the original lasso area, without regard for the mask blur or mask expansion values. So if you add blur, or expansion, then it exceeds the boundaries of that bounding box, and causes a hard edge. But even with 0 mask blur, and 0 mask expansion, the mask still causes a straight hard edge on top/bottom, and left/right. Screenshot 2023-09-18 100921 Screenshot 2023-09-18 100913

cdmusic2019 commented 1 year ago

This is my setting, you refer to it, may be able to solve your problem! 2023-09-19_090951 2023-09-19_092041 2023-09-19_092543

AbdullahAlfaraj commented 1 year ago

Hey there, @Jonseed and @cdmusic2019! The issue is due to the mask being a bit too large and coming into contact with the selection area’s border.

In the normal inpaint mode, you can solve this by either reducing the size of the mask or expanding the selection box.

As for the lasso mode, I’m planning to introduce a slider option that will allow you to adjust/offset the selection box.

Jonseed commented 1 year ago

@AbdullahAlfaraj The lasso offset in 1.3.3 seems to have fixed this. Thank you!

But now the mask blur doesn't seem to work, or only works about half the time. I have mask blur set, but the generation comes back with a hard edge on the entire lasso'd shape mask. Do you know why that might be?

AbdullahAlfaraj commented 1 year ago

@Jonseed can you provide an example, and screenshot of the settings including thumbnails of initial image and the mask