AbdullahAlfaraj / Auto-Photoshop-StableDiffusion-Plugin

A user-friendly plug-in that makes it easy to generate stable diffusion images inside Photoshop using either Automatic or ComfyUI as a backend.
MIT License
6.76k stars 509 forks source link

Lasso mode doesn't inpaint at full resolution #398

Open Jonseed opened 1 year ago

Jonseed commented 1 year ago

When using lasso mode and "inpaint at full res", the resolution gets reduced to the base resolution dimension square around the shape. It doesn't maintain the resolution set with the width/height sliders.

So, for example, if I have ratio mode selected, and 768 px as base, and then make a wide selection, the width/height sliders say something like 2171 x 768 px. That is correct, and that is the resolution that I expect the generation to be made at (or perhaps slightly bigger to account for mask expansion, blur, inpaint padding, and offset). But then when it generates the area, the size of the generation is changed to a square, and is generated at only 768 x 768px. Shouldn't it generate the area at the full resolution of 2171 x 768 px (or larger), especially if "inpaint at full res" is checked? And why does it change it to a square?

AbdullahAlfaraj commented 1 year ago

There are two distinct issues:

1) The sliders function differently in the inpaint mode at full resolution. They resize the masked area, but the final resolution will always match the size of the inital image. It’s counterintuitive, but this is how it works in Automatic1111 as well.

2) The second issue is that if you use the lasso mode with the ratio or precise selection mode, you will always end up with a square image. This is because that’s the ratio at which SD inpaint performs best. image

Jonseed commented 1 year ago

In Auto1111, with the inpaint area set to "only masked," the resolution of the inpainting generation is whatever is set as width and height, resizing the masked area to that full resolution, generating the image, then resizing the generated image back down to merge into the initial image.

In the plugin, if I use the regular inpaint mode (not lasso mode) at "full res," it works as I expect. If I paint a mask and put a rectangular selection around it, say 636 wide x 364 height px on the image, and set inpaint to ratio mode with 768 px as base resolution, then the generated inpainting is 1341 wide x 768 height px. It resizes the masked area to the full res, and generates a full resolution 1341 x 768 px image smart object, which is resized down to fit in 636 x 364 px in the initial image. It is not square, and it is the full resolution as set with the sliders. Why doesn't lasso mode work similarly, just simplifying the masking method to lasso selection instead of painting/rec-selection?

Currently with lasso mode, that same mask with sliders set to 1341 x 768 px would be generated as a 768 x 768 px square around the area. Width of the mask is only generated as 768 px, not 1341 px. The height of the masked area is also reduced since it makes the larger dimension the base resolution, the height of the masked area I want to inpaint gets reduced proportionally to generate only 439 px within the 768 px square. It isn't inpainting at full resolution, either width or height. The sliders say 1341 width x 768 height px, but it is inpainting the mask to generate only 768 width x 439 px, which is just half the resolution intended.

AbdullahAlfaraj commented 1 year ago

I’m marking this as a suggestion to ensure I remember it. Currently, I’m fully occupied, but when I return to this feature, I will introduce an option to switch between a square ratio or the ratio of the smallest bounding rectangle.