zero01101 / openOutpaint

local offline javascript and html canvas outpainting gizmo for stable diffusion webUI API 🐠
MIT License
499 stars 43 forks source link

Border with img2img erroneous #194

Closed grummelgrisgram closed 1 year ago

grummelgrisgram commented 1 year ago

What happened?

The border seems to drastically influence the generated content, even the one not touching the border. For more detail, see below.

Steps to reproduce the problem

See below.

What should have happened?

Border should not influence the content, the content should still be unrelated to source image with "Denoising Strength" parameter set to 1. In general, I should still be able to influence the content, it should not (nearly) only be influenced by the present image if a border is set. See below for more detail.

Commit where the problem happens

Current state, a0e7608.

What platforms do you use to access openOutpaint?

Windows.

What browsers do you use to access the UI ?

Brave, same with Chrome.

Browser Extensions/Addons

n.a.

AUTOMATIC1111 webUI Commandline Arguments

--listen --enable-insecure-extension-access --xformers --api

Additional information

There seems to be a bug in the "Border Mask Gradient" feature in combination with img2img. I expect this to behave like the following:

a) If the border size ("Keep Border Size") is set to 0 and "Border Mask Gradient" is disabled, simply the whole cursor size should be updated. This works as expected as you can see here (with Denoising Strength set to 1 to get an "unrelated image"). Although it does not seem to be 100% unrelated... But at least no border is considered obviously:

Source image right before generating: image Result: image

b) If the border size ("Keep Border Size") is set to 64 and "Border Mask Gradient" is still disabled, the border is expected to not be updated. But the rest of the cursor should be updated, similar to case (a) where the whole cursor content was updated. This is not working as expected, see: image Please note, that the border is not touched as expected and the rest of the cursor is modified. But this is by no means a "Denoising Strength" of 1 here. Instead, it is only modified in a very minor way! I'd expect something similar to (a) here instead.

c) If the border size ("Keep Border Size") is set to 64 and "Border Mask Gradient" is enabled, a similar image as in (a) would be expected but the border should be faded to the original image. This is also not working as expected, see: image Also here, there are minor updates inside the cursor but with "Denoising Strength" set to 1, I'd expect a lot more.

Since I only changed the two border parameters ("Keep Border Size" and "Border Mask Gradient") between case (a) and (b) resp. (c), I think something is going wrong with the border calculation here...

zero01101 commented 1 year ago

so out of curiosity, would you pop open the debug info tab and enable the "toggle debug" setting and try that again with one of your non-working setups? it should download an image with both your image and the mask generated and sent to the API - if you'd be so kind as to crop those into two separate images and try them with equal settings in the standard A1111 img2img tab?

i completely agree with your expectation and premise, but it seems to operate this way in webUI itself :/

image

grummelgrisgram commented 1 year ago

Thanks for the quick response, zero01101 - and the marvelous openoutpaint nevertheless! You're right, I have the same problem there at the standard img2img. So, what can we do? Submit a similar issue at automatic1111 UI?

zero01101 commented 1 year ago

thanks :) the grand majority of the marvelousness is @seijihariki's doing but i appreciate the kind words :D also, your detail in your issue report is greatly appreciated

i presume A1111's repo would be where to submit the issue, but i genuinely don't know if it might be further upstream, like in the parent compvis stable diffusion or something :/

grummelgrisgram commented 1 year ago

Thanks :-)

I submitted the issue to automatic1111 repo: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7611

So this issue here could probably be closed I guess.

seijihariki commented 1 year ago

Hi! Hey, @zero01101 have been a bit offline due to traveling for work. Will be closing this issue, then!

@grummelgrisgram, if you have any more issues, we'll be glad to help you solve it.

zero01101 commented 1 year ago

Hey, @zero01101 have been a bit offline due to traveling for work.

welcome back! no worries at all of course and hope you got to go somewhere new and interesting at least :D i've been kinda swamped professionally here too so just basically dropping some garbage here and there when i get a chance ;)