Image to Image: Regenerate Only Masked Area Deepfries rest of Image

[RubenWB]

What happened?

TLDR: Regenerate Only Masked Area gradually Affects and Corrupts Image outside of Mask Area- with shifts in image hue and introducing artefacts and noise

I made a quick test to reproduce the issue: Image Info: Delicious Egg seed:1 width:512 height:512 steps:50 cfg_scale:7.5 sampler:k_lms and then in Image to Image Setting the mask mode to regenerate only masked area and making sure not to mask anything (you can also reproduce by masking the same spot, but drawing nothing allows for reproducing the issue faster since it then technically should have a perfectly preserved output)

Input Image:

Output after 8 Iterations:

Happy to provide more info or Sample Images And also sorry if I butcher any terminology or use any terms incorrectly

Basically the parts of the Image I don't want to change, gets deep fried when I generate new images

Version

0.0.1 (Default)

What browsers are you seeing the problem on?

Chrome

Where are you running the webui?

Windows

Custom settings

Using Default Settings

Relevant log output

Technically N/A since Log does not give usable info:

Iteration: 1/1
[MemMon] Recording max memory usage...

100%|████████████████████████████████████████████████████████████████████████████████| 50/50 [00:04<00:00, 10.16it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.22it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.66it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.67it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.68it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.55it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.68it/s]
[MemMon] Stopped recording.

[MemMon] Recording max memory usage...

Iteration: 1/1
100%|████████████████████████████████████████████████████████████████████████████████| 38/38 [00:03<00:00, 10.66it/s]
[MemMon] Stopped recording.

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[codedealer]

When processing an image it heavily shifts colors towards blues. I always assumed it's needed for SD to work. Would be cool if someone can clear this up.

[RubenWB]

Yeah, I've experienced that as well and up till now mostly just been manually correcting the hue afterwards in krita, but this seems to be affecting the area that is marked by the mask to be not affected? I don't really fully understand how it works under the hood, nor will I pretend to, but from a logical standpoint the area I mark with the masking tool to not get affected gets affected anyway?

Almost as if that area is being processed somehow unintentionally

I'd Imagine the mask just makes a cutout of the generated image and then overlays it on top of the original image and maybe pulls a few tricks to make it blend nicely with the background

Issue is it is almost as if the blank area does still contain noise data and it is mixing with the original Image?

Could also be that the original image is being overlayed with the original image + masked result and in that way is affecting the new resulting output?

[Dawgmastah]

This is related to issue 153 on the main repo: https://github.com/hlky/stable-diffusion/issues/153

That thread offers more info on which branch last worked without the problem

[codedealer]

I'm not too miffed about mask being a bit unsteady, as I understand it the current approach allows for better blending with the rest of the image.

I'm specifically annoyed by shifting to blues/magentas.

[Goldenkoron]

This happens even in crop made from my testing, pretty much any img2img generation. The bug first appeared in #308 on 8/30, I have another bug report active right now on it.

[TingTingin]

The quality of the image drops because of this https://github.com/hlky/stable-diffusion-webui/pull/308 there needs to be some kind of change as people at https://github.com/hlky/stable-diffusion/issues/153 were talking about the issues with this hlky hasn't really given any call on this yet

This change drops quality of the entire image to try to improve blending it was an oversight by the dev that people would make more than one change to the image https://github.com/hlky/stable-diffusion/issues/153#issuecomment-1233141374

There was a potential fix here that never got merged because hlky never commented https://github.com/anon-hlhl/stable-diffusion-webui/commit/a38dfe252b47d03fd7db06adb314d67527e54df2 https://github.com/hlky/stable-diffusion/issues/153#issuecomment-1233225883

[Dawgmastah]

The quality of the image drops because of this #308 there needs to be some kind of change as people at hlky/stable-diffusion#153 were talking about the issues with this hlky hasn't really given any call on this yet

This change drops quality of the entire image to try to improve blending it was an oversight by the dev that people would make more than one change to the image hlky/stable-diffusion#153 (comment)

There was a potential fix here that never got merged because hlky never commented anon-hlhl@a38dfe2 hlky/stable-diffusion#153 (comment)

I tried implementing the fix, didn't get it to work (behavior stayed the same and I couldn't see the gradio option)

[RubenWB]

I'm not fully sure how Masking works and if the old worse version worked exactly like this, but from reading the discussions from the post above, maybe in favour of better blending, maybe the process can be as follow?

If the mask has a opacity value where 1 a fully white pixel and 0 is a fully black pixel maybe the blending where image quality is sacrificed can be where the value is less than 1 and greater than 0? Potentially this logic could produce a better blended image? so where the mask is at 1 it is the replacement image and 0 is the original image and the values in between can be the less optimized blended image

You can potentially finetune the settings so it does not have to be exactly 1 or 0 and activate logarithmically or a custom setting since it might produce some hard edges?

At least maybe this could be a potential way to have the best of both solutions?

[Goldenkoron]

Is img2img fixed now after latest repo merge? I don't want to take risk and update from #307 until I know for sure.

[TingTingin]

Don't believe the repo is in a downloadable state right now maybe give it a day or too for things to settle

[Goldenkoron]

This is definitely not fixed still....

I updated to a new version because I thought I saw that color correction was implemented.

[Dawgmastah]

There were a ton of changes today to img2img Is this problem still present?

[lzardy]

Still present on gradio master.

[xaedes]

I added an option in the frontend to restore the image based on mask in the same way it was done before the change that resulted in image degradation.

https://github.com/xaedes/stable-diffusion-webui/tree/mask-restore-665

It will still use the masked sampling, but restore the image afterwards - but only IF the option is enabled.

[Dawgmastah]

@xaedes Nice I will take a look. Question though, does this new implementation option revert the mask to how it was before? Or is the overall problem with loopback deepfrying still present?

If so, is there a way to have an option for fully having the previous implementation for loopback applications?

[vylcount]

this issue still happening to me, im on latest update to the repo it heavily distorts the image after awhile

[codedealer]

this issue still happening to me, im on latest update to the repo

The PR hasn't been merged yet. You can track the progress here #898

[xaedes]

@xaedes Nice I will take a look. Question though, does this new implementation option revert the mask to how it was before? Or is the overall problem with loopback deepfrying still present?

If so, is there a way to have an option for fully having the previous implementation for loopback applications?

@Dawgmastah

Before the deepfry change the mask was used to blend the generated image with original image. Thus the (un)masked parts where taken entirely from the original image -> no image degradation.

The new option brings back this behaviour. To generate the image it still uses the improved masked sampling that was introduced in the deepfry change. But the result will be blended with the original image according to the mask.

In my tests I could see no more deepfrying, also tested it with loopback.

Imho it looks a bit worse with the option enabled, as the generated part has slightly different colors overall - (maybe color correction like it is already used in loopback would also help here?. The regenerated part looks more "cheaply pasted in", so to say. [edit: just tested it together with color correction from #936 and it looks much better] But it absolutely ensures that the rest of the image stays intact. By increasing the mask blur it will blend more smoothly on the contours, that helps - but only so far.

Currently I use it for lots of masked regenerations in succession (maybe with mask or prompt change) and then make a final img2img pass over the whole image with very low denoising strength to make it just blend it all nicely together without changing too much.