[Extension]: Poor IMG2IMG outputs with FreeU and Dynamic Thresholding

[a2ZxgdqHeSs8F51cIyuK]

Issue Description

IMG2IMG outputs awful images that become worse and worse with every pass.

Let's go through the steps to reproduce. I choose a very basic prompt, a very basic negative prompt and a good, untouched model off Civitai (I tested several models and the issue was present on all of them) and a VAE. Here's our simple TXT2IMG output. This is just a basic, decent looking image to display the issue.

Now we take this image to IMG2IMG. Same prompt, same negative prompt, same model. I set it to resize 1 (keep the same size) and denoise 0.4. I run the image through 6 times, just to really show how bad it gets, but it's clearly a negative effect even after 1 pass. It looks like absolute garbage! The background became a smudged mess, the eyes went from clearly blue to a vague grey splotch, the clothes lost all detail, the hair also lost detail and the slight glow from the original picture turned into strange lights and smoke, ignoring my negative prompt apparently. It also looks a bit more realistic in proportions and expression, again ignoring the negative prompt. And the model, for that matter, this is an anime model.

I use IMG2IMG a lot in my normal workflow, this kind of behaviour simply did not happen before. Sure, an IMG2IMG can sometimes introduce errors but it's never done anything like this. In fact, I like to run IMG2IMG a few times so it can add or reinforce details, but now it's doing the opposite, taking away details with every pass. I also tried a few different settings on my many tests, activating or deactivating "brownian noise" and "discard penultimate sigma", changing sampler, changing the "Sigma algorithm" but they don't affect this behaviour.

UPDATE: I redid the test, but this time knocking down the CFG and deactivating FreeU and Dynamic Thresholding. Running 6 IMG2IMG passes has introduced some mistakes, but it hasn't destroyed the image like before. It has become a bit washed out though.

Let's try just FreeU with the 1.4 recommended settings, which is what I have been using for quite a while before the update 6 passes in, it's kept the details, but made some really bright lights all over the place.

Let's try just Dynamic Thresholding (bringing the CFG back up) 6 passes it and it looks like absolute garbage.

So it seems it's not an issue with WebUI itself, but rather new incompatibilities / bad behaviours when paired with FreeU and especially Dynamic Thresholding. FreeU seems to introduce some issues it wasn't generating in the old versions. Dynamic Thresholding on the other hand is absolutely demolishing the image. It almost feels like it's turning off the negative prompt (or even adding it to the positive) with the way the smoke turns up despite being in negatives. But they only seem to appear in IMG2IMG, TXT2IMG doesn't seem to have the same issues.

Version Platform Description

Version: app=sd.next updated=2023-11-07 hash=56cf80b8 Clean install, Windows, Firefox, Nvidia 4090.

Relevant log output

No response

Backend

Original

Model

SD 1.5

Acknowledgements

• [X] I have read the above and searched for existing issues
• [X] I confirm that this is classified correctly and its not an extension issue

[vladmandic]

great job with narrowing the problem down. note that freeu was introduced in the previous release and has not changed at all in this release. and dynamic thresholding is an extension, so you might want to check in their repo.

[a2ZxgdqHeSs8F51cIyuK]

I have been using both extensions together for quite some time though and they never caused this behaviour. FreeU since shortly after it became available for Stable Diffusion WebUI and Dynamic Thresholding for many months before that. I never used FreeU on its own on the previous version so I'm not sure if the smaller issues it causes were present already. Maybe Dynamic Thresholding (when it works) helps contain them? But Dynamic Thresholding I have absolutely used on its own for a ton of generations of varying styles, it has never acted like this. The earliest gen I have using it is from mid April. Should I open an issue with DT anyway, pointing to this one?

[vladmandic]

i believe you, i'm just saying that based on your results dynamic thresholding is causing last part of the problem (and yes, combination with freeu is making it worse) - and there is not much i can do about dynamic thresholding since its not my code.

[311-code]

Animatediff doesn't seem to move too much when sending to img2img and checking animatediff, it's overall 16 frames of the same thing. Not sure if I'm doing something wrong, just doing same process as auto1111

[vladmandic]

Animatediff doesn't seem to move too much when sending to img2img and checking animatediff, it's overall 16 frames of the same thing. Not sure if I'm doing something wrong, just doing same process as auto1111

That might be, but how is that relevant in this thread?

[a2ZxgdqHeSs8F51cIyuK]

Hello, since the author of Dynamic Thresholding seems to be MIA, I have continued to investigate this on my own. I tested it on a new, clean, updated install of the normal A1111 WebUI and it doesn't have these issues. I did notice a difference with the extension between A1111 and this fork however. On SD.NEXT the extension options look like this: However on A1111 they looks like this: As you can see the two minimum value sliders are missing in SD.Next! Those sliders are actually very important, their values have a massive effect on the output. Can you please take a look at this Vlad? I think if the sliders can be made to appear (again, I think? I wouldn't bet on it but I seem to recall them being there on older versions) in SD.Next then the extension should work like before. On A1111 if I mess with those sliders I can get both good outputs and also smudged outputs similar to the issues I showed in the first post of this issue.

EDIT: Actually they do appear in SD.Next if I mess with the menus a bit randomly. Strange. I'm doing more testing. EDIT2: Yeah, if I can get the sliders to show up and configure them sensibly I can get much, much better outputs. I'm playing with the settings but it's definitely a lot better. EDIT3: Here's Test Girl in all her glory after 6 completely unedited IMG2IMG passes: Much, MUCH better! There are some errors here and there but it's the kind of stuff I'd manually fix or inpaint, for an unedited output and abused with 6 passes, this is what I was expecting (and getting before the update)

[vladmandic]

i'm glad you're making progress. but i'm not sure what else can be done in this issue?

[a2ZxgdqHeSs8F51cIyuK]

Check why the two "minimum value" sliders don't show up? I can get them to appear if I fiddle with other settings in the extension, but in A1111 they just show up normally (so I don't think that's an issue with the extension itself). I'd guess something with the UI only makes them show up if they're 'activated' somehow by changing other settings. Could affect other extensions too potentially if they use the same type of menu.

[vladmandic]

looking at https://github.com/mcmonkeyprojects/sd-dynamic-thresholding/blob/master/scripts/dynamic_thresholding.py its clear that extension itself is changing visible property for its controls on interaction. not something i'm going to debug or change. you can report to extension repo.

[a2ZxgdqHeSs8F51cIyuK]

So should I close this and just tell https://github.com/mcmonkeyprojects to look at your comment about the UI to see if he can change how he handles those two sliders appearing? Or you can close it, the actual issue is solved.