Image quality degradation over past weeks

[Lmaookay04]

I run forge webui from a custom notebook in Google colab, I use the A100 GPU (NVIDIA), it has 40GB VRAM. Using for continuously since the support of flux has made it my preference.

Over the past few weeks though, (with no changes to my routine) attempting similar tasks now nearly always results in terrible image quality that was not present in the earlier days (like early August to early September) of Flux support. The image issues usually are along the lines of blurry, artifact ridden, pixelated, unfinished, uncanny… and other similarities of that nature.

Interestingly though, reverting to a previous commit does not solve the issue… Making me think maybe it’s a dependency within the colab environment, or some update to their memory, GPU/AI architecture, or other changes variable related to processing on forge, maybe? I don’t know.

I’m seeking any recommendations or ideas that anyone has about this issue, particularly if you have similar troubles and have potentially found some workaround? Thank you all in advance.

++ If you have a commit recommendation please share as I will be happy to try!

[derfasthirnlosenick]

Pls provide setting/prompt/checkpoint for a test case that looks different on your system with two different versions.

[Eroy09]

Pls provide setting/prompt/checkpoint for a test case that looks different on your system with two different versions.

Will do, give me a few hours and i'll launch it up

[Eroy09]

• Below are screenshots containing my entire colab notebook workflow for forge webui and the codes' outputs. -

• Below is a .txt file containing both the latest versions' and a mid august commit's output logs from launch. - Logs.txt

• Below is a Screen shot from the UI page, used for both generations. -

• Below are both the 1st and 2nd versions' generation outputs, i used the (PNG Info / Send to txt2Img) From the first generation to generate the second output. I also set "GPU weights" to the same value for both. -

(1st) -

(2nd) -

[derfasthirnlosenick]

I didn't do a binary comparison, but the pics look the same... am i missing something?

[Eroy09]

I didn't do a binary comparison, but the pics look the same... am i missing something?

Yeah they're the mostly similar, i'm more referring to their quality, it's really bad, and this is really uncommon for flux, i just did two generations to show the issues is persistent across commits via google colab.

P.S. - I went to do a comparison in comfyui (sure that it was going to be way better, because just yesterday i was doing reproduction comparisons, and comfyui was producing great images from the exact parameters that were given to forge, forge though handled those generations poorly.) but starting today, comfyui is now producing for me the same quality issues, and was not before. It's like the forge situation all over again, but forge started acting funny for me around Oct 20. I also reproduced similar issues in huggingface spaces

[Eroy09]

@lllyasviel

[derfasthirnlosenick]

I didn't do a binary comparison, but the pics look the same... am i missing something?

Yeah they're the mostly similar, i'm more referring to their quality, it's really bad, and this is really uncommon for flux, i just did two generations to show the issues is persistent across commits via google colab.

P.S. - I went to do a comparison in comfyui (sure that it was going to be way better, because just yesterday i was doing reproduction comparisons, and comfyui was producing great images from the exact parameters that were given to forge, forge though handled those generations poorly.) but starting today, comfyui is now producing for me the same quality issues, and was not before. It's like the forge situation all over again, but forge started acting funny for me around Oct 20. I also reproduced similar issues in huggingface spaces

You can just fall back to an old commit and see if the same prompt/settings generates something different.

[Eroy09]

I didn't do a binary comparison, but the pics look the same... am i missing something?

Yeah they're the mostly similar, i'm more referring to their quality, it's really bad, and this is really uncommon for flux, i just did two generations to show the issues is persistent across commits via google colab. P.S. - I went to do a comparison in comfyui (sure that it was going to be way better, because just yesterday i was doing reproduction comparisons, and comfyui was producing great images from the exact parameters that were given to forge, forge though handled those generations poorly.) but starting today, comfyui is now producing for me the same quality issues, and was not before. It's like the forge situation all over again, but forge started acting funny for me around Oct 20. I also reproduced similar issues in huggingface spaces

You can just fall back to an old commit and see if the same prompt/settings generates something different.

Thats what the 2 images are, the latest version at the time and and an old commit. When i take old generations that were good and feed the parameters back now no matter what it commit it seems it just wont reproduce the image