[Issue]: Generate immediately auto-aborts sometimes

[exodia52]

Issue Description

This is getting out of hand. The Generate Button freezing getting worst and worst. Sometimes even just 1 Generate already Freeze it. It happens randomly. Even when "Generating Forever" it stop by itself because of the Frozen Generate button. Couldn't Generate anymore but able to go to other "Tabs" like "Settings", "Img2Img", "Trains" etc.

Every time the Generate button freeze it will always tries to generate an image then shows the "Progress Bar" then fully complete but no output was shown or done.

Setup.log doesn't show any error. Same goes for the Console. It just refuse to be pressed until you Refresh the WebUI and sometimes it will let you Generate 1 image then froze again.

Version Platform Description

Version: 09a14802 Mon Apr 17 15:52:26 2023 -0400

Window 10, Ryzen 7 2700, RTX 3060 Ti

[vladmandic]

I'll take a look, but its really hard without the way to reproduce. i've added --disable-queue to cmd flags that disables Gradio queues and forces it to use HTTP instead of WebSockets - can you try with that?

i'm marking the issue as question if anyone can provide extra information.

[exodia52]

Generate > It does it job > Generate again > Tries to do it job > Says it completed but no output > Freeze.

The amount of Generate is really random sometimes 1 generate already can freeze the button and sometimes can do 20 generate before it does that.

For some reason i can easily replicate it every single times. Video link below to show what i did to replicate it. https://youtu.be/UKQkfJlkrF0

I'll take a look, but its really hard without the way to reproduce. i've added --disable-queue to cmd flags that disables Gradio queues and forces it to use HTTP instead of WebSockets - can you try with that?

i'm marking the issue as question if anyone can provide extra information.

Currently i'm trying the --disable-queue waiting for it to freeze using Generate Forever.

[exodia52]

The Generate Button freezed again. After 8 Generated Image.

[vladmandic]

when this happens, is progess bar still displayed? is it marked as at 100%?

[exodia52]

It shows for awhile. Says "Waiting" then it died and disappear. Like nothing happened.

It shown at the video at time 0:05

[mart-hill]

I can confirm that it happened to me with older UI commit (shortly after it was reported in #190 [the GENERATE button didn't do anything and couldn't do anything anymore with it, I had to restart the UI; the rest of the UI was responsive though, I didn't check, if the queue is locked out as well]). Could really fast double-click, or accidental right- then left-click on GENERATE button cause that behavior?

Edit: I forgot to mention it happened to me twice, intermittently. And I have live previews on, just changed the value from 1000 to 50, testing!

[vladmandic]

i have an idea, can you try something? ui -> settings -> live preview -> Progressbar/preview update period default is 1000, set it to some really low value like 50

[exodia52]

Hmm.. After the latest commit i cannot even Generate or Restart the UI at the Extension or at the Settings.

[mart-hill]

On my side, I'm yet to encounter the freeze - I threw many things at once at the UI for generating just one picture - 4 LoRAs/LyCORIS' (in additional networks), Tiled VAE with 512pix tile, Composable LoRA extension active, Sonar sampler add-on active (with latent [bicubic antialiased] hiresfix upscaler).

[vladmandic]

Hmm.. After the latest commit i cannot even Generate or Restart the UI at the Extension or at the Settings.

which commit hash? i've inadvertently created a bug that breaks generate completely, but fixed it 10min later. you should be on e8d8dae4c71f3bc5043e972620c37921cdf7def3

[exodia52]

Hmm.. After the latest commit i cannot even Generate or Restart the UI at the Extension or at the Settings.

which commit hash? i've inadvertently created a bug that breaks generate completely, but fixed it 10min later. you should be on e8d8dae4c71f3bc5043e972620c37921cdf7def3

No wonder. My bad also. I didn't update my folder. . . Even tho i saw already there's a new fix in commit.

Finally on Version: e8d8dae4 atm. Trying the above solution after my console finish loading.

[vladmandic]

Trying the above solution after my console finish loading

its not a solution, its an idea i have on what's the root cause. if that changes behavior, then i know what the issue is and fix will come shortly.

[exodia52]

Alrighty, Testing my trusty Generate Forever to catch that bugger. Will update when it freeze again. Usually won't take long. Hopefully.

[mart-hill]

I've noticed interesting thing:

That's despite having negative prompt (two negative embeddings names included). The both numbers are white, when the prompt fields are empty. But either after generation, or after a while, the negative part stops parsing. Could that be something?

[vladmandic]

nah, thats totally separate

[exodia52]

Currently no freezing yet. Will let it run on Generate Forever for the night. Usually it will stop by itself after awhile.

I'll update if it still freeze after 6 hrs+ running. When i woke up.

[vladmandic]

my working theory is that javascript polling logic was broken - so anytime generate completes before front end actually updates, it will never mark it as complete as its waiting for task to start when it already completed.

its the logic from a1111, but just more visible here as my default polling interval is a bit longer.

[vladmandic]

i've just pushed the update that hopefully fixes this (if not, i'll reopen). a (big) benefit is that i think i figured out how to reconnect browser to active session upon browser restart!

[mart-hill]

Thank you, that's a BIG benefit! 🙂

[exodia52]

Thanks the update !

For my Generate button freezing sometimes it happens after 20+ Generate. Which is better but the problem still floating.

---

I been playing whack a mole game with another problem pop out. Tell me if you want me to open another issue thread instead of dumping it here.

Traceback (most recent call last): File "C:\Users\zerok\Downloads\Test\automatic2\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict output = await app.get_blocks().process_api( File "C:\Users\zerok\Downloads\Test\automatic2\venv\lib\site-packages\gradio\blocks.py", line 1078, in process_api data = self.postprocess_data(fn_index, result["prediction"], state) File "C:\Users\zerok\Downloads\Test\automatic2\venv\lib\site-packages\gradio\blocks.py", line 994, in postprocess_data block = self.blocks[output_id] KeyError: 373

This happened when restarting the UI at Extension setting. Which make the UI freeze. The Error appear when clicking the " had to fully restart the Console after this Error appears.

---

Another problem when Console starting there's an issue with Diffusers downgrade to 0.14.0 instead of staying on 0.15.0 as stated in the Requirement.txt. Even after upgrading it, Diffusers downgrade itself during next launch.

[vladmandic]

for this issue, lets stay on topic of frozen generate button. and to clarify, when this happens: progress bar is not visible, generate is visible, just when you press generate nothing happens? is there anything in a) console, b) browser console

regarding keyerror, on-the-fly restart of gradio is pretty much a best effort - there is no way to get to 100% unless you restart server from command line. so lets not add that to this issue.

regarding diffusors - can you clarify? current requirements state diffusers==0.15.0 - do you want to use 0.14.0 (and why?) or is it opposite, you want to use 0.15, but something is downgrading them? if its latter, its 99% some extension you've installed that has different requirements (for example, dreambooth extension is notorious for doing that) and forces installation of its requirements upon load.

[exodia52]

for this issue, lets stay on topic of frozen generate button. and to clarify, when this happens: progress bar is not visible, generate is visible, just when you press generate nothing happens? is there anything in a) console, b) browser console

Alright will stay on the topic.

The progress bar does show up but it really fast now after we set the Preview to ui -> settings -> live preview -> Progressbar/preview update period to 50.

The console does not try to "Generate" the picture even tho the Button has been pressed but the button does Grey out where you can see "Stop/Skip" for few sec and turn back to Orange "Generate" button.

Sadly this the worst part. There's no Error in the Console. Where can i find this "Browser Console" ? If you meant by notification pop-out on the top right there's no error.

Even tried the --disable-queue and had no success for this issue.

Do note that i'm using Opera GX but i always been using this Browser with other WebUI and had no issue.

Additional info for above issue.

The Console doesn't tries to Load the weights of the Lora and the Generate button didn't get to send the info to the console to tries Generate an image.

Usually when generating a Lora it will tries to "Loading weights:....." and start generating.

I also confirmed that the Lora i use is different and have the same issue and also is a "LORA" file. Not Lycoris.

regarding diffusors - can you clarify? current requirements state diffusers==0.15.0 - do you want to use 0.14.0 (and why?) or is it opposite, you want to use 0.15, but something is downgrading them? if its latter, its 99% some extension you've installed that has different requirements (for example, dreambooth extension is notorious for doing that) and forces installation of its requirements upon load.

I do have Dreambooth extension. Probably because of that. It just being notorious on downgrading mine while i keep trying to upgrade it to prevent any issue. Will noted on this and remove Dreambooth for now to avoid any issue.

Will skip this Diffusers issue for now till i can pin down the Freezing issue.

[vladmandic]

Where can i find this "Browser Console" ? If you meant by notification pop-out on the top right there's no error.

pretty much every browser has browser console. on edge/chrome, its inside of "developer tools" or sometimes its called "inspector". again, on edge/chrome, you can show it via menu or by pressing ctrl+shift+c. i have no idea about opera.

The console does not try to "Generate" the picture even tho the Button has been pressed but the button does Grey out where you can see "Stop/Skip" for few sec and turn back to Orange "Generate" button.

thats not how it looks in default black-orange theme. theme should not matter, but can you check just so we get closer to nailing this down.

The progress bar does show up but it really fast now

so progress bar shows up, but then dissapears and nothing gets generated? that sounds like it started the task, but then aborted it.

[exodia52]

pretty much every browser has browser console. on edge/chrome, its inside of "developer tools" or sometimes its called "inspector". again, on edge/chrome, you can show it via menu or by pressing ctrl+shift+c. i have no idea about opera.

I found the browser console but not sure what i should keep my eyes on.

The console does not try to "Generate" the picture even tho the Button has been pressed but the button does Grey out where you can see "Stop/Skip" for few sec and turn back to Orange "Generate" button.

thats not how it looks in default black-orange theme. theme should not matter, but can you check just so we get closer to nailing this down.

I'm using different Theme. Was wondering the same thing because i need to narrow down the issue. Changing it back to default now.

so progress bar shows up, but then dissapears and nothing gets generated? that sounds like it started the task, but then aborted it.

Yeah, i kinda agree on this. It doesn't load the Weight since all Lora generated image send Lora Weight to the Console.

Currently, I'm Generating Forever for the Browser Console to catch any error.

[vladmandic]

so basically, no errors in browser console as well (it would show up already). i need to track down why the task gets aborted, non-trivial. i wish i could reproduce, that would make it sooo much simpler.

[exodia52]

Been generating over 100 images now. Felt like the Theme changes might be the issue.

The theme i used was Stardust and Gradio/Default

[vladmandic]

theme should really not matter, i just asked to switch to default so we're talking about same visuals (whats visible, whats not, etc)

[exodia52]

Then i'll be Generating Forever again. Seems like the issue didn't appear. Which is quite rare for me to get more than 100 Generated image while not getting any Freeze.

Will update in the morning.

[exodia52]

Been generating for more than 6 hr+ and no issue. Generated more than 400 Image.

I changed the ui -> settings -> live preview -> Progressbar/preview update period to 100. bcs 50 were too fast. Server arguments: disable-queue,--medvram,--no-half-vae, --skip-git Deleted Dreambooth extension. Off Show previews of all images generated in a batch as a grid since i don't need it. Updated to the latest Commit.

That's the only notable thing i did.

Even restarted my Console because felt like this is just too good to be true. Still running with no freeze.

[vladmandic]

50ms update period is definitely not needed, that was just for me to confirm my theory on the root cause. now that progress check is done differently, even very high update period like 5000ms (5sec) should not cause issues, but i think 250ms is a good default. faster than that puts a lot more stress on the browser so if you don't have high end pc, it can cause slowdowns.

anyhow, glad its working now. closing of now, but if it reoccurs, i can reopen the issue.

[exodia52]

50ms update period is definitely not needed, that was just for me to confirm my theory on the root cause. now that progress check is done differently, even very high update period like 5000ms (5sec) should not cause issues, but i think 250ms is a good default. faster than that puts a lot more stress on the browser so if you don't have high end pc, it can cause slowdowns.

anyhow, glad its working now. closing of now, but if it reoccurs, i can reopen the issue.

I will try the 250ms when i get back from work. Sometimes on the 50ms or 100ms i get memory ran out and the browser crashed. Not a bit deal since simple refresh, easily able to resume what i been doing.

Thank you so much for the active helps. Really appreciate what you been doing. Have a great day.

[merodiro]

I have the same issue. This is the browser console output. it works for some time when I refresh my browser without restarting the server And I can also generate from other tabs such as txt2img if the img2img generate button is frozen without needing to refresh

[vladmandic]

doesn't look the same, can you create a new issue for this?

[mart-hill]

For the time being, I can use the UIs (@vladmandic's fork and the @anapnoe's fork) thanks to the --disable-queue parameter. Freezing is inevitable for me, no matter if I use xformers or SDP optimization. The other way to try to avoid it would be to "be adventurous" and try torch 2.1 nightly, as @vladmandic suggested. 🙂 I'm really curious, as to why websockets communication seems to be at least the catalyst for this freeze to happen...

[anapnoe]

For the time being, I can use the UIs (@vladmandic's fork and the @anapnoe's fork) thanks to the --disable-queue parameter. Freezing is inevitable for me, no matter if I use xformers or SDP optimization. The other way to try to avoid it would be to "be adventurous" and try torch 2.1 nightly, as @vladmandic suggested. 🙂 I'm really curious, as to why websockets communication seems to be at least the catalyst for this freeze to happen...

can you try to generate a small image like 32x32 px? for me it seems like there is some kind of overflow on the client side that crash the script engine if you disable the preview does it work?

[mart-hill]

I had the Generate button freeze even without a preview, so this option is the only way for me. It's exactly as @exodia52 showed in his video up here. I'll try to generate such small image with your fork, since it freezes earlier on me. Of course it won't, when I use --disable-queue parameter, so I'll run it without it.

Could that be that the shared.demo.queue(64) in your fork's webui.py vs @vladmandic's shared.demo.queue(16) matters somehow?

[anapnoe]

This parameter is used to set the number of worker threads in the Gradio server that will be processing requests in parallel. so maybe 64 it to high I haven't tested I will lower the value 16 to 32 seems like a more reasonable range

[vladmandic]

It was 16 in my fork until two days ago, so that's not it.