[Bug]: Stable diffusion stops generating if left alone a certain time [circumvented]

Anonym121 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What happened?

I run SD from an external HDD Leaving SD in continuous generation overnight produced less than expected images. It turns out it gets "frozen" at some point and stops generating new batches. If I unlock the screen in the morning, it suddenly starts to make new batches and laptop fans start cooling (prior to unlock, it is not doing anything). I already asked in SD discord, tech help discord, etc and they told me to change powerplan and suspension settings in windows and to set SD processes priorities to high, above normal and real time in task manager. None of that worked.

Steps to reproduce the problem

Open Stable diffusion webui
Set parameters for a generation (I used randomizer plugin, hires fix and wildcards; but I don't think it is related)
Click on generate forever
Measure the time for each batch of images generated
Wait 5+ hours
You will notice fans are off and the computer is not doing anything apparently. Unlocking it will cause generate forever to resume.
Compare the number of batches you got with the number you would expect for all the time you left SD on. You should get less than 33% the expected number of batches.

What should have happened?

Generation should not have stopped.

Commit where the problem happens

22bcc7be428c94e9408f589966c2040187245d81

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Microsoft Edge

Command Line Arguments

set COMMANDLINE_ARGS=--deepdanbooru --medvram --xformers --api --cors-allow-origins=http://127.0.0.1:3456 --no-half-vae --skip-install

List of extensions

Console logs

This problem is not reflected on logs, the cmd progress just gets frozen until you unlock the computer.

Additional information

It must happen when it finishes a batch since, when unlocking, SD is always at the step in which it is at 0% in cmd and doesn't display a progress number in the webui interface, which happens between batches.

I found a way to circumvent this issue by downloading at extremely low speed a torrent to the same drive SD is running in. This makes SD go on and not stop. Other ways to force the disk to do tasks overnight should work too. Nevertheless, a bug.

I updated to latest commit (5ab7f213bec2f816f9c5644becb32eb72c8ffb89). I'll check if it still happens here.

missionfloyd commented 1 year ago

It's not freezing, your computer is going to sleep. I use this when I need it to stay on.

Anonym121 commented 1 year ago

Not sure, it might be sleeping; however all settings have been changed so it theoretically never sleeps. In fact screen does not turn off, the only sign of something happening when I leave it is that the screen locks.

Other programs I have used that do lengthy tasks do not induce this behavior even when the computer settings said to sleep after a certain time. EG Topaz AI, Format factory, torrent...

So while it might not be a bug, it is still a feature other programs that run a long time have but SD doesn't. If it didn't have the generate forever option, I would say it was not devised for long runs; but it has it, so this happening interferes directly with a native option.

missionfloyd commented 1 year ago

Try the program I linked above. It does the same thing those other programs do to keep the system awake. You can add it to your webui-user.bat so it starts automatically.

Ramoyald commented 1 year ago

I have the exact same issue where every couple of generation, the tool just freeze during a generation and won't become responsive to anything. You can't even cancel the current generation and are forced to completely close the app and restart. I tried the lock screen and priority workaround but it didn't worked for me either.

This has nothing to do with sleep mode since this is not active on my current build and I'm using powertoys which is better than what was suggested to avoid sleep mode. I have no idea what made this happens and since there's no logs about any error, we don't really know how to proceed to find ressources for a fix.

The webui can still be used for a little bit but forget about hours and hours of generation, it doesn't work. As an additionnal note, this started occurring for me only after I switched to SDXL models. I remember controlNet being picky about permission on installation as well and still keeps saying permissions are wrong even after multiple fixes. These are my two possible culprits atm.

Edit: I decided to transfert a x265 10bit video on the ssd where the webui is installed and its not enough to starts the generation again.

Here's the error I was able to grab from trying some stuff around with generate forever.

INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 4 images in a total of 1 batches.
  0%|                                                                                           | 0/40 [00:00<?, ?it/s]
*** Error completing request
*** Arguments: ('task(fhwrs69hiaj5npu)', '__XLPrompts__', '__Negatives__', [], 40, 'Euler a', 1, 4, 5, 512, 768, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x0000014B803A8340>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x00000149F9917F10>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x00000149DE35EC20>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000014B81B82290>, False, False, 'Matrix', 'Columns', 'Mask', 'Prompt', '1,1', '0.2', False, False, False, 'Attention', [False], '0', '0', '0.4', None, '0', '0', False, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None, None, False, None, None, False, 50, [], 30, '', 4, [], 1, '', '', '', '') {}
    Traceback (most recent call last):
      File "J:\AI\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "J:\AI\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "J:\AI\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "J:\AI\stable-diffusion-webui\modules\processing.py", line 732, in process_images
        res = process_images_inner(p)
      File "J:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "J:\AI\stable-diffusion-webui\modules\processing.py", line 867, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "J:\AI\stable-diffusion-webui\modules\processing.py", line 1140, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "J:\AI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "J:\AI\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "J:\AI\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "J:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "J:\AI\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "J:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "J:\AI\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 104, in forward
        uncond = prompt_parser.reconstruct_cond_batch(uncond, self.step)
      File "J:\AI\stable-diffusion-webui\modules\prompt_parser.py", line 301, in reconstruct_cond_batch
        res[k][i] = param
    RuntimeError: The expanded size of the tensor (77) must match the existing size (154) at non-singleton dimension 0.  Target sizes: [77, 2048].  Tensor sizes: [154, 2048]

Remington67 commented 5 months ago

I have exactly the same problem.

I have tried the Insomina program but no change, after one hour (approx) generation stops.

And, ofcourse I have disabled every energysavings and so on. I only have a screenlock but it kicks in after 3 minutes and it keeps generating 57 minutes after that so it shouldnt be the problem.

So the problem is not that the computer is going to sleep its something else, If I unlock the screen it continues, no problem and no hang ups of gui or something like that.

chaimkohn commented 5 months ago

@Remington67 I have been struggling with the same, I have narrowed it down to browser tab sleeping, solution is probably here

(edge://settings/system > Optimize Performance > Never put these sites to sleep > Add) add URL to exception list

Remington67 commented 5 months ago

Ahh, @chaimkohn , you made my day!! Its been generating for two hours now so this seems to work (why didnt i figure that out my self ...) I thougt that it was the commmand window that made all the work so I dint even think of the browser setting.

Great response!!

AUTOMATIC1111 / stable-diffusion-webui