search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
142.14k stars 26.84k forks source link

[Bug]: Iteration slow down by half if you let Windows 10 turn off your monitor #8300

Open huyzer opened 1 year ago

huyzer commented 1 year ago

Is there an existing issue for this?

What happened?

If you let the computer/Windows put the monitor to sleep, it will halve your iterations per second.

This didn't occur when I turned off my monitor with it's physical Off button. It's only if Windows does it.

To fix, go to Power & sleep settings > Additional power settings > Change when the computer sleep > Change advanced power settings > (expand by hitting + symbol) Display > Turn off display after (two options to "Never").

Steps to reproduce the problem

  1. Go to .... Power & sleep settings > Additional power settings > Change when the computer sleep > Change advanced power settings > (expand by hitting + symbol) Display > Turn off display after (two options to "1 minute"). Wait, and let the monitor turn off.

  2. Check your iterations/per sec. It should be half.

What should have happened?

Normal behavior should be the iterations stay constant if you let Windows turn off the display for you. I did multiple tests and settings to confirm this was the only variable.

Screenshot (41590)

Screenshot (41587)

Commit where the problem happens

3715ece0

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

No, I don't think I am.  Here's what's inside webui-user.bat

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=

call webui.bat

List of extensions

StylePile https://github.com/some9000/StylePile.git unknown multi-subject-render https://github.com/Extraltodeus/multi-subject-render.git unknown sd-extension-steps-animation https://github.com/vladmandic/sd-extension-steps-animation.git unknown sd_grid_add_image_number https://github.com/AlUlkesh/sd_grid_add_image_number.git unknown stable-diffusion-webui-depthmap-script https://github.com/thygate/stable-diffusion-webui-depthmap-script.git unknown stable-diffusion-webui-images-browser https://github.com/AlUlkesh/stable-diffusion-webui-images-browser.git unknown stable-diffusion-webui-inspiration https://github.com/yfszzx/stable-diffusion-webui-inspiration.git unknown

Console logs

I believe this is not relevant.

Additional information

No response

ClashSAN commented 1 year ago

is it your gpu entering power saving modes? It might be specific to your gpu configuration.

Cyberbeing commented 1 year ago

As clashsan said, your GPU likely enters power saving mode when the monitor is off.

First thing you could try is setting Prefer Maximum Performance for Python.exe in a NVCPL application profile.

If that doesn't fix it, you can use nvsmi to lock your clocks and your gpu will never downclock outside of power/temp throttling. nvsmi is usually found in C:\Program Files\NVIDIA Corporation\NVSMI. To make life easier I recommend setting the nvsmi directory in system environmental variable PATH if it isn't already, so nvsmi can easily be called from cli or bat files.

For example on my RTX A4000 with oc'ed memory I use: nvsmi -lgc 2000 nvsmi -lmc 8001

Which locks 2000Mhz Core and 8001Mhz GDDR6 Memory Clocks when not throttled. Recommended to values specific to your GPU. Keep in mind that nvsmi will never clock higher than the maximum boost clocks possible for P0 power state even if you set a higher value.

To remove locking and reset core and memory back to defaults use: nvsmi -rgc nvsmi -rmc

It's also possible your CPU could be entering a low power state, but check the GPU route first.

huyzer commented 1 year ago

First, I'd like to thank you both for so quickly getting back to me and giving me input on the matter! I really appreciate it. Secondly, I tried your fixes @Cyberbeing , but sadly it didn't work, so I think I'll just stick to my solution. Third, if you're curious, this is what I did try:

I added "C:\Program Files\Python310\python.exe" to "Prefer Maximum Performance" and that didn't work, even after reboot.

I tried looking into the PATH thing, but that's a bit more complex than I feel comfortable with. By the way, my own NVSMI isn't in "C:\Program Files\NVIDIA Corporation\NVSMI".

Then I went to CMD window:

So I typed: nvidia-smi -q and found this:

Max Clocks Graphics : 2100 MHz SM : 2100 MHz Memory : 6001 MHz Video : 1950 MHz

So I'm guessing I should write:

nvsmi -lgc 2100 nvsmi -lmc 6001

I'm running on my ASUS laptop with a "NVIDIA GeForce RTX 3060 Laptop" gpu.

Those above commands didn't work, so I tried:

nvidia-smi -lgc 2100

That gave me: The current user does not have permission to change clocks for GPU 00000000:01:00.0. Terminating early due to previous errors.

Ah, so I have to run "cmd" as an administrator.

Did the same for -lmc 6001. Worked!

Now, is this setting persistent, or do I have to do this everytime I reboot?

Did the same command, and it worked. Yay! Haha. GPU clocks set to "(gpuClkMin 2100, gpuClkMax 2100)" for GPU 00000000:01:00.0 All done.

But then I ran my test, let the display turn off (controlled by Windows), and it halves again. Haha. Oh well. It's alright. I'll just stick to "Never" for turning off the display, locking the account (which I thought was the problem, but wasn't), and manually turning off my monitor with its physical button.

I tried many ways. Nothing worked:

nvidia-smi commands, started Stable Diffusion test nvidia-smi commands, reboot, started Stable Diffusion test reboot, nvidia-smi commands, started Stable Diffusion test

Cyberbeing commented 1 year ago

The NVIDIA GPU shouldn't throttle a nvsmi lock, so that should rule out your GPU. And yes, nvsmi is not persistent through reboots. The Intel iGPU can't use CUDA, so it shouldn't switch use that, but maybe your laptop just has some strange power saving feature forced when the screen is off.

Other things you could try:

Just remember to note down what you've changed, since for laptops in particular power saving features are important for battery life. You'd likely want to set them back to normal when not using Stable Diffusion.

And yes, the simple solution of just setting a Solid Black Screensaver and disabling automatic screen shutoff may also avoid the issue while still saving some battery power.

huyzer commented 1 year ago

@Cyberbeing I absolutely appreciate you taking the time to help me like this on a Saturday. :D And, so quick in replies, too. So, thank YOU!

I think I'll just stick to the simple fix, and not get myself into trouble. I know enough about computers to not worry too much about picking and prodding, but enough to be dangerous to my computer. :D Haha! I looked into BIOS, but it didn't seem to have anything about power settings.

I went into Graphics Settings and added Python to "High Performance" as well as my Armoury Crate (and also saw that iGPU was already off when I opened up Armoury Crate), but both didn't help even after restart. So, I think I will just be satisfied now, even though my curiosity did make me try just to try. Thanks, again, for your time! I truly appreciate it. Have a great weekend!

Screenshot (41641)

Anonym121 commented 1 year ago

I am having a similar problem. The difference is that I leave SD overnight and it literally stops creating images at some point after the screen is locked. I know this because when I unlock it in the morning, SD starts doing batches at normal speed and I have never found it in the middle of a batch when I do this.

I already asked in SD discord and in a general tech support forum and implemented all the battery options they suggested. Now I have a custom power plan in best performance with never sleep in all the fields. BTW, I think I can rule out the possibility that it is a problem with my computer since I used to also leave topaz video AI overnight with 5-6h long tasks and it didn't stop.

I have estimated that with certain settings, if in 5 hours I should get 32-33 batches, I'm getting 8 instead. BTW, the computer does not sleep, the screen is permanenly turn on (I'm using a laptop with an external 4k screen) and I am running SD from an external HDD. I also considered that it could be that the pagination space was insufficient but currently I have 12 GB free in System with a current pagefile size of 4 GB. There is a field called disk sleep in power plan but I set it to never just in case. I also tested manually setting python and the other SD processes to "above normal", "high" and "highest" priority in task manager.

Specs: Ryzen 4800-H, NVIDIA 2060, 6 GB VRAM, 16 GB RAM

huyzer commented 1 year ago

@Anonym121 Did you try my simple solution where I mentioned, "But then I ran my test, let the display turn off (controlled by Windows), and it halves again. Haha. Oh well. It's alright. I'll just stick to "Never" for turning off the display, locking the account (which I thought was the problem, but wasn't), and manually turning off my monitor with its physical button." ?

Anonym121 commented 1 year ago

@huyzer The display does not ever turn off now. I configured the laptop to never turn it off because I also thought that could be it. Also, I already tried leaving it overnight with both, screen on and off (physical button).

It doesnt seem to be directly related to screen lock since if it was it, it would only do one or two batches (each batch in the conditions I tested were 8-10 min) before it locks; but it does 8 instead. I do not dare to disable the lock feature completely since it needs to tamper with the register and I also need it on at work.

In a few nights I'll try an idea I got, leaving a big torrent I dont need downloading in the same drive where SD is running, that way I can discard it being something drive specific (like it sleeps regardless of windows config) since it will be doing another task in paralel.

gonultasbu commented 9 months ago

I have the same problem on a headless setup with a general machine learning training pipeline. Does anyone have a conclusive solution to this?

Anonym121 commented 9 months ago

I have the same problem on a headless setup with a general machine learning training pipeline. Does anyone have a conclusive solution to this?

In my case, disabling any power setting did not do anything to solve this. What I did was disabling the automatic screen lock in windows 10, the one where you need to write your password again to access the device. Other programs work perfectly when the screen is locked; but SD does not. I had no idea if they had patched it since I never changed my settings again; but thanks to you, I know they didn't.

Idk how exactly it went; but I had read that to disable the screen lock, you needed to use the registry. When I searched, I found a way to do it without that; but I don't remember how I did it.

chaimkohn commented 4 months ago

HI, if anyone still looking, for me it was edge putting tbs to sleep, adding the tab to the exception page resolved hits (edge://settings/system > Optimize Performance > Never put these sites to sleep > Add)