SpaceinvaderOne / nvidia_powersave

script to save power on Unraid server with Nvidia GPU
69 stars 6 forks source link

nvidia persistence mode is deprecated warning #1

Open dwilson1251 opened 2 years ago

dwilson1251 commented 2 years ago

Warning message following run of script in log

NVRM: Persistence mode is deprecated and will be removed in a future release. Please use nvidia-persistenced instead

Looks like this is a utility that is compiled with the linux driver, however not sure if it's available for the one used w/unraid. Looking through the syslog at boot it doens't seem to load it at system init, but not an expert here. I followed the following references below and modified the script to use nvidia-persistenced --persistence-mode instead of nvidia-smi --persistence-mode=1.

That removed the error and returned the following in the syslog nvidia-persistenced: Started (22826). I'm not 100% clear on how this impacts the intent of this script, stability issues, etc, but will run it to see if any.

Reference Links: https://download.nvidia.com/XFree86/Linux-x86_64/396.51/README/nvidia-persistenced.html https://docs.nvidia.com/deploy/driver-persistence/index.html http://manpages.ubuntu.com/manpages/trusty/man1/alt-nvidia-331-persistenced.1.html https://github.com/NVIDIA/nvidia-persistenced

dwilson1251 commented 2 years ago

@SpaceinvaderOne - Quick update on this issue, it appears the using nvidia-persistenced --persistence-mode is a valid option, and negates the need for a periodic script. I'm not absolutely positive, but after about 24 hours of intermittent use the power draw seems to go back to the base state as expected.

Solverz-0 commented 2 years ago

@SpaceinvaderOne - Quick update on this issue, it appears the using nvidia-persistenced --persistence-mode is a valid option, and negates the need for a periodic script. I'm not absolutely positive, but after about 24 hours of intermittent use the power draw seems to go back to the base state as expected.

I found the same result with nvidia-smi --persistence-mode=1 if I just ran it manually, the state goes to P0 when running a process and back to P8 when no processes are running.

Of course due to the way unraid works, this would need to be run after every reboot or fresh system start.

ich777 commented 6 months ago

@dwilson1251 & @Solverz-0 nowadays it is enough to run nvidia-persistenced once at boot, no need to run it in intervals (except you want to get the power savings that @SpaceinvaderOne implemented), once at boot is enough. How you set that up it's up to you, it's enough to put the command in the go file or you simply run a User Script that is executed ad first boot.

But keep in mind that if you have two Nvidia GPUs in your system an you use one for Docker and one for a VM (which is not stubbed - not bound to VFIO) and you start the VM with nvidia-persistenced running in the background you will most likely crash your server. However you could of course send a kill signal to nvidia-persistenced before the VM is started and run nvidia-persistenced again after the VM is started, but that is a bit more involved.

SpaceinvaderOne commented 6 months ago

Hi @ich777 @dwilson1251 @Solverz-0 Yeah, I've steered clear of changing the script to include nvidia-persistenced mainly because, just as you mentioned, @ich777 , having it run while passing a GPU through to a VM makes the server crashing unless we stub it. But if we stub the card for passthrough, we're left without any power management when it's not being used by a VM because it won't have a driver, except for when it's actively passed through and relying on the guest's drivers.

So i thought it better to not use nvidia-persistenced in the script. Nvidia's been flagging the deprecation warning for ages now, but fingers crossed, it stays operational. I'd like to think Nvidia might keep supporting it, to maintain backward compatibility for people and businesses running on older setups, cause if they pull out the feature it might break systems and workflows around it, so hopefully it will stay around for now.

But yeah, @ich777, you're right, if you dont need to pass through a GPU that's not stubbed, puting nvidia-persistenced into the go file or a script to run at first boot is probably better.