dotnet / BenchmarkDotNet

Powerful .NET library for benchmarking
https://benchmarkdotnet.org
MIT License
10.35k stars 955 forks source link

Why Priority is High, not Real-Time? #1884

Open YegorStepanov opened 2 years ago

YegorStepanov commented 2 years ago

https://github.com/dotnet/BenchmarkDotNet/blob/f372668e028161d0d1bf675811a6168967175034/src/BenchmarkDotNet/Extensions/ProcessExtensions.cs#L25-L35

There is RealTime priority, which is the highest priority. It will reduce Windows overhead, because it's super aggressive mode. See Raymond Chen's article.

A little problem: the IDE/cmd must be launched in admin mode, otherwise the mode will be High.

Suggestion:

If it is safe, change the default priority and notify a user that Admin Mode increases the accuracy. Or add [RealTimePriority] attribute that will require Admin Mode.

Questions:

How much does it decrease OS overhead? What happens in MacOS/Linux?

Notes

The tests are passed as usual. SO discussion, some users force RealTime to play PC games.

@adamsitnik I mentioned you because you have written this code. A speedy recovery to you! 🍀

timcassell commented 2 years ago

That's interesting. I would want to make it an option to set the priority, but I wouldn't do RealTime by default. I sometimes manually kill my running benchmark processes, and from reading that article, that would be impossible if the priority is RealTime, right?

Also, by default BDN launches each benchmark as a separate process. Is it possible for it to launch them in admin mode without prompting the user if the parent process is already in admin mode? I haven't tried that before myself. BDN might also be able to build the parent process and run it as a separate process than the base process for a single admin prompt if that works. I'm not sure if that would also be fine with the InProcess toolchain, though.

YegorStepanov commented 2 years ago

When no-admin mode, nothing is changed.

If you starts IDE in Admin mode, the RealTime priority will be set, so all child processes inherit Admin mode automatically. I have increased admin privileges only one time (on IDE launch), not for every test :)

I have benchmarked the sum of 100 billion numbers, PC is just lagging, not critical, but noticeably. I could open sites, watch YouTube (notebook, i5-6400u)

[Upd] PC experience is feeling like rendering in Blender.

adamsitnik commented 2 years ago

@YegorStepanov the article that you have provided mentions:

Real-time priority is really dangerous

We can't use really dangerous things by default. Moreover, since people hardly ever use it, the reported time could be unrepresentative and simply far from reality.

A speedy recovery to you!

Thanks!

How much does it decrease OS overhead?

You could find some unstable benchmarks and see if setting the priority to real time helps. If it does, we could introduce a config that would allow the users to specify the priority on their own.

timcassell commented 1 year ago

This can be worked around by setting it in GlobalSetup.

[GlobalSetup] public void Setup()
    => Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.RealTime;

Of course this is advanced, so user should already know they need admin mode.