Consistent CPU performance for benchmarking

[sfiruch]

The Request

I regularly want to benchmark my code, to see the performance impact of recent changes. This is becoming more and more difficult these days, because of energy-saving measures, dynamic overclocking (Base+Boost), thermal throttling, and different profiels for plugged-in vs. not-plugged-in states. On my device (Surface Pro), there does not seem to be way to control all these factors in the firmware or windows. Issues like #78 can make it difficult to even notice thermal throttling.

A generally supported way to increase the "performance consistency" of my system would be valuable.

Workarounds I use at the moment

Currently, some of my benchmarks run faster on colder days and vice versa. On battery, the results are not comparable to those when plugged-in. It means that I have to re-run the baseline almost every time, and also make sure that my system is in the same thermal throttling phase.

Non-solutions

This is not to be confused with something like "Game Mode" or "High-Performance Mode" where the aim is maximum performance. I'd be very happy with meek performance, as long as it is consistent.

Implementation

I don't know whether the right implementation is an API or a toggle in the UI. Ideally, the mode would be enabled automatically by my IDE of choice when running certain projects. Or an API would probably be used by libraries like BenchmarkDotNet

It would fix the clock speeds of the CPU, GPU, system buses, etc. with limits low enough to prevent thermal throttling. Ideally, it would also limit activity of background processes, i.e. no background downloads, AV scans, etc..

[sylveon]

I don't think this is particularly feasible, especially in mobile devices like yours where the CPU is locked, so it's impossible to do any kind of tweaking to the clock rate or other factors (for example, using Intel XTU to set a fixed clock rate won't work on your Surface)

[sfiruch]

Allowing detection of (thermal) throttling would be a useful first step. Quoting Bruce Dawson on Twitter:

Is there a user-mode way to detect CPU thermal throttling in Windows? Ideally I'd like to know the requested CPU frequency, actual CPU frequency, and if the frequency is throttled the reason why. I've seen CPUs throttled due to overheating and due to insufficient power.

[dmachaj]

You might be able to lock your CPU frequency at a value far enough below the maximum that it will avoid throttling. The following commands should keep your CPU at 85%.

https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/hardware/power/power-performance-tuning#minimum-and-maximum-processor-performance-state

powercfg.exe /setacvalueindex scheme_current sub_processor procthrottlemin 85
powercfg.exe /setdcvalueindex scheme_current sub_processor procthrottlemin 85
powercfg.exe /setacvalueindex scheme_current sub_processor procthrottlemax 85
powercfg.exe /setdcvalueindex scheme_current sub_processor procthrottlemax 85
powercfg.exe /setactive scheme_current
shutdown /r /t 0

(Note that both AC and DC power settings are being set with that command)

[rickbrew]

Even on a desktop (Ryzen 5950X), its performance is temperature sensitive. In order to benchmark my app's startup performance, I have to close/kill/suspend anything else that's running, turn the CPU fans to max, and open the window (even in winter). Those background processes that sip teeny tiny bits of CPU can cause gross fluctuations in the timings for startup performance measurements.

[AvriMSFT]

@deiruch thanks for reporting this issue and apologies for the delayed response. Your idea of the mode being "enabled automatically by my IDE of choice when running certain projects. Or an API would probably be used by libraries like BenchmarkDotNet" makes a ton of sense. There is a team here just spinning up that's focused on developing performance testing tooling for your applications, and this sounds like an awesome feature request for that tool :). I'll make sure to share this with the team.

[AdamBraden]

I've created a new feature suggestion in the Dev Home repo. That is where a feature like this would make the most sense. Closing this issue.