On some of the test systems, the GPU draws more power when the temperature
increases and starts throttling the core frequency when reaching the power
limit. This leads to unstable performance, as the execution time of a kernel
depends on the temperature when the kernel is executed. If the GPU has been
idling before starting a kernel, the GPU temperature is low and the execution is
fast. However, if other kernels have been executed before, the GPU's temperature
and power draw are high, resulting in throttling, in turn resulting in a longer
execution time.
The file expenv-paper applies contains per-host settings, for which the GPU
never throttles and thus yields stable performance.
On some of the test systems, the GPU draws more power when the temperature increases and starts throttling the core frequency when reaching the power limit. This leads to unstable performance, as the execution time of a kernel depends on the temperature when the kernel is executed. If the GPU has been idling before starting a kernel, the GPU temperature is low and the execution is fast. However, if other kernels have been executed before, the GPU's temperature and power draw are high, resulting in throttling, in turn resulting in a longer execution time.
The file expenv-paper applies contains per-host settings, for which the GPU never throttles and thus yields stable performance.