search

NVIDIA / cuda-samples

Samples for CUDA Developers which demonstrates features in CUDA Toolkit
Other
6.28k stars 1.79k forks source link

GPU power consumption doesn't go back to idle state after CUDA finishes GeMM execution #74

Open afzalxo opened 3 years ago

afzalxo commented 3 years ago

I am running some experiments using NVML and CUDA GeMM implementation for power consumption. I measured the following trend of power consumption for multiplication of two 16384 sized square matrices. The horizontal axis is time in seconds while the vertical axis is power in Watts. The measurements are made on a single Tesla V100S.

Screenshot 2021-04-21 at 8 29 58 PM

As can be seen from the plot, consumption in idle state before the CUDA kernel is called is roughly 25W. After the CUDA kernel finishes and memory is deallocated, the power settles at roughly 50W instead of the original 25W idle state value that it should settle to. In the "Finished" state, the GPU is totally free, all variables deallocated. The measurements are made using NVML C++ API.

Is there something that I'm missing here?

Thanks

afzalxo commented 3 years ago

To add to this, the consumption returns to 25W several seconds after the deallocation phase.