search

NVIDIA / go-nvml

Go Bindings for the NVIDIA Management Library (NVML)
Apache License 2.0
311 stars 64 forks source link

How to understand the SmUtil returned by nvmlDeviceGetProcessUtilization? #51

Open nezhababa opened 1 year ago

nezhababa commented 1 year ago

I'm writing a program that monitors how processes use the GPU and I found an API provided by nvml, nvmlDeviceGetProcessUtilization.

Acordding the comment of this API, It reads recent utilization of GPU SM (3D/Compute), framebuffer, video encoder, and video decoder for processes running.

I called the API every 10 seconds or 1 seconds and printed out the samples, as follows:

The "ReadTime" indicates when my program called the API. The "sample" are the samples returned by the API.

ReadTime:10:58:56.194 - sample:[Pid:28128, Timestamp:10:58:55.462519, SmUtil:05, MemUtil:01, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:28104, Timestamp:10:58:55.127657, SmUtil:05, MemUtil:02, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:28084, Timestamp:10:58:48.051124, SmUtil:03, MemUtil:01, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:28050, Timestamp:10:58:53.944518, SmUtil:03, MemUtil:01, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:27989, Timestamp:10:58:47.043732, SmUtil:03, MemUtil:01, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:27976, Timestamp:10:58:53.604955, SmUtil:09, MemUtil:03, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:27814, Timestamp:10:58:48.386200, SmUtil:19, MemUtil:07, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:27900, Timestamp:10:58:56.132879, SmUtil:17, MemUtil:06, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:27960, Timestamp:10:58:51.423172, SmUtil:06, MemUtil:02, EncUtil:00, DecUtil:00] ReadTime:10:58:56.194 - sample:[Pid:27832, Timestamp:10:58:47.883811, SmUtil:21, MemUtil:08, EncUtil:00, DecUtil:00] SUM - GPUId:0, process:10, smSum:91, memSum:32, encSum:0, decSum:0

My question is:

  1. Why is there only one sample per process in the samples returned by this API, regardless of whether I call it once every second or every 10 seconds?
  2. The timestamps in each sample seem to be almost irregular. How does nvml determine the sampling time?
  3. How is SmUtil derived? According to the description of nvmlUtilization_st struct in the nvml.h header file, it is used by nvmlDeviceGetUtilizationRates. The GPU usage refers to “Percent of time over the past sample period during which one or more kernels was executing on the GPU”. As my understanding, that is, even if the GPU has multiple cores, if only one core in the GPU is occupied in a time slice, the entire CPU is considered occupied. This time slice is used as a numerator to calculate the utilization of the entire GPU. So, how do you understand the SmUtil of the process returned by nvmlDeviceGetProcessUtilization?