Closed mrocklin closed 3 years ago
Some useful nvidia-smi queries from the web
The main things I want to see dashboarded over time are the utilization % numbers:
gpu_util
memory_util
If they mean what I think they do, the memory transfer counters would also be super useful:
tx_util
rx_util
gpu_temp
might be interesting to detect hardware issues if you are working with less well engineered servers and workstations. Power consumption is entertaining, but not really important to understand performance.
Thanks @seibert ! I missed the tx_util
and rx_util
values entirely in there. Thanks for highlighting them.
Here is what the nvidia-smi API in this library produces for a single GPU. What of this information is useful and would we maybe want to make dashboards out of?
cc @seibert @kkraus14 @sklam @randerzander