[Question] Memory bandwidth utilization of GPUs?

[walkieq]

Required prerequisites

• [X] I have read the documentation https://nvitop.readthedocs.io.
• [X] I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
• [X] I have tried the latest version of nvitop in a new isolated virtual environment.

Questions

Is there a way to measure the runtime memory bandwidth utilization of GPUs? Or is it possible to estimate/calculate the memory bandwidth utilization using the numbers reported in nvitop?

[XuehaiPan]

Is there a way to measure the runtime memory bandwidth utilization of GPUs?

@walkieq The memory bandwidth utilization rate can be retrieved by:

from nvitop import Device

devices = Device.all()
for device in devices:
    print(f'Memroy usage percentage:           {device.memory_percent() / 100:%}')
    print(f'Memory bandwidth utilization rate: {device.memory_utilization() / 100:%}')

Also, you can get the memory throughput via:

device.pcie_throughput()
device.pcie_tx_throughput()
device.pcie_rx_throughput()
device.pcie_tx_throughput_human()
device.pcie_rx_throughput_human()

device.nvlink_throughput()
device.nvlink_tx_throughput()
device.nvlink_rx_throughput()
device.nvlink_tx_throughput_human()
device.nvlink_rx_throughput_human()

[walkieq]

Thank you so much! I have tested the API but I got some interesting results.

I am running the following:

from nvitop import Device

devices = Device.all()
for device in devices:
    print(f'Memroy usage percentage:           {device.memory_percent() / 100:%}')
    print(f'Memory bandwidth utilization rate: {device.memory_utilization() / 100:%}')
    print(device.pcie_throughput())
    print(device.pcie_tx_throughput())
    print(device.pcie_rx_throughput())
    print(device.pcie_tx_throughput_human())
    print(device.pcie_rx_throughput_human())

and the results are:

Memroy usage percentage:           94.600000%
Memory bandwidth utilization rate: 0.000000%
ThroughputInfo(tx=177000, rx=177000)
175000
182000
177.7MiB/s
168.9MiB/s

It looks like the device.memory_utilization() is different to the device.pcie_throughput(). Which one shall I refer to?

[XuehaiPan]

It looks like the device.memory_utilization() is different to the device.pcie_throughput(). Which one shall I refer to?

Device.memory_utilization refers to:

Percent of time over the past sample period during which global (device) memory was being read or written.

The sample period may be between 1 second and 1/6 second depending on the product.

Device.pcie_throughput refers to:

The current PCIe throughput in KiB/s.

This function is querying a byte counter over a 20ms interval and thus is the PCIe throughput over that interval.

You can see the detailed definition via:

nvidia-smi --help-query-gpu | less --pattern=utilization

Both the memory utilization rate and the PCIe throughput have their own meaning. You can also calculate the PCIe bandwidth utilization rate by:

PCIe utilization rate = 100 * current PCIe throughput / theoretical PCIe throughput (via PCIe generation and PCIe width)

[XuehaiPan]

Closing due to inactivity. Please feel free to ask for a reopening if you have more questions.