ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

Add GPU info to like_top.py #112

Closed telegraphic closed 5 years ago

telegraphic commented 6 years ago

For my own purposes, I modified like_top to report GPU usage. Could be added in, but needs better handling of multiple GPUs, and if it's a CPU-only build.

Here's the code gist:

def _getGpuMemoryUsage():
    """ Grab nvidia-smi output and return a dictionary of the memory usage. """
    q_flag   = '--query-gpu=memory.used,memory.total,memory.free,power.draw,power.limit'
    fmt_flag = '--format=csv,noheader,nounits'
    p = Popen(['nvidia-smi', '-i', '0', q_flag, fmt_flag],
                   stdin=PIPE, stdout=PIPE, stderr=PIPE)
    output, err = p.communicate()
    # Parse the line and turn them all into integers
    output = map(int, map(float, output.strip().split(',')))
    data_label = ['memUsed', 'memTotal', 'memFree', 'powerDraw', 'powerLimit']
    data = dict(zip(data_label, output))
    return data

And then in the display loop:

gpu  = _getGpuMemoryUsage()
...
### GPU memory usage
output  = 'GPU:  %8iMi total, %8iMi used, %8iMi free' % (gpu['memTotal'], gpu['memUsed'], gpu['memFree'])
output += '   Power:  %i/%i W\n' % (gpu['powerDraw'], gpu['powerLimit'])
k = _addLine(scr, k, 0, output, std)

And then you get this extra line:

screen shot 2017-12-01 at 4 40 02 pm
jaycedowell commented 6 years ago

That's a cool idea. It would also be good if we could add in GPU utilization somehow. I think nvidia-smi only supports that for computing cards but maybe there is a way to get around it. Is power usage a good proxy of utilization?

jaycedowell commented 6 years ago

Oh, it looks like the gaming cards don't provide the power draw through nvidia-smi.

jaycedowell commented 6 years ago

I made a few changes to like_top.py to address this in 1e61eda. It should be able to handle CPU-only builds and the limited nvidia-smi information you get on some cards.