How to plot GPU usage % ?

[arjunskumar]

i can use prerecord for plotting CPU monitoring. Is there anyway to plot GPU usage % ?

[alexmyczko]

From https://github.com/alexmyczko/ruptime/blob/debian/ruptime you can easily get GPU usage in %, mind you it does not into account if you have mixed/different GPUs, but multiple GPUs in one node is fine:

s=$(uptime | sed "s/,//g" | awk -v c=$(getconf _NPROCESSORS_ONLN) '{printf "%6.2f\n", $(NF-2)*100/c}')
gpu=$(which nvidia-smi >/dev/null && printf "%6.2f %6.2f\n" $(timeout 29 nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv |sed 1d|sed "s/%,//g;s/MiB//g;s/,//g;s/  / /g" |datamash -t" " median 1-2 2>/dev/null))
m=$(printf "%6.2f\n" $(free -m -t |tail -1 |awk '{print 100*$3/$2}'))
printf "%-28s %s %s %s %s\n" "$k" "$s" "$m" "$gpu"

This will print CPU usage in %, memory usage in %, GPU usage in %, GPUmemory usage in %