Closed berhane closed 1 year ago
@berhane,
Unfortunately, the dcgmi stats
are not supported on MIG-enabled configurations.
The stats functionality uses the Accounting mode under the hood, and it does not support MIG.
WBR, Nik
@berhane,
Unfortunately, the
dcgmi stats
are not supported on MIG-enabled configurations. The stats functionality uses the Accounting mode under the hood, and it does not support MIG.WBR, Nik
Thanks for the clarifications, @nikkon-dev. I'll keep looking for ways to get the "Max GPU Memory" used by a Slurm job running on a particular MIG device.
@berhane,
If you could use DCGM C API directly, DCGM can store watched metrics for a configured amount of time, and the API provides a way to get all stored values since some timestamp.
Unfortunately, either dcgmi tool or dcgm-exporter provides only the latest stored value.
Thank you, @nikkon-dev . I'll see if ChatGPT can write that code for me so that I don't have to learn DCGM's C API :). I wasn't lucky on my first try.
Hi,
I have a server with 4 A100-80GB GPUs and each GPU is partitioned to 3g.40gb, 2g.20gb and 2x 1.10gb partition. All the documentation at the following links suggests that we can use DCGM usage metrics for MIG partitions, but I have not been able to get any despite running the latest NVIDIA drivers/535.x, CUDA/12.2, DCGM/3.1.8 ... etc
https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#supported-profiles https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html?highlight=mig
I've tried with older NVIDIA drivers/495.x, CUDA/11.5, DCGM/2.4.1 with no luck.
1) I submitted a job using Slurm to use one 1g.10gb partition
DCGMI seems to see all the partitions
2) I created group ID 2 to monitor the relevant MIG device (instance 3)
3) I enabled monitoring and started collecting stats
4) I ran a GPU calculation
5) I monitored the usage as the calculation is running, but no relevant stats appear.
I would appreciate it if you would let me know if it is still possible to get statistics for MIG partitions. My main interest is to get the GPU memory utilization for every Slurm job so that I can advise users to submit their jobs to a MIG partition of the proper size. If there are other tools to accomplish the same task, I would love to learn about them as well.
Thanks.
Perhaps other relevant info:
OS:
Relevant Packages: