How to get the maximally used GPU memory during a period?

gpuopenanalytics / pynvml

Provide Python access to the NVML library for GPU diagnostics

BSD 3-Clause "New" or "Revised" License

205 stars 31 forks source link

How to get the maximally used GPU memory during a period? #27

Closed JunMa11 closed 3 years ago

JunMa11 commented 3 years ago

Dear all,

Thanks very much for sharing such a great tool.

When running a deep learning model, I want to measure the maximally used GPU memory during a period (e.g., 10 mins).

How can I achieve this goal?

Any comments would be highly appreciated.

Kindest regards, Jun

quasiben commented 3 years ago

I think you would need to sample the GPU and record the max. Here is an example which stores the peak_mem with a user defined sampling period: https://gist.github.com/pentschev/2e2e3fe8059240f2679b6f7002faa891

rjzamora commented 3 years ago

I agree with @quasiben - Sampling the memory in a loop is probably the best way to go, since it is only possible to query the memory usage at an "instant" in time with nvml (I think).

JunMa11 commented 3 years ago

@quasiben @rjzamora Got it. Thank you so much for your kindly help.