[FEA] Use NVML to manage clocks, etc

[alliepiper]

Adding NVML as an optional dependency would allow some cool features:

• Lock clock frequency.
  • Per-device default frequency.
  • Per-device maximum frequency.
  • Explicit frequency.
• Log various device stats per measurement
  • SM/Mem clock frequencies.
  • Device utilization
  • Power state/usage
• Check throttle state after each measurement.
  • Log a warning with the throttle reason and details (e.g. for thermal throttle, show current temp and thresholds).

[cliffburdick]

@allisonvacanti does it make sense to also print out an error/warning if the clocks aren't fixed? I know google bench has similar printouts for CPU governor settings.

[alliepiper]

@cliffburdick Yes, that is something I plan to add if the NVML APIs provide this information.

[alliepiper]

46 added the basics:

• Got NVML dependency sorted out
• Locking clocks to base/max implemented for Volta (SM 7.0) and above
• Toggling persistence mode on Linux

Still a lot more we can do here, though. Unassigning myself for now since I'll be busy with other work for a while.

[saurabh-singh-rajput]

is there a way to "Log various device stats per measurement"?

[alliepiper]

Not currently, I'd still like to get that in eventually.