Ganglia GPU Monitoring Enhancements

[ranacseruet]

Metrics Added:

• ‘_ecc_db_error’ : Double Bit ECC Error.
• ‘_ecc_sb_error’ : Single Bit ECC Error.
• ‘_power_violation_report’ : GPU Power violation.
• ‘_encoder_util’ : Encoder Utilization.
• ‘_decoder_util’ : Decoder Utilization.
• ‘_bar1_memory’ : Bar1 Memory.
• ‘_shutdown_temp’ : Shutdown temperature.
• ‘_slowdown_temp’: Slowdown temperature.

Metrics Modified:

• ‘_graphics_speed’ to ‘graphics_clock_report’. Reason: Changed ‘speed’ to ‘clock’ for better meaningful name. Added ‘_report’ for custom php graph file support.
• ‘_sm_speed’ to ‘_sm_clock_report’. Reason: same as above.
• ‘_mem_speed’ to ‘_mem_clock_report’. Reason: same as above.
• ‘_mem_used’ to ‘_fb_memory’. Reason: match with NVML and eliminate ambiguity between ‘Frame Buffer Memory’ and ‘Bar1 Memory’.
• ‘_power_usage’ to ‘_power_usage_report’. Reason: custom php graph file support.
• ‘_max_graphics_speed’ to ‘_max_graphics_clock’. Reason: Changed ‘speed’ to ‘clock’ for better meaningful name.
• ‘_max_sm_speed’ to ‘_max_sm_clock’. Reason: as above.
• ‘_max_mem_speed’ to ‘_max_mem_clock’. Reason: as above. Moved ‘_max*’ metrics from graph to top constant metrics section(host overview).

Metrics Deleted:

• power_man_mode.
• perf_state.

Custom Graph Modifications:

• Clear old php graph files which wasn’t working.
• Generalization of No# Of GPUs for which custom graphs can be supported. Now, single php file can render custom graph for each GPU#.
• Added Max line on some graphs to provide better overview of limit/usage ratio.

[dpocock]

Rana, Github says there is a merge conflict. Can you provide an additional commit on your branch to resolve the merge conflict? Maybe somebody else has change one of the files while you were waiting for this to be accepted.

[dpocock]

Link to email discussion: http://www.mail-archive.com/ganglia-developers@lists.sourceforge.net/msg06725.html

Link to report: http://www.mail-archive.com/ganglia-developers@lists.sourceforge.net/msg06725/GPU_Monitoring_Enhancement_Report-_Draft.pdf

[ranacseruet]

Thanks Daniel. I have merged with the master branch and seems now it can be merged automatically.

[eshelman]

@ranacseruet This is good - I didn't realize we were missing this many metrics!

Would you please explain why you removed the power_man_mode and perf_state metrics? Those metrics seem to be working fine on our systems.

[ranacseruet]

@eshelman , thanks for reviewing! Actually most of the new metrics are from taken from new NVML version.

To answer your question, I have removed those metrics based on suggestion/discussion with my mentor Rajat Phull(rphull@nvidia.com). The reasons are given below:

power_man_mode: For the current generation and some of the recent generations of GPUs, the Power management module is always available on the devices. So we thought to turn it off. If needed by community, we can enable this.

perf_state: We thought that looking at current clocks gives a better representation of the performance state, and giving pstate in constant/host section may not add a lot of value. But again, if community needs it, we will get it back.

[eshelman]

Thanks for the details! Your explanation makes sense - these changes wouldn't be a problem for us.

[ranacseruet]

That's great then. Thanks!

[dpocock]

Thanks for this contribution Rana