search

NVIDIA / Bobber

Containerized testing of system components that impact AI workload performance
MIT License
14 stars 5 forks source link

Add "high" value and "low" value to "avg" value output #61

Open laytonjbgmail opened 3 years ago

laytonjbgmail commented 3 years ago

Bobber will print out the average for performance metrics when run several times. Can the "high" and "low" values also be printed out with the averages? Something like

low: xx.xx avg: xx.xx high: xx.xx

This would probably only for 3 or more runs.

Printing out the high and low may be easier for people to digest than something like standard deviation.

roclark commented 3 years ago

Great suggestion @laytonjbgmail! Should be fairly easy to handle and I agree that lows and highs are probably a better call over standard deviations. Do you have any thoughts on what the output would look like? The default behavior of the parser right now is to display a table of the results. Are you thinking it would be good to add additional columns for the lows and highs (or even include them in the box)? Or would this only be for the --verbose output which displays the text-based information?

laytonjbgmail commented 3 years ago

Personally, I would stick to the tabular output (that looks nice). I would just add columns for each. Personally, I would order them from left to right as "min", "avg", "max". But if you want to put min and max after avg that would be good too.

Again, personally, I would output the min and max in any form if there are 3 or more iterations. I find knowing the min, avg, max to be better than just avg.

Thanks!

joehandzik commented 3 years ago

This also functions as a bit of a starter bridge to verifying that performance doesn't drop over time - could we perhaps also consider printing which iteration of the test produced the max and the min? That'd help give insight not only to the peaks and valleys but also flag if the low results tended to be towards the end of testing, or if it was somewhat random across the test history.