Open bernhardmgruber opened 2 months ago
Another question is whether these new statuses should appear by default or you would want to have a CLI switch between the PASS
/FAIL
and the PASS
/SLOW
/FAST
.
I support this. The comparison script is still very much a work in progress and has much room for improvement. Clarifying whether the test was faster or slower would be a nice QoL change.
One suggestion: I don't think PASS
fits with FAST
and SLOW
anymore. Maybe SAME
for when the new samples are within the tolerance?
A related improvement would be to make the coloring more sensible, adding --color
and --no-color
flags to explicitly set the color state, and detect whether the output device supports color by default. IIRC we always use color, and this gets annoying when piping to a file, etc.
Currently, when I compare a baseline with new result JSON using
nvbench_compare.py base.json new.json
, I get output like the following:With
PASS
colored green, and `FAIL colored red. It would be more useful if the the Status column could not only highlight significant changes, but show me whether the timings improved or regressed. Therefore, I propose an output like this (only status column changed):With
PASS
uncolored,FAST
colored green. andSLOW
colored red.You can of course pick any better word for status, or different colors. But there is value in differentiating whether the time became faster or slower.