NVIDIA / nvbench

CUDA Kernel Benchmarking Library
Apache License 2.0
510 stars 64 forks source link

Python Comparison Scripts #9

Open alliepiper opened 3 years ago

alliepiper commented 3 years ago

NVBench has a work-in-progress JSON output format and I'm working on a very basic python script to compare two JSON files.

We should grow this functionality into a more complete set of analysis tools. At minimum, this should cover the features provided by Google Benchmarks' excellent comparison scripts.

If anyone is interested in writing some python to help with this, let me know. I'll update this issue once I have finalized the JSON output format.

Basic Regression Testing

These should:

Analysis modes

Compare benchmarks with different names. Answers questions:

These will need some way of specifying the sets of configurations to compare. Google benchmark has worked out a general syntax for specifying this, we should adapt what they've done to use the NVBench axis syntax.

Output

Ideally markdown formatted, similar to NVBench's default output.

References

shwina commented 3 years ago

I'd be interested to help out here!

vyasr commented 3 years ago

I can help as well if you need extra hands.

alliepiper commented 3 years ago

Initial work is in NVIDIA/thrust#14.

vyasr commented 2 years ago

@allisonvacanti what's next here? NVIDIA/thrust#14 helped close the gap, but I don't recall exactly how far it got us or what we still need to do. RAPIDS is making a push to formalize and analyze our benchmarks more, so migrating fully to nvbench is probably going to become a priority in the near future and I'm happy to help out in making sure that we have sufficient feature parity with google bench. CC @shwina in case you want to continue being involved too.

robertmaynard commented 2 years ago

Basic thresholding and comparing multiple files was added in NVIDIA/thrust#48

alliepiper commented 2 years ago

There's a lot we could still do, such as filtering the results by benchmark name/index and axis values. But I don't think these are essential right now.

Are there any "must have" features for RAPIDS that we're missing?

BTW, I'm working on a branch that makes some changes to the JSON file layout to make things more consistent. I hope to have that merged by the end of the week, time permitting 🤞

vyasr commented 2 years ago

I assume the changes you're referring to are NVIDIA/thrust#70? It looks great! 🎉

@robertmaynard @jrhemstad @harrism any thoughts on what we would need to see in nvbench to make the transition from gbench smooth for RAPIDS?

jrhemstad commented 2 years ago

The gbench compare script had the ability to do a U Test between two samples to determine if there was a statistically significant difference between the populations. Do we have anything like that in nvbench yet?

It's very helpful when looking at small differences in performance to establish if the difference is just "noise" or actually meaningful.

alliepiper commented 2 years ago

@vyasr Yep! That PR has all of my pending changes to the JSON/Python stuff.

@jrhemstad We don't have anything like that at the moment.


One feature that I'd like to see at some point is the ability to compare performance between different benchmarks that use the same axes.

For example, see NVIDIA/cccl#720, which points out that thrust::all_of is slower than thrust::count_if. It'd be nice to be able to write some automated tests that check the performance of equivalent algorithms and identify these sorts of issues.