Where can I find benchmark results?

cupy / cupy-performance

5 stars 3 forks source link

Where can I find benchmark results? #7

Open marcinz opened 2 years ago

marcinz commented 2 years ago

I am wondering if the benchmarks in this repo (or some other cupy benchmarks) are run regularly somewhere. If they are, is there a place where I can find the results?

emcastillo commented 2 years ago

They aren't 😄

kmaehashi commented 2 years ago

This is in our TODO list for a long time... How can we make it public? I'm thinking of:

For every release, run a benchmark against the wheel. Persist the raw benchmark results and rendered HTML to GCS.
For every pull-requests, run the benchmark against the code, then compare with the raw benchmark results in GCS, then persist rendered HTML to GCS.

leofang commented 2 years ago

@kmaehashi FYI: https://github.com/spcl/npbench

jakirkham commented 2 years ago

Interesting hadn't heard of that one before.

Another popular one is airspeed velocity (asv), which NumPy & SciPy both use to run and show benchmarks. Here's NumPy's benchmark results.

jakirkham commented 2 years ago

cc @pentschev (in case you have thoughts here)

pentschev commented 2 years ago

I agree ASV is potentially the best candidate for a library like CuPy as it will provide a historic view on a per-commit basis, making it really useful to catch regressions too.