Open ZzEeKkAa opened 1 year ago
@ZzEeKkAa great initiative, such a feature is essential.
Putting here some of the tools I was referring to when previously I was thinking of this work:
Few comments on your list of metrics:
We should decide decide what benchmarks, tests make sense here. I will like to see standard benchmark suites: npbench, Rodhinia, microbenchmarks, and important apps like @fcharras' kmeans implementation
For performance reporting, I do not think it is that useful. Unless, it was a known good test that is now failing or timing out.
For the last three we should look at the https://pv.github.io/numpy-bench/ created using airspeedvelocity
Another important question is where do we run the CI run. We can start with github default CPU runners, but I will see if we can do better.
There is no performance reporting, so we don't know how good we are doing on weekly basis.
Proposal: Create github CI, to run dpbench weekly and add performance report to gh pages using default github runners.
List of metrics to be added:
Timeseries on average timing improvements of numba_dpex agains numpy.
Could be calculated as
avg[ dpex_time/numpy_time - 1 ]
. Positive number will indicate improvements.