search

ocurrent / current-bench

Experimental benchmarking infrastructure using OCurrent pipelines
Apache License 2.0
33 stars 17 forks source link

Github bot messages #319

Open art-w opened 2 years ago

art-w commented 2 years ago

It's easy to forget checking the benchmarks on each PR, because it's generally not necessary.

Current-Bench could detect when a metric moves by more than 10% (?) and send a warning message to the PR: "You probably want to check those metrics!" For inspiration, see an example codecov report with pretty graphs embeded (but I think we should aim for a shorter message than this!)

Both regressions and improvements should be reported: a benchmark improvement can actually be indicative of a bug where real work has been skipped! And it's otherwise a nice confirmation that the PR fixed a performance issue :)

punchagan commented 2 years ago

See also #143 though this seems nicer to me.