ocurrent / current-bench

Experimental benchmarking infrastructure using OCurrent pipelines
Apache License 2.0
33 stars 17 forks source link

Bench old commits #482

Open ElectreAAS opened 7 months ago

ElectreAAS commented 7 months ago

Fixes #379.

~~Note: This PR requires some changes to the API of ocurrent (namely this PR: https://github.com/ocurrent/ocurrent/pull/442), and hence is in draft mode to make sure it doesn't get merged before the PR on ocurrent is properly reviewed. It is, however, ready to be reviewed in the current-bench context. The API changes in ocurrent are trivials from our point of view.~~

Edit: ocurrent/ocurrent#442 has been merged, this is ready for review

art-w commented 7 months ago

Thanks! This was a much awaited feature!

A couple of thoughts which prevents this from going live:

ElectreAAS commented 7 months ago
  • Can we disable it for repos that use a cron schedule? (ocaml/ocaml runs its benchmarks every week, precisely to avoid benchmarking too many commits)

I have no idea

  • Is the commit order preserved when plotted on a graph?

I also have no idea, but it seems testable. I don't have time to allocate to this nowadays, but someone else could.

  • How do we get rid of old commits that were removed? (e.g. following a PR rebase)

This PR does not take that scenario into account.