ocurrent / current-bench

Experimental benchmarking infrastructure using OCurrent pipelines
Apache License 2.0
33 stars 17 forks source link

Improve support for multiple configurations for a repository #453

Closed punchagan closed 1 year ago

punchagan commented 1 year ago

Currently, we can add multiple configurations for a repository to run different benchmarks (or to use different workers) for the same repository. For instance, the current configuration allows dune to run two different sets of benchmarks using different Dockerfiles using the following configuration:

    {
      "name": "ocaml/dune",
      "worker": "fermat",
      "dockerfile": "bench/monorepo/bench.Dockerfile",
      "notify_github": true
    },
    {
      "name": "ocaml/dune",
      "worker": "autumn"
    }

But, the status updates on GitHub are made to the same app with only the default URL. This makes the status of the benchmark runs unclear when one of the benchmarks is still running, while the other has finished (in this case, the monorepo benchmark, which takes longer).

GitHub Actions, for instance, add multiple status URLs for each different kind of CI run (for example, see this dune PR). We should do something similar where each configuration posts a different update to GitHub.