ocurrent / current-bench

Experimental benchmarking infrastructure using OCurrent pipelines
Apache License 2.0
33 stars 17 forks source link

WIP: Look for changed significantly metrics and notify authors #358

Closed punchagan closed 2 years ago

punchagan commented 2 years ago

This PR looks for PRs where metrics have changed significantly and attempts to post messages to the corresponding Github PR to notify authors to check the metrics.

punchagan commented 2 years ago

A sample bot message/comment would look like this:


#2 (40a7f39) changes the metrics as follows in comparison to main (b5a7c3b) when running on autumn (ocaml/opam:debian-11-ocaml-4.13):

Benchmark: default

Test: bench_1_test_1

Test: bench_1_test_2

art-w commented 2 years ago

So cool! Would it be possible to add a setting per repository to opt-in this feature? (I would like to test it in production on safe repositories before enabling it for everyone ^^' )

punchagan commented 2 years ago

Would it be possible to add a setting per repository to opt-in this feature?

I've added a config option that can be added to development.conf or production.conf to opt-in for Github notifications, per repository.

punchagan commented 2 years ago

image

I see some new empty boxes added in the graph, and with an initial look at the API, I wasn't sure how to hide or label them. Could you help with that, @art-w ? :see_no_evil:

art-w commented 2 years ago

I think the empty-boxes are due to the various let* that we should eventually get rid of (but weren't introduced by your nice PR!)

https://github.com/ocurrent/current-bench/blob/37ad8acc485f6fe70ebd0a56d3c91b2679b1d5d9/pipeline/lib/pipeline.ml#L147

art-w commented 2 years ago

Thanks a lot for all the changes!