ocurrent / current-bench

Experimental benchmarking infrastructure using OCurrent pipelines
Apache License 2.0
33 stars 17 forks source link

Some benchmarks seem to run multiple times #360

Open punchagan opened 2 years ago

punchagan commented 2 years ago

@art-w reported that:

For example, this benchmark seems to run more than expected https://autumn.ocamllabs.io/Zineb-Ada/bechamel-fact?worker=autumn&image=ocaml%2Fopam%3Adebian-11-ocaml-4.13

I tried to debug/understand this issue further, and here's some debugging information:

         run_at         |                  commit                  |         repo_id         | worker |          docker_image
------------------------+------------------------------------------+-------------------------+--------+---------------------------------
 2022-05-12 13:32:33.7  | 424ae1762b04c61510a9a91505c5d11086ff03de | Zineb-Ada/bechamel-fact | autumn | ocaml/opam:debian-11-ocaml-4.13
 2022-05-16 17:10:20.01 | 424ae1762b04c61510a9a91505c5d11086ff03de | Zineb-Ada/bechamel-fact | autumn | ocaml/opam:debian-11-ocaml-4.13
 2022-05-23 17:10:36.8  | 424ae1762b04c61510a9a91505c5d11086ff03de | Zineb-Ada/bechamel-fact | autumn | ocaml/opam:debian-11-ocaml-4.13

I'm not sure what else could be causing the bug. @art-w @Zineb-Ada do you have any other ideas or hunches on what could possibly be going on?

punchagan commented 2 years ago

I found a bunch of similar other instances, like

# select run_at,commit,repo_id,worker,docker_image from benchmarks where commit = '5f9a76290089a5f80b5103879941ebe4a6c35350' and test_index=0;
         run_at         |                  commit                  |        repo_id        | worker |          docker_image           
------------------------+------------------------------------------+-----------------------+--------+---------------------------------
 2022-03-14 12:03:32.21 | 5f9a76290089a5f80b5103879941ebe4a6c35350 | ocaml-ppx/ocamlformat | autumn | ocaml/opam:debian-11-ocaml-4.13
 2022-03-14 17:05:30.48 | 5f9a76290089a5f80b5103879941ebe4a6c35350 | ocaml-ppx/ocamlformat | autumn | ocaml/opam:debian-11-ocaml-4.13
 2022-03-21 17:06:18.77 | 5f9a76290089a5f80b5103879941ebe4a6c35350 | ocaml-ppx/ocamlformat | autumn | ocaml/opam:debian-11-ocaml-4.13