search

rust-vmm / rust-vmm-ci

Apache License 2.0
18 stars 33 forks source link

Benchmarks: add test that can run at every PR #25

Closed alxiord closed 4 years ago

alxiord commented 4 years ago

24

This PR introduces a new integration test that can be included in per-crate Buildkite pipelines to run the crate's benchmark tests and assess the impact of a PR. The test runs cargo bench on the tip of the upstream crate and on the HEAD of the PR and compares the results with critcmp. For the moment, the test only prints the results.

alxiord commented 4 years ago

What's the plan going forward?

  1. Merge just the bench tests in some crate. linux-loader or vm-memory are good candidates. This is needed for the very first run of the test, when it tries to cargo bench the upstream tip.
  2. Merge this PR to have a starting point, and iterate on it.
  3. Figure out an acceptable tolerance for the bench numbers, or if we trust critcmp enough to give it authority to fail a PR. Use the printed output to get a sense of how the test behaves in the wild.
  4. Update the test with PASS/FAIL conditions in a new PR.

Does it make sense?

alxiord commented 4 years ago

Please also update the README.md to specify how to run this test and how it works.

Done, PTAL