Automated Performance Tests

[SaveTheRbtz]

Goal

We want to have a standardized automated set of performance benchmarks.

Milestones

• [x] Add ability to test different commits and change the BigQuery schema accordingly: https://github.com/onflow/flow-go/pull/3633
• [x] Recreate dashboards for TPS tests utilizing new schema and split into two:
  • [x] Raw data for one-off tests and overall system monitoring: https://dapperlabs.grafana.net/d/BLSHi8ZVz/flow-performance-benchmarks?orgId=1&from=now-90d&to=now&var-branch=All&var-env=All&var-type=All
  • [x] Aggregated data for performance analysis: https://dapperlabs.grafana.net/d/mA42tHK4z/flow-performance-benchmarks-raw?orgId=1&from=now-24h&to=now&var-branch=All&var-env=All&var-type=All
• [x] Improve error handling in benchmark code: https://github.com/onflow/flow-go/pull/3720
• [x] (optional) speedup test setup: https://github.com/onflow/flow-go/pull/3641
• [ ] Add gRPC API to the benchmark/ci and make it a daemon.
• [ ] Add CLI interface to the daemon so we can control it manually.
• [ ] Move control code out of dataplane an use gRPC interface to drive tests.
• [ ] Add a CLI to control the scheduling (e.g. submit ad-hoc tests)

Full plan here: https://www.notion.so/dapperlabs/TPS-Benchmark-Plan-36e2109bfe7c4f13946c7671217721bc

Previous art

• Vitess nightly: https://benchmark.vitess.io/macrobench
• Rust Compiler Nightly: https://perf.rust-lang.org/

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.