Integrate performance testing for individual PRs so core devs benefit from the performance tests

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

https://redpanda.com

9.45k stars 579 forks source link

Integrate performance testing for individual PRs so core devs benefit from the performance tests #15584

Closed mfleming closed 2 months ago

mfleming commented 9 months ago

[x] Run microbenchmarks as part of the "merge to dev" step
[x] Send Nyrkiö change points to performance team slack channel ~~- [ ] Add comments to PRs when change points are detected~~
[x] Run microbenchmarks as part of PR to do outlier detection using hunter open source tool ~~- [ ] Include public url in slack msgs so core devs don't need to login~~
[x] Store results in perf db

JIRA Link: CORE-1655

mfleming commented 7 months ago

The current plan is to send Nyrkiö results to a Slack channel instead of sending them to a (likely already closed) PR.

mfleming commented 6 months ago

For the microbenchmarks in PRs task the team would like to

have a blessed set of microbenchmarks to run (currently just produce bench?)
Only display test results when a performance change is detected
Give some indication that perf tests ran (e.g. "Ran X/Y microbenchmarks successfully")
compare performance against the merge base, and issue a warning if there's no result yet (i.e. if the merge-to-dev microbenchmarks haven't completed)
Make sure all PR microbench data is persisted in the perf db

mfleming commented 6 months ago

For nightly perf tests (OMB) the expectation is that they'll run as part of the nightly buildkite job and any detected change points can be sent to a slack channel.

StephanDollberg commented 5 months ago

@ballard26 can you please update with what still needs doing etc. or close if good to do so.

StephanDollberg commented 2 months ago

Issues got kinda out of sync. Closing this one for now.