search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
28.53k stars 6.31k forks source link

Changes in noise in regression test results #12288

Open ajkr opened 9 months ago

ajkr commented 9 months ago

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

The performance regression results have low noise

Actual behavior

They fluctuate a bit, except during August - mid-November 2023. I wonder what happened then to make the results so stable, and whether that situation can be recreated.

Steps to reproduce the behavior

The dashboard is here - https://search-rocksdb-bench-k2izhptfeap2hjfxteolsgsynm.us-west-2.es.amazonaws.com/_dashboards/app/dashboards#/view/9efe4fc0-fdd9-11ec-9f1d-b7bf4fe8d037. AFAIK it isn't publicly available, but here's a sample -

Screenshot 2024-01-24 at 12 59 34 PM

adamretter commented 2 months ago

@ajkr The Continuous Benchmarking has now been re-enabled after we just migrated it from CircleCI to GitHub Actions. Could you take a look at provide us with any updates/feedback please (cc @rhubner @alanpaxton)