IanHoang
commented
1 year ago Last week, I've setup two EC2 instances, one that triggers a test with traditional ingestion and subsequent search and another that runs snapshot restoration and subsequent search. Both tests target clusters of the same configuration (OpenSearch 2.9 single node clusters) and ingest their metrics and results into the same metrics store. Based on the last 5 days that the setup has run, runs that use snapshot restoration are showing less variance compared to runs doing traditional ingestion.
asc_sort_size was the only sorting operation that showed that it's variance was worse than traditional ingestion. Will gather more results in the next week to determine if the variance changes.
Follow ups
- Validate accuracy of data: Since this data is based off of benchmark-results documents that are published to the datastore, we should aim to reproduce it with documents from benchmark-metrics documents. Benchmark-results documents contain aggregations of metrics from benchmark-metrics documents.
- Determine the resource differences (CPU and JVM) from the two types of runs: Since tests are being run with
node-statstelemetry device, the datastore contains documents related to both test clusters' CPU and JVM usage. We should do an analysis on these to see if we can spot any patterns. - Run profilers during the tests: Profilers will help us understand where the most time is spent for each operation.
msfroh
Synopsis and Motivation
OpenSearch Benchmark is currently running nightly runs with various configurations. In a few cases, some queries have been detected to have more variance than others. For example, a nightly run on a multi node cluster with the
http_logsworkload has shown variance forsortqueries, as depicted below.Variances in test runs prevent the community from accurately identifying regressions. The goal of this RFC is to find optimal OpenSearch configurations and rule out any causes of variances, which will in turn improve the effort of obtaining consistent and reproducible numbers.
Questions
We should aim to answer the following:
Strategy
Instead of using the data from nightly runs (or at benchmarks.opensearch.org), we should replicate the nightly runs setup but on a smaller scale. Our setup will include the following characteristics:
Workload:
http_logsOpenSearch versions:2.9(or any other stable release should be sufficient) Test Clusters configurations (subject to change throughout the process):The tests will be run on a nightly basis and will replace ingestion phase with restoring from snapshot. This should have a couple benefits. First, it should reduce the overall testing time. Second, it could potentially reduce the query variance since it's suspected that the ingestion phase can be inconsistent and influence query performance. Metrics and test results will be channeled into a external metrics datastore, which will be used to build visualizations for each query. These visualizations will give us a better idea of which queries have variance and will spawn deeper investigations.
Next Steps
Preliminary Testing
Further Testing
--telemetry=node-statsto gather CPU and JVM metrics. A profiler should also be enabled on each cluster so that we can understand where most of the time is spent during each search operation in each cluster.Some Additional Areas to Check
Current Experiments
How Can You Help?