search

opensearch-project / opensearch-benchmark

OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch
https://opensearch.org/docs/latest/benchmark/
Apache License 2.0
104 stars 74 forks source link

[RFC] System Under Test Provisioning Expansion #113

Open engechas opened 2 years ago

engechas commented 2 years ago

Background

OpenSearch Benchmark supports creating a self-managed OpenSearch cluster on the same machine running OpenSearch Benchmark. Clusters can be created by building OpenSearch from source, downloading a distribution, or using a docker image. A variety of cluster settings can be specified such as heap size, JVM flags, and the garbage collection algorithm.

Proposal

OpenSearch Benchmark’s system under test provisioning capabilities will be expanded to support cloud infrastructure creation. This will allow clusters to be provisioned on infrastructure external to the machine running OpenSearch Benchmark.

Implementations for self-managed OpenSearch running on EC2 instances and managed Amazon OpenSearch Service #100 are in scope for this project. The existing functionality to provision a cluster on the local machine will continue to be supported.

Design

There will be no change to the existing execute_test workflow. The BuilderActor will continue to be invoked as the first step of the BenchmarkActor’s workflow. The WorkerCoordinatorActor will not receive a request to start the test until the BuilderActor has notified the BenchmarkActor that the cluster is ready. Once the test has completed, the BuilderActor will receive a request to delete the cluster from the BenchmarkActor.

image (2)

The Builder System will consist of the core logic required to provision a local cluster running self-managed OpenSearch. Additional plugins can be defined to provision cloud infrastructure when creating a cluster. Eventually these additional plugins will be moved to a separate repository and fetched via a PluginLoader #98. In the short term, any infrastructure-provider plugins will be implemented as modules within OpenSearch Benchmark’s Builder System.

image (3)

The existing NodeBuilderActor class will be removed in favor of handling the provisioning logic within the Core Builder. The Dispatcher will remain an intermediary step between the builder plugins and the BuilderActor in order to prevent the CreateCluster call from blocking the BuilderActor while provisioning is underway. Based on the user input, the Core Builder will determine if a provisioning plugin is required and load it for use by the Builder System.

Builder System

The Builder System is responsible for orchestrating the logic to create and delete a system under test. It consists of multiple components that are executed sequentially to perform the provisioning logic. The top-level class and the interface into the Builder System is a Builder.

A Builder has four primary components that handle the provisioning and deprovisioning work. The three existing components, the Downloader, Installer, and Launcher will continue to be used. A fourth component, called the Provisioner, will be created to handle the infrastructure logic.

image (4)

Builder plugins can create new concrete implementations of the Provisioner, Downloader, Installer, and Launcher for use in the Builder class. The below class diagram shows a sample of the Core Builder and AWS Builder Plugin’s component implementations that feed into the Builder class.

image (5)

User Experience

A new version (v2) of provision config instances will be created for the Builder System expansion. The v2 configs will incorporate all of the existing options available in v1 configs as well as new options necessary to provision cloud infrastructure. v2 configs will be relied on as the central location for defining a cluster, removing any and all definitions in other locations, such as benchmark.ini and base config.ini files.

The existing CLI options related to provisioning will remain unchanged, with the exception of the --pipeline parameter. If the new builder system is specified by the user, --pipeline=benchmark-only will continue to be used to specify a pre-provisioned cluster. Any other value, or the omission of this parameter, will result in a cluster being provisioned and the parameter value being ignored in favor of the cluster definition in the provision config instance.

Provision Config Instances

In order to simplify the user experience, all mandatory provision config options will have a sensible default value defined in the Core Builder. Each Builder Plugin will have a unique default provision config instance file where the variables section can defined. These variables will be additive to the default provision config instance file in the Core Builder. Users will be able to define their own provision config instances and mixins with as few or as many config options as needed to define their use case.

Additional CLI Commands

The existing top-level OSB CLI commands related to provisioning a cluster outside of a test will remain unchanged. Two new commands will be added to invoke the infrastructure provisioning and deprovisioning workflow.

opensearch-benchmark provision ...
opensearch-benchmark tear_down ...

The list CLI parameter will be expanded to support listing the registered builders.

opensearch-benchmark list builders

Tasks

Appendix

Naming Changes

bbarani commented 1 year ago

@rishabh6788 @gkamat @IanHoang Can we use opensearch-cluster-cdk for this purpose?

IanHoang commented 1 year ago

Yes @bbarani, opensearch-cluster-cdk can be used to for provisioning the SUT in AWS infra. Since this design was done roughly 8 months ahead of opensearch-cluster-cdk repository, we will look into refactoring it.

IanHoang commented 9 months ago

Small detail: In addition to creating self-managed and managed OpenSearch clusters in AWS, it'll be good to also add support for creating Amazon OpenSearch Serverless collections.