[META] Automate Testing

[setiah]

Current state and Action plan

Plugin artifacts are built with github workflows today. Plugin teams manually provide their artifacts to the Infra team. Infra team owns a Jenkins fleet, which is in Amazon internal network, so github workflows cannot interact with Jenkins directly via APIs due to no network access. So Infra team manually kicks off a jenkins job to bundle these plugin artifacts together with OpenSearch min distribution to create a full distribution. Once the full distribution is ready, they manually kick off another Jenkins job to run integration tests, for each of the plugins. These integration tests run on single node clusters inside a docker environment. The integration tests run in parallel for each plugin, in separate environments (one per plugin). These test results are available on Jenkins, but only visible to Infra team due to restricted access. The integration test run, takes less than an hour for all plugins.

Going forward, we are planning to automate this workflow with two pipelines (conceptually) - build and test infra. The build pipeline will assemble all bundles - includes - full distribution, min distribution, docker, non-docker, arch64, arm, linux, debian, etc. and publish them to an S3 bucket. Each bundle will have a manifest file which lists the commit ids a bundle was built from. The test infra pipeline will read these snapshot bundles from the S3 bucket. The trigger would be based on a periodic schedule to start with, and will later evolve to support events from build pipeline. Once the test infra pipeline is triggered, it will kick off nightly tests on these snapshot bundles. These nightly tests include per plugin integration tests, plugin and core engine backward compatibility tests, and rally based performance benchmarking. All tests will run inside a containerized environment on Jenkins.

Automated Test infrastructure will be built on Jenkins. It is currently missing support for running performance benchmarking tests and backward compatibility (aka upgrade) tests. We will add Jenkins jobs for running rally based performance benchmarks against the distribution support matrix. These benchmark results will be ingested in a long running OpenSearch cluster which can then be visualized via OpenSearch Dashboards. For the backward compatibility testing, the plugins today do not have framework for backward compatibility. These tests are run manually by every plugin team before the release. Going forward, we will add a backward compatibility test framework in OpenSearch plugins. We will also add a new jenkins job for Backward compatibility testing on full bundle, which will validate compatibility with previous versions of OpenSearch.

Action Items

Below are the key components of Automated test infra pipeline. We have created meta issues for each of these key components to track their respective sub-tasks as github issues.

• [x] https://github.com/opensearch-project/opensearch-build/issues/126
• [x] https://github.com/opensearch-project/opensearch-build/issues/90
• [x] https://github.com/opensearch-project/opensearch-build/issues/124
• [x] https://github.com/opensearch-project/opensearch-build/issues/167

[peternied]

[Triage] @minalsha could you update this issue with the current state?

[bbarani]

Closing this issue as we have automated integration, BWC (at plugin level) and performance testing for OpenSearch builds. We will track the implementation of performance testing for distribution builds in this issue.