[META] Automate backwards compatibility test support for OpenSearch project

[abhinavGupta16]

Following up on the previous META issue - https://github.com/opensearch-project/opensearch-build/issues/90

Is your feature request related to a problem? Please describe.

We do a lot of manual testing. Need visible and repeatable automation for end-to-end upgrade paths described in https://opensearch.org/faq/#c3, including rolling and restart upgrades. Plugins should be able to contribute setup steps in an older version (data) and integration tests that run in mixed and upgraded cluster states.

Proposal

We would like to see CI/CD that shows that a specific bundle build of OpenSearch has been tested on a list of upgrade paths. Here is our proposal for BWC/Upgrade testing for OpenSearch project.

All BWC tests include:

1. Rolling upgrade
2. Mixed Cluster search
3. Snapshot upgrade
4. Restart upgrade

All components within opensearch project will define bwctests:

1. Run BWC tests in each component.
2. Run BWC tests in each component in PR push.
3. Run BWC tests in release CI against individual component.
4. Run BWC tests in release CI against the bundle.

Current Status

To run bwc-tests -

1. Deploy docker image - opensearchstaging/ci-runner:centos7-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211028
2. Clone opensearch-build repository
3. run build.sh to build the artifacts
4. run assemble.sh to install all the components
5. run test.sh with bwc-tests

To complete P0 we need the dependency installer to install each plugin and run the test for each plugin on a separate cluster.

Tasks

P0

• [x] Adding bwc test framework support for plugins: https://github.com/opensearch-project/OpenSearch/issues/1002
• [x] Adding BWC test framework automation and documentation: https://github.com/opensearch-project/opensearch-plugins/issues/53
• [x] Adding support to run BWC tests in plugin CI: https://github.com/opensearch-project/opensearch-build/issues/222

Release CI BWC test workflow: [Meta] https://github.com/opensearch-project/opensearch-build/issues/232

• [x] Adding BWC test framework in release/nightly CI: https://github.com/opensearch-project/opensearch-build/issues/275
• [x] Adding Jenkins job support: https://github.com/opensearch-project/opensearch-build/issues/492
• [x] #1897
• [x] Add support to onboard to test manifest: https://github.com/opensearch-project/opensearch-build/issues/572
• [x] Add support for TestRecorder/TestPublisher: https://github.com/opensearch-project/opensearch-build/issues/277
• [x] Add support BWC test onboarding for developers: https://github.com/opensearch-project/opensearch-build/issues/223

P1

• [ ] BWC Tests against the bundle (Meta issue): #121
• [ ] Add support for Snapshot upgrades: https://github.com/opensearch-project/OpenSearch/issues/1088
• [ ] Add support to run BWC tests in workflow via Docker #278
• [ ] Add support to run BWC tests in parallel: https://github.com/opensearch-project/opensearch-build/issues/243
• [ ] Add support to download previous version artifacts predictably for plugins: https://github.com/opensearch-project/opensearch-build/issues/184

Here is the design proposal for #121 :

1. All repositories define their bwc tests for A and B.
2. Nightly builds will run A for repo level testing via bwctest.sh script defined in each repository. Ref: https://github.com/opensearch-project/anomaly-detection/blob/main/integtest.sh Eg: ./integtest.sh bwcTest
3. We will rely on https://github.com/opensearch-project/opensearch-build/issues/122 and a job to the orchestrator to spin up various kinds of BWC test clusters.
4. Nightly builds will also run B for bundle level testing via bwctest.sh defined in each repository using the clusters setup by the orchestrator. The orchestrator job will make calls to the bwctest.sh script to perform various steps of testing during upgrade. Eg: For Rolling Upgrade, the orchestrator job will spin an old version (the versions should be defined via a manifest, TBD). a. Kick off ingestion via ./bwctest.sh RollingUpgradeIngestTest b. Upgrade a node to new version c. Kick off mixed cluster test via ./bwctest.sh MixedClusterSearchTest d. Upgrade all nodes to the latest version. e. Kick off upgraded cluster test via ./bwctest.sh UpgradedClusterTest f. Report test status and repeat for every repository.
5. Report bundle level test status.

[zelinh]

Closing in favor of https://github.com/opensearch-project/opensearch-build/issues/2870. Will update in that issue for progress.