search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
138 stars 272 forks source link

[META] Automate backwards compatibility test support for OpenSearch project #90

Closed dblock closed 2 years ago

dblock commented 3 years ago

Is your feature request related to a problem? Please describe.

We do a lot of manual testing. Need visible and repeatable automation for end-to-end upgrade paths described in https://opensearch.org/faq/#c3, including rolling and restart upgrades. Plugins should be able to contribute setup steps in an older version (data) and integration tests that run in mixed and upgraded cluster states.

Proposal

We would like to see CI/CD that shows that a specific bundle build of OpenSearch has been tested on a list of upgrade paths. Here is our proposal for BWC/Upgrade testing for OpenSearch project.

All BWC tests include:

  1. Rolling upgrade
  2. Mixed Cluster search
  3. Snapshot upgrade
  4. Restart upgrade

All components within opensearch project will define bwctests:

  1. Run BWC tests in each component.
  2. Run BWC tests in each component in PR push.
  3. Run BWC tests in release CI against individual component.
  4. Run BWC tests in release CI against the bundle.

Tasks

P0

Release CI BWC test workflow: [Meta] https://github.com/opensearch-project/opensearch-build/issues/232

P1

Here is the design proposal for #121 :

  1. All repositories define their bwc tests for A and B.
  2. Nightly builds will run A for repo level testing via bwctest.sh script defined in each repository. Ref: https://github.com/opensearch-project/anomaly-detection/blob/main/integtest.sh Eg: ./integtest.sh bwcTest
  3. We will rely on https://github.com/opensearch-project/opensearch-build/issues/122 and a job to the orchestrator to spin up various kinds of BWC test clusters.
  4. Nightly builds will also run B for bundle level testing via bwctest.sh defined in each repository using the clusters setup by the orchestrator. The orchestrator job will make calls to the bwctest.sh script to perform various steps of testing during upgrade. Eg: For Rolling Upgrade, the orchestrator job will spin an old version (the versions should be defined via a manifest, TBD). a. Kick off ingestion via ./bwctest.sh RollingUpgradeIngestTest b. Upgrade a node to new version c. Kick off mixed cluster test via ./bwctest.sh MixedClusterSearchTest d. Upgrade all nodes to the latest version. e. Kick off upgraded cluster test via ./bwctest.sh UpgradedClusterTest f. Report test status and repeat for every repository.
  5. Report bundle level test status.
dblock commented 3 years ago

https://github.com/opensearch-project/opensearch-build/issues/58 and https://github.com/opensearch-project/opensearch-plugins/issues/53 are related.

saratvemulapalli commented 3 years ago

Modified this issue to be a meta issue to track all backward compatibility test support.

shwetathareja commented 3 years ago

Ideally, the integration tests which are written like Rest APIs or any feature test, the framework should allow to run them in BWC mode (e.g. mixed mode). This will help in catching any issues when there are mixed nodes in the cluster.

dblock commented 3 years ago

@saratvemulapalli is it time to close this issue given the work you've done here?

saratvemulapalli commented 3 years ago

@saratvemulapalli is it time to close this issue given the work you've done here?

I dont think so, there are bunch of items which have to be done for P0 and P1.

dblock commented 3 years ago

If you care, you can turn the lists above into - [ ] issue and GitHub will help you see and track what was resolved and what wasn't

gaiksaya commented 2 years ago

@abhinavGupta16 is looking into rewriting this issue with requirements, what we have and implementation with current codebase.

abhinavGupta16 commented 2 years ago

Closing this META issue to replace with the updated issue - https://github.com/opensearch-project/opensearch-build/issues/1873