OpenSearch Migrations Engine

Table of Contents

1. Overview
2. Key Features
3. Supported Versions and Platforms
4. Issue Tracking
5. User Guide Documentation
6. Getting Started
   • Local Deployment
   • AWS Deployment
7. Continuous Integration and Deployment
8. Contributing
9. Security
10. License
11. Acknowledgments

Overview

The OpenSearch Migrations Engine is a comprehensive set of tools designed to facilitate upgrades, migrations, and comparisons for OpenSearch and Elasticsearch clusters. This project aims to simplify the process of moving between different versions and platforms while ensuring data integrity and performance.

Key Features

• Upgrade and Migration Support: Provides tools for migrating between different versions of Elasticsearch and OpenSearch.

  • Metadata Migration: Migrate essential cluster components such as configuration, settings, templates, and aliases.
  • Multi-Version Upgrade: Easily migrate across major versions (e.g., from Elasticsearch 6.8 to OpenSearch 2.15), skipping intermediate upgrades and reducing time and risk.
  • Downgrade Support: Downgrade to an earlier version if needed (e.g., from Elasticsearch 7.17 to 7.10.2).
  • Existing Data Migration with Reindex-from-Snapshot: Migrate indices and documents using snapshots, updating your data to the latest Lucene version quickly without impacting the target cluster.
  • Live Traffic Capture with Capture-and-Replay: Capture live traffic from the source cluster and replay it on the target cluster for validation. This ensures the target cluster can handle real-world traffic patterns before fully migrating.

• Zero-Downtime Migration with Live Traffic Routing: Tools to seamlessly switch client traffic between clusters while keeping services fully operational.

• Migration Rollback: Keep your source cluster synchronized during the migration, allowing you to monitor the target cluster's performance before fully committing to the switch. You can safely revert if needed.

• User-Friendly Interface via Migration Console: Command Line Interface (CLI) that guides you through each migration step.

• Flexible Deployment Options:

  • AWS Deployment: Fully automated deployment to AWS.
  • Local Docker Deployment: Run the solution locally in a container for testing and development.
  • Contribute to add more deployment options.

Supported Migration Paths and Platforms

Migration Paths

Note that testing is done on specific minor versions, but any minor versions within a listed major version are expected to work.

Platforms

• Self-managed (cloud provider hosted)
• Self-managed (on-premises)
• Managed cloud offerings (e.g., Amazon OpenSearch, Amazon OpenSearch Serverless)

Issue Tracking

We encourage users to open bugs and feature requests in this GitHub repository.

Encountering a compatibility issue or missing feature?

• Search existing issues to see if it’s already reported. If it is, feel free to upvote and comment.
• Can’t find it? Create a new issue to let us know.

For issue prioritization and management, the migrations team uses Jira, but uses GitHub issues for community intake:

https://opensearch.atlassian.net/

User Guide Documentation

User guide documentation is available in the OpenSearch Migrations Wiki.

Getting Started

Local Deployment

Refer to the Development Guide for more details.

AWS Deployment

To deploy the solution on AWS, follow the steps outlined in Migration Assistant for Amazon OpenSearch Service, specifically deploying the solution.

Continuous Integration and Deployment

We use a combination of GitHub actions and Jenkins so that we can publish releases on a weekly basis and allow users to provide attestation for migration tooling.

Jenkins pipelines are available here

Contributing

Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

Please refer to the Development Guide for details on building and deploying.

Security

See SECURITY.md for information about reporting security vulnerabilities.

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.

Acknowledgments

• OpenSearch Community
• Contributors and maintainers

For more detailed information about specific components, please refer to the README files in the respective directories.