Add reindexFromSnapshotWorkerSize to cdk with default and maximum modes

opensearch-project / opensearch-migrations

Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.

Apache License 2.0

39 stars 28 forks source link

Description

Added an RFS High Performance Mode that does the following:

Increases EBS Throughput for faster s3 download and extract
Increases worker size to 16vCPU 32GB
Sets max concurrent requests to 100
Enables Target Compression

With these settings, i was able to use 5 workers to migrate a 5 shard source where each source shard was 1.9TiB (9.2TiB total) in 7 hours with a target cluster composed of 10 r7gd.16xlarge data nodes (Increased target index shard count to 400).

Also improved the UX by adding a buffer on max shard size which can reduce friction when shards are slightly uneven and a customer calculated their max shard size by taking primary size / shards.

Also fixed a bug in imageHash for builds with "docker-compatible" clients like podman when image Id just returns the hash and not a :.

Category: New Feature
Why these changes are required? Enable vertical scaling when horizontal scaling is limited
What is the old behavior before changes and new behavior after changes? Vertical scaling not supported, now supported

Issues Resolved

MIGRATIONS-2144

Is this a backport? If so, please add backport PR # and/or commits #

Testing

Testing: With these settings, i was able to use 5 workers to migrate a 5 shard source where each source shard was 1.9TiB (9.2TiB total) in 7 hours with a target cluster composed of 10 r7gd.16xlarge data nodes (Increased target index shard count to 400).

Added unit testing

Check List

[x] New functionality includes testing
- [x] All tests pass, including unit test, integration test and doctest
[x] New functionality has been documented
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 80.66%. Comparing base (e2c5be7) to head (aab31a7). Report is 1 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1085 +/- ## ========================================= Coverage 80.66% 80.66% Complexity 2893 2893 ========================================= Files 383 383 Lines 14361 14361 Branches 989 989 ========================================= Hits 11584 11584 Misses 2184 2184 Partials 593 593 ``` | [Flag](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1085/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | | |---|---|---| | [gradle-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1085/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `78.75% <ø> (ø)` | | | [python-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1085/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `90.33% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#carryforward-flags-in-the-pull-request-comment) to find out more.

opensearch-project / opensearch-migrations