opensearch-project / opensearch-migrations

Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.
https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/
Apache License 2.0
39 stars 28 forks source link

Rfs document transformations #1110

Closed AndreKurait closed 2 weeks ago

AndreKurait commented 3 weeks ago

Description

Implements rfs document transforms.

input/output spec:

{
   "index": {
       "_index": "your_index_name",
       "_type": "your_type", // only for ES <=6.
       "_id": "doc_1234"
   }
   // Document source
   "source": {}
}

Note: This changes the current default behavior for ES6 regarding docIds, before we were retaining the lucene id, which would be modifying the docId as ES/OS sees it. Before: id = mytype#docId, now id = docId

Also, moved transformationPlugins from TrafficCapture to top level transformer package

Issues Resolved

MIGRATIONS-2152

Is this a backport? If so, please add backport PR # and/or commits #

Testing

Added Unit testing

Performed performance testing in the cloud in a case where we were cpu bound on the workers. 5TB migration went from 35 -> 49 minutes with 200 workers

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 54.91803% with 55 lines in your changes missing coverage. Please review.

Project coverage is 80.43%. Comparing base (dc326f0) to head (f62c670). Report is 17 commits behind head on main.

Files with missing lines Patch % Lines
...h/migrations/transform/TransformerConfigUtils.java 0.00% 20 Missing :warning:
...rch/migrations/bulkload/common/BulkDocSection.java 73.84% 17 Missing :warning:
...org/opensearch/migrations/RfsMigrateDocuments.java 0.00% 14 Missing :warning:
.../opensearch/migrations/replay/TrafficReplayer.java 0.00% 2 Missing :warning:
.../migrations/bulkload/common/DocumentReindexer.java 87.50% 1 Missing :warning:
...h/migrations/bulkload/common/OpenSearchClient.java 50.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1110 +/- ## ============================================ - Coverage 80.55% 80.43% -0.13% - Complexity 2930 2947 +17 ============================================ Files 390 393 +3 Lines 14462 14534 +72 Branches 998 1000 +2 ============================================ + Hits 11650 11690 +40 - Misses 2209 2243 +34 + Partials 603 601 -2 ``` | [Flag](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1110/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | | |---|---|---| | [gradle-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1110/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `78.50% <54.91%> (-0.14%)` | :arrow_down: | | [python-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1110/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `90.33% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.