opensearch-project / opensearch-migrations

Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.
https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/
Apache License 2.0
39 stars 28 forks source link

Metadata Support ES7+ Legacy Templates #1128

Closed AndreKurait closed 1 week ago

AndreKurait commented 1 week ago

Description

Currently, legacy templates (created with _template) were only supported on ES 6. Since the API is deprecated and not removed, customers still use this type of template for ES 7 and above and it should be supported and tested.

This change mirrors the existing transformation logic that is in Index processing and applying to Legacy templates for ES7. E.g.

 * Starting state:
 * {
 *   "mappings": [
 *     {
 *       "foo": {
 *         "properties": {
 *           "field1": { "type": "text" },
 *           "field2": { "type": "keyword" }
 *         }
 *       }
 *     }
 *   ]
 * }
 *
 * Ending state:
 * {
 *   "mappings": {
 *     "properties": {
 *       "field1": { "type": "text" },
 *       "field2": { "type": "keyword" },
 *     }
 *   }
 * }

Issues Resolved

MIGRATIONS-2217

Is this a backport? If so, please add backport PR # and/or commits #

Testing

Added E2E tests across different criteria for legacy templates on all versions with snapshot and api retrieval

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 80.70%. Comparing base (41bbfc4) to head (adb819e). Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...load/version_es_6_8/GlobalMetadataData_ES_6_8.java 0.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1128 +/- ## ============================================ + Coverage 80.63% 80.70% +0.06% - Complexity 2910 2924 +14 ============================================ Files 399 399 Lines 14829 14846 +17 Branches 1007 1007 ============================================ + Hits 11958 11982 +24 + Misses 2260 2257 -3 + Partials 611 607 -4 ``` | [Flag](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1128/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | | |---|---|---| | [gradle-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1128/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `78.75% <92.00%> (+0.08%)` | :arrow_up: | | [python-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1128/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `89.75% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

AndreKurait commented 1 week ago

Could you also please provide the before/after of what the template from legacy to latest? That could be in the PR or (better), the commit message.

Added context in the PR Overview. This isn't net-new transformation or functionality, just doing the same processing on the templates mapping key that we apply in ES7 indexes (and ES6 templates/indexes)