opensearch-project / opensearch-migrations

Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.
https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/
Apache License 2.0
39 stars 28 forks source link

Add Basic Jenkins Integ Test for Demo Setup #1099

Closed lewijacn closed 3 weeks ago

lewijacn commented 1 month ago

Description

This change introduces a new Jenkins workflow for a full Migration workflow (Metadata, Backfill, C&R) using the Capture Proxy and Target Proxy ECS services and ALB as opposed to the previous workflow of having the capture proxy installed directly on the source nodes.

This is a basic integration test in the sense that it is only moving a few key documents that get added in different parts of the workflow and is verifying source and target connections within the ALB and outside of it. In its current state it should add a lot of value for checking this setup, but within the Epic of the Jira Issue linked, there are other tasks for adding more interesting data and checking an ALB cutover from source to target that would be great additions

Also updates were added after discussion to combine this with the previous ES 6.8 RFS only test and make this the default PR test that is executed

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-2149

Testing

Jenkins testing here: https://migrations.ci.opensearch.org/job/full-default-e2e-test/19/console Update test here: https://migrations.ci.opensearch.org/job/full-es68source-e2e-test/4/

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 80.55%. Comparing base (db0075e) to head (e0b241d). Report is 17 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1099 +/- ## ============================================ - Coverage 80.66% 80.55% -0.11% - Complexity 2893 2930 +37 ============================================ Files 383 390 +7 Lines 14361 14462 +101 Branches 989 998 +9 ============================================ + Hits 11584 11650 +66 - Misses 2184 2209 +25 - Partials 593 603 +10 ``` | [Flag](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1099/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | | |---|---|---| | [gradle-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1099/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `78.63% <ø> (-0.12%)` | :arrow_down: | | [python-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1099/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `90.33% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.