opensearch-project / opensearch-migrations

Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.
https://aws.amazon.com/solutions/implementations/migration-assistant-for-amazon-opensearch-service/
Apache License 2.0
39 stars 28 forks source link

Added `console backfill pause`, `console backfill stop` archives working state #1140

Closed chelma closed 6 days ago

chelma commented 1 week ago

Description

Issues Resolved

Testing

(.venv) bash-5.2# console backfill start
Backfill started successfully.
Service migration-aws-integ-reindex-from-snapshot set to 5 desired count. Currently 0 running and 0 pending.

(.venv) bash-5.2# console backfill status
BackfillStatus.STARTING
Running=0
Pending=5
Desired=5

(.venv) bash-5.2# console backfill stop
Backfill stopped successfully.
Service migration-aws-integ-reindex-from-snapshot set to 0 desired count. Currently 0 running and 5 pending.
Archiving the working state of the backfill operation...
RFS Workers are still running, waiting for them to complete...
Backfill working state archived to: /shared-logs-output/migration-console-default/backfill_working_state/working_state_backup_20241115174822.json

(.venv) bash-5.2# cat /shared-logs-output/migration-console-default/backfill_working_state/working_state_backup_20241115174822.json
[
{
    "shard_setup": {
        "numAttempts": 1,
        "scriptVersion": "poc",
        "leaseHolderId": "ip-12-0-2-40.us-east-2.compute.internal",
        "creatorId": "ip-12-0-2-40.us-east-2.compute.internal",
        "expiration": 1731693108,
        "completedAt": 1731692808
    },
    "bwc_index_1__0": {
        "numAttempts": 1,
        "scriptVersion": "poc",
        "leaseHolderId": "ip-12-0-2-40.us-east-2.compute.internal",
        "creatorId": "ip-12-0-2-40.us-east-2.compute.internal",
        "expiration": 1731693408,
        "completedAt": 1731692809
    }
}
]

(.venv) bash-5.2#console clusters cat-indices

WARNING: Cluster information may be stale. Use --refresh to update.

SOURCE CLUSTER
health status index       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   bwc_index_1 TUs7jjaWSvya-yWztVZQPg   1   0          1            0      3.8kb          3.8kb

TARGET CLUSTER
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .opensearch-observability 8HOComzdSlSWCwqWIOGRbQ   1   1          0            0       416b           208b
green  open   .plugins-ml-config        9tld-PCJToSUsMiyDhlyhQ   5   1          1            0      9.5kb          4.7kb
green  open   bwc_index_1               rPkA0eYNRma70xpJMTStdw   5   1          1            0     11.4kb          5.6kb
green  open   .kibana_1                 JICtu5p6ReiNUg8nU152tQ   1   1          1            0     10.4kb          5.2kb

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 89.09091% with 6 lines in your changes missing coverage. Please review.

Project coverage is 80.72%. Comparing base (66ae561) to head (65eca1f). Report is 15 commits behind head on main.

Files with missing lines Patch % Lines
...rationConsole/lib/console_link/console_link/cli.py 88.88% 2 Missing :warning:
...b/console_link/console_link/models/backfill_osi.py 33.33% 2 Missing :warning:
.../console_link/console_link/models/backfill_base.py 66.66% 1 Missing :warning:
...b/console_link/console_link/models/backfill_rfs.py 95.83% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1140 +/- ## ============================================ + Coverage 80.63% 80.72% +0.08% - Complexity 2910 2947 +37 ============================================ Files 399 399 Lines 14829 14965 +136 Branches 1007 1017 +10 ============================================ + Hits 11958 12080 +122 - Misses 2260 2274 +14 Partials 611 611 ``` | [Flag](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1140/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | | |---|---|---| | [gradle-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1140/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `78.74% <ø> (+0.07%)` | :arrow_up: | | [python-test](https://app.codecov.io/gh/opensearch-project/opensearch-migrations/pull/1140/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `89.95% <89.09%> (+0.19%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.