search

eBay / HomeStore

Storage Engine for block and key/value stores.
Apache License 2.0
23 stars 21 forks source link

Add support for raft repl dev replace member. #546

Closed sanebay closed 2 months ago

sanebay commented 2 months ago

When replacing a member, add the new member, sync raft log for replace and finally remove the old member. Once we add new member, baseline or incremental resync will start. Remove the old member will cause nuraft mesg to exit the group and we periodically gc the destroyed group. Made the repl dev base test common so that both tests files can use. Tests by default create repl group with num_replica's. Dynamic tests create additional spare replica's which can be added to the test dynamically by calling replace member.

Testing

  1. Bring replica-1, replica-2, replica-3. Replica-1 is the leader. Replace the replica-3 with a new spare replica-4.
sanebay commented 2 months ago

we may need add_member and remove_member, if we do expansion/shrink. Our use case is only replace and we want to update in the homeobject PG metablk in single write atomically. The member out will have its PG metablk with destroyed flag enabled. GC will use this delete all blobs for that PG.

JacksonYao287 commented 2 months ago

The member out will have its PG metablk with destroyed flag enabled. GC will use this delete all blobs for that PG.

do we need gc for the member out? I think if we remove it from the group , we can just re-fromat it with homestore, then it will be clean , no?

sanebay commented 2 months ago

The member out will have its PG metablk with destroyed flag enabled. GC will use this delete all blobs for that PG.

do we need gc for the member out? I think if we remove it from the group , we can just re-fromat it with homestore, then it will be clean , no?

One SM could have multiple PG's on same disk. We could move out some PG for space issues or load balance.

JacksonYao287 commented 2 months ago

the code generally looks good to me. pls resolve the conflict

codecov-commenter commented 2 months ago

:warning: Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 60.78431% with 20 lines in your changes missing coverage. Please review.

Project coverage is 67.27%. Comparing base (1a0cef8) to head (afb64fc). Report is 64 commits behind head on master.

Files with missing lines Patch % Lines
src/lib/replication/repl_dev/raft_repl_dev.cpp 56.81% 18 Missing and 1 partial :warning:
src/lib/replication/repl_dev/common.cpp 0.00% 1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #546 +/- ## =========================================== + Coverage 56.51% 67.27% +10.76% =========================================== Files 108 109 +1 Lines 10300 10448 +148 Branches 1402 1408 +6 =========================================== + Hits 5821 7029 +1208 + Misses 3894 2743 -1151 - Partials 585 676 +91 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.