eBay / HomeStore

Storage Engine for block and key/value stores.
Apache License 2.0
23 stars 21 forks source link

enable home raft log store UT #515

Closed JacksonYao287 closed 3 months ago

JacksonYao287 commented 3 months ago

1 enable home_raft_log_store UT and add it to docker file and nightly runing 2 remove android directory in github CI in the stage of building cache(previous in create-and-test-package) ,so the space in github CI VM will be definitely freed for every pipeline, since one of CI pipeline does not have create-and-test-package . 3 increase the disk size to 2GB for UT 4 fix log dev flush bug. add a loop to create multiple logGroup to make sure all the expected logs flushed . 5 fix a bug in home_Raft_log_store#pack to make sure the available size of packing buffer is able to hold entry.size() and the length of this entry 6 use truncation instead of flush when home_raft_log_store#compact. this is only a in-memory change. the real truncation will be scheduled by resource manager. also create issue to handle the start_index case when recovery from a crash #530, wo do it in a separate PR. 7 add commit_config log so we can see when a config change is made. 8 fix a bug in handle_raft_event. we need to indentify whether a log entry should be appended to log store. if the req#lsn is not -1, it means this log has been localized and appended before, we should skip it. 9 add a force_leave api for now to handle the case that a follower and a leader has destroyed the raft goup , but the second follower fail to receive this message and will stuck. this is used for fixing raft_repl_dev UT. we can revisit here later if necessary. 10 fix the bug of raft_repl_dev UT when setting last_committed_idx in write_Snapshot_data in follower side. we should get the commited_index from raft_server itself. in our UT, commit_config is not taken account when increasing the commit_count. 11 fix bug bug of raft_repl_dev UT in read_snap_shot of leader. when we search the next start lsn which is the start of sending data. we can not use std::map::find , since if the requested next_lsn is a config_change , it will not be put into kvDB(lsnindex) and as a result , std::map::find wil return std::end() and nothing will be sent to follower. instead , we should use low_bound, so that we can get the first data kv to be sent , config will be skipped.

codecov-commenter commented 3 months ago

:warning: Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 52.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 68.21%. Comparing base (1a0cef8) to head (65f7777). Report is 55 commits behind head on master.

Files with missing lines Patch % Lines
src/lib/logstore/log_dev.cpp 50.00% 6 Missing and 2 partials :warning:
src/include/homestore/replication/repl_dev.h 0.00% 2 Missing :warning:
src/lib/replication/repl_dev/raft_repl_dev.cpp 0.00% 0 Missing and 1 partial :warning:
src/lib/replication/repl_dev/raft_repl_dev.h 0.00% 1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #515 +/- ## =========================================== + Coverage 56.51% 68.21% +11.70% =========================================== Files 108 109 +1 Lines 10300 10433 +133 Branches 1402 1400 -2 =========================================== + Hits 5821 7117 +1296 + Misses 3894 2638 -1256 - Partials 585 678 +93 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.