search

tikv / raft-engine

A persistent storage engine for Multi-Raft log
Apache License 2.0
565 stars 88 forks source link

force rewrite memory tables when outdated log is not compact on time #193

Closed glorv closed 2 years ago

glorv commented 2 years ago

ref tikv/tikv#12011

Problem Summary

In the current implementation, when raft-engine detect outdated log entries that will block log file purge, it will report these regions to tikv and expect tikv to do force compact. Since tikv only do force compact on region leader and ignores request on follower. There are cases that the raft log in leader store is rewrite by tikv but not on one or more follower. Then when the log on follower need to be compact, tikv will ignore the requests because it expect region leader to do so. If there are no newer log be written into this region, as leader's log is already rewritten, it will not trigger log purge. Thus, the old log on the follower will block log file purge.

Changes

Add a HashMap to record each regions live generation after each purge phase. If a region lives after three times of force compact, raft engine will force rewrite this region to avoid blocking log file purges.

codecov[bot] commented 2 years ago

Codecov Report

Merging #193 (15f0930) into master (1ce767e) will increase coverage by 0.02%. The diff coverage is 99.13%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #193      +/-   ##
==========================================
+ Coverage   96.93%   96.96%   +0.02%     
==========================================
  Files          28       28              
  Lines        7478     7547      +69     
==========================================
+ Hits         7249     7318      +69     
  Misses        229      229
Impacted Files Coverage Δ
src/purge.rs 97.43% <97.14%> (+0.15%) :arrow_up:
src/engine.rs 97.16% <100.00%> (+0.09%) :arrow_up:
src/memtable.rs 98.98% <100.00%> (+<0.01%) :arrow_up:
src/write_barrier.rs 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 283606e...15f0930. Read the comment docs.

tabokie commented 2 years ago

@glorv The last commit is not signed.