xline-kv / Xline

A geo-distributed KV store for metadata management
https://xline.cloud
Apache License 2.0
562 stars 71 forks source link

Refactor the `LogEntryVecDeque` in the log.rs #800

Closed Phoenix500526 closed 2 weeks ago

Phoenix500526 commented 2 months ago

After the pr #704 being merged, we got a LogEntryVecDeque implementation like this:

/// For the leader, there should never be a gap between snapshot and entries
pub(super) struct Log<C: Command> {
    /// Log entries, should be persisted
    /// Note that the logical index in `LogEntry` is different from physical index
    entries: LogEntryVecDeque<C>,
    /// The last log index that has been compacted
    pub(super) base_index: LogIndex,
    /// The last log term that has been compacted
    pub(super) base_term: u64,
    /// Index of highest log entry known to be committed
    pub(super) commit_index: LogIndex,
    /// Index of highest log entry sent to after sync. `last_as` should always be less than or equal to `last_exe`.
    pub(super) last_as: LogIndex,
    /// Index of highest log entry sent to speculatively exe. `last_exe` should always be greater than or equal to `last_as`.
    pub(super) last_exe: LogIndex,
    /// Contexts of fallback log entries
    pub(super) fallback_contexts: HashMap<LogIndex, FallbackContext<C>>,
    /// Tx to send log entries to persist task
    log_tx: mpsc::UnboundedSender<Arc<LogEntry<C>>>,
    /// Entries to keep in memory
    entries_cap: usize,
}

struct LogEntryVecDeque<C: Command> {
    /// A VecDeque to store log entries, it will be serialized and persisted
    entries: VecDeque<Arc<LogEntry<C>>>,
    /// entry size of each item in entries
    entry_size: VecDeque<u64>,
    /// the right index of the batch (offset)
    /// batch_range: [i, i + batch_index[i]]
    batch_index: VecDeque<usize>,
    /// the first entry idx of the current batch window
    first_entry_at_last_batch: usize,
    /// the current batch window size
    last_batch_size: u64,
    /// Batch size limit
    batch_limit: u64,
}

As you can see, batch_index stores an offset. With this offset, we can calculate a log batch of size less than or equal to batch_size for any given index i (FYI: issue #368 ). We record the offset instead of the right bound in that the log may be compressed. However, there is a method named li_to_pi in Log, which can transform the logical index to the physical index (The compression only changes the physical index ):

/// Transform logical index to physical index of `self.entries`
    fn li_to_pi(&self, i: LogIndex) -> usize {
        assert!(
            i > self.base_index,
            "can't access the log entry whose index is not bigger than {}, might have been compacted",
            self.base_index
        );
        (i - self.base_index - 1).numeric_cast()
    }

Therefore, we think we can do some refactoring stuff: move li_to_pi and relevant fields, like base_index and base_term into LogEntryVecDeque.

github-actions[bot] commented 2 months ago

👋 Thanks for opening this issue!

Reply with the following command on its own line to get help or engage: