search

rosedblabs / rosedb

Lightweight, fast and reliable key/value storage engine based on Bitcask.
https://rosedblabs.github.io
Apache License 2.0
4.58k stars 632 forks source link

Proposal: raft protocol support #290

Closed 0x5487 closed 7 months ago

0x5487 commented 1 year ago

Scenario: As a core service. It always needs to be high availability. If one node was down, other nodes needed to take ownership.

Proposal: use raft protocol to enhance high availability to avoid one node being down. Also, need to implement the snapshot feature to work with the raft.

roseduan commented 1 year ago

This is a long-term job, and may need a lot of time. If this is truly what you want, I can add it to the plan.

0x5487 commented 12 months ago

Yep, agree. That is a long-term job. I can share feedback on my investigating.

snapshot: I plan to create a snapshot file of the current state of RoseDB. There are two ways that have come to my mind at the moment.

idea 1: copy all the segment files to another directory and log the current time. After that, create a goroutine to iterate WAL files and compare each record's time. If the record's created time is greater than the current time, the record needs to be removed.
However, the records struct doesn't have a created time field at the moment. Maybe the field can be added later?

idea 2: If WAL were using a sparse file, we can log the actual size when the snapshot function is called. After that, we can just copy all the segment files and truncate the last active file's size to the actual size.