khonsulabs / okaywal

A Write Ahead Log (WAL) implementation in Rust
Apache License 2.0
215 stars 11 forks source link

Add a synchronous "before checkpoint" hook? #12

Open bjchambers opened 1 year ago

bjchambers commented 1 year ago

For some use cases of a write ahead log, I'd like to keep an in-memory representation (which gets recovered) of the contents of the log. When a checkpoint is made, I can drop the check-pointed information from memory since it is now part of the checkpoint.

One difficulty, as far as I can tell, is that the checkpoint method is called in a separate thread -- this means the following could happen:

  1. Commit some data A to the log
  2. Initiate a checkpoint
  3. Commit some data B to the log
  4. Actually perform the checkpoint

The problem I have is that at that point, the contents of A and B are in the in-memory representaiton, while I only want to checkpoint (and potentially drop) A. For my use case, I'm fine keeping the in-memory representation behind a Mutex (there should be only one writer to the log), which also simplifies a lot of the logic.

There are a few things that could help me:

  1. If the result of committing data indicated that a checkpoint would be taken. I think this would only help if there was a lock, but then I could detect that a checkpoint would be taken after A, and arrange to have a separate in-memory representation started for data added after that.
  2. If there was a "pre-checkpoint" hook that was called before other commits were allowed. Again, this would allow me to split the in-memory representation for B.
ecton commented 1 year ago

Thank you for the idea for the additional use case for this crate. I think the easiest thing to offer would be your second suggestion.

If you need this quickly, I would love a pull request. Otherwise I will eventually be getting back onto this project, but right now it's not my primary focus.