The goal of this PR is to make the current segment recoverable in case of a crash. Taking inspiration from SQLite's wal, we introduce a grinning checksum to the current log, where each frame is hashed with the hash of the previous frame. If we find an unsealed segment on startup, we enter recovery mode for that segment, meaning that we iterate over its frames, recomputing the checksums. We stop whenever we hit a checksum that breaks the list invariant and patch the segment. This means that in NORMAL synchronous mode, we can lose committed data on the crash, but this is expected. We'll introduce a more robust sync mode later that flushes on every commit and has stronger guarantees.
This scheme introduces a bit more complexity:
double flush on seal: on seal, we first flush all data to disk before updating the header and flushing again. This makes the SEALED flag on the segment reliable: if sealed is present, we can guarantee that all data was flushed before that, and therefore we can trust the segment's integrity. Without that, issues could arise on a partial flush where the header was flushed but not the rest of the segment.
When we overwrite frames in a transaction, we need to recompute checksums.
I have added a test that simulates a crash during flush, where not all pages are flushed to disk, to demonstrate recovery.
The goal of this PR is to make the current segment recoverable in case of a crash. Taking inspiration from SQLite's wal, we introduce a grinning checksum to the current log, where each frame is hashed with the hash of the previous frame. If we find an unsealed segment on startup, we enter recovery mode for that segment, meaning that we iterate over its frames, recomputing the checksums. We stop whenever we hit a checksum that breaks the list invariant and patch the segment. This means that in NORMAL synchronous mode, we can lose committed data on the crash, but this is expected. We'll introduce a more robust sync mode later that flushes on every commit and has stronger guarantees.
This scheme introduces a bit more complexity:
I have added a test that simulates a crash during flush, where not all pages are flushed to disk, to demonstrate recovery.