search

polarsignals / frostdb

❄️ Coolest database around 🧊 Embeddable column database written in Go.
Apache License 2.0
1.29k stars 65 forks source link

db: ignore writes below new block transaction on recovery #864

Closed asubiotto closed 3 months ago

asubiotto commented 3 months ago

Previously, the recovery code was ignoring writes below the persist transaction of a table block. However, this could lead to dropped writes since the active block is swapped atomically on rotation before the old block is persisted. Writes in between these two events would be written to the new table block, but ignored on recovery given the recovery code assumed they were in the old table block.

asubiotto commented 3 months ago

Data race in snapshot on recovery test. Will fix

asubiotto commented 3 months ago

Race should be fixed. I ended up having to store the next non-persisted txn in the wal proto, given the replay semantics had to otherwise be slightly modified with code to deduce this txn, so I decided it was easier to just store and retrieve it with minimal changes to the recovery code.

I also added a second commit with a fix for duplicate writes found by DST due to an off-by-one WAL error.