search

fireproof-storage / fireproof

Realtime database, runs anywhere. Install Fireproof in your front-end app or edge function, and sync data via any backend.
https://fireproof.storage
Other
218 stars 16 forks source link

compactor modes and crdt fallback #95

Closed jchris closed 3 months ago

jchris commented 4 months ago

If I drive a database with heavy automated clicks from 3-4 users for an hour, I'll get this kind of thing which should impossible. It seems to be that merging two compacted datasets can sometimes require a block neither compaction required in order to resolve on it own, but is needed for the merge.

The code is intentionally error-happy at the moment to expose these questions, there's a design pass to make the compactor intentionally prune history, which would require the readers to use the oldest available crdt as the beginning of history. This would allow even flattened databases to serve without error.

Another approach is to move "online" for block search when the block is not found. It might be that servers run in this mode and clients rely on pruned history. This effectively reads through the compaction by chasing the history back before compaction (today we log the compact cids but don't dereference them yet).

Implementing this will simplify the compaction model on the client because we can compact to nearly flat history as long as it is older than the conflict merge window we care about (maybe very flat for some apps).

Screenshot 2024-02-16 at 9 43 22 AM
jchris commented 4 months ago 
Error: missing block:
at EventFetcher.get
at async findAncestorCandidate
at async findCommonAncestor
jchris commented 4 months ago

discord discussion:

I implemented a default compactor (all blocks to one file): https://github.com/fireproof-storage/fireproof/blob/bdfb742af714df8713fe940ceee34e0a1e8e10cf/packages/encrypted-blockstore/src/transaction.ts#L118

But it's like 300MB of storage where the Fireproof compactor is closer to 3MB. Here is where the Fireproof connector is disabled in favor of the default blockstore compactor: https://github.com/fireproof-storage/fireproof/blob/bdfb742af714df8713fe940ceee34e0a1e8e10cf/packages/fireproof/src/crdt.ts#L51 the default compactor doesn't have the issues described in issue 95 so I've switched to it for now, but it has perf issues when the data set gets big

jchris commented 4 months ago

A mature approach will move to something like multi-level compaction https://www.speedb.io/blog-posts/understanding-leveled-compaction

jchris commented 3 months ago

this was fixed in #98 with a network lookup mode to recover any missing blocks from the network. This makes concurrent merges safe