extract encrypted blockstore sync and storage system

[jchris]

From: https://fireproof.storage/posts/roadmap-to-1.0/

The block store interface is compatible with any code that is written expecting a global IPFS network for content delivery. In an encrypted context globally addressable CIDs don't make sense -- instead, you need to decrypt the block index first, and then fetch encrypted blocks on demand. Fireproof implements a transaction block store with encrypted CAR files, which can be useful to anyone wanting to move in-the-clear IFPS apps to encrypted storage. Hopefully, by extracting the block store as an independent module, it's also easy for others to add compatibility with advanced crypto like WNFS and UCAN.

Going through the codebase, here are some refactors that could pave the way for extracting the blockstore:

The Loader / Loaders

Currently there are DbLoader, FileLoader, and IndexLoader classes. They are use by the blockstore to fetch blocks by cid, and to commit new transactions (which are sets of blocks with cids). The differences are in the car file header they write, and the work they do to the internal metadata when they commit. There are also differences in the callback they use when updates come from the cloud. The db header application handles the CRDT merging etc. The idx header handles snapping to the new index head, and the files headers are never applied because they are lazy loaded.

Transaction

Transactions are triggerd by blockstore.transaction(tblocks => tblocks.put(cid, bytes))

The transaction blockstore takes care of serving blocks from memory when available, and loading from the loader when needed. Transactions are written through the loader.

The transaction is committed in loader.ts

  async _commitInternal(
    t: Transaction,
    done: IndexerResult | BulkResult | FileResult,
    opts: CommitOpts = { noLoader: false, compact: false }
  ): Promise<AnyLink> {

Here are the internal steps of that function:

• Make car header
• Make (encrypted) car file
• Save the car file locally and remotely
  • RemoteWAL (should it be per-connection?)
  • Car store
  • Meta store
  • Local
• For Index and db cars:
  • Compaction
  • Update Carlog

So it should probably be split into different implementations for indexer, db, file, instead of sharing one function.

RemoteWAL

The remote wal accumulates outbound writes before a connector is applied, so it needs to be in the main package. It also shares the build issues with the local data adapter.

[jchris]

Some work has been done, check out the refactored commit internal: https://github.com/fireproof-storage/fireproof/blob/f0f64b8d727651c3a0ba56749439d885c24b4400/packages/fireproof/src/loader.ts#L182

[jchris]

if anyone wants a fun refactor, you could change loader.ts commit into commitFiles, commitIndex, and commitTransaction, and update the callpoints in transaction.ts and crtd-helpers.ts -- you should be able to get away with only changing loader.test.js to get it working

[jchris]

released in 0.16 and available here https://www.npmjs.com/package/@fireproof/encrypted-blockstore