This replaces the sqlite-based write-ahead log with a more featureful implementation. The new implementation uses append-only log files and a "wal manager" that is in charge of scheduling batched merges from WAL to storage and rotating logs.
This is substantially faster to write to than sqlite, and also fixes a bug in the service around handling records that exceed the max blob size in sqlite.
In the course of adding support for the new merging flow, I have made changes to the nodestore/treemgr/tree relationships. There is no more concept of "staging". Instead, the tree functions work on "tree writers" and "tree readers", and there is one implementation (memtree) that is used for staging writes to the WAL as well as writes to storage from WAL.
Another implementation of the tree reader is backed by the nodestore, allowing data from the WAL to be merged with data from storage in one operation.
With this change, the "tree-aware" machinery in the nodestore such as tree merging has been moved to either the tree or treemgr module. The tree module contains methods for traversing generic tree readers/writers while the treemgr module contains methods related to interfacing trees with the WAL.
This replaces the sqlite-based write-ahead log with a more featureful implementation. The new implementation uses append-only log files and a "wal manager" that is in charge of scheduling batched merges from WAL to storage and rotating logs.
This is substantially faster to write to than sqlite, and also fixes a bug in the service around handling records that exceed the max blob size in sqlite.
In the course of adding support for the new merging flow, I have made changes to the nodestore/treemgr/tree relationships. There is no more concept of "staging". Instead, the tree functions work on "tree writers" and "tree readers", and there is one implementation (memtree) that is used for staging writes to the WAL as well as writes to storage from WAL.
Another implementation of the tree reader is backed by the nodestore, allowing data from the WAL to be merged with data from storage in one operation.
With this change, the "tree-aware" machinery in the nodestore such as tree merging has been moved to either the tree or treemgr module. The tree module contains methods for traversing generic tree readers/writers while the treemgr module contains methods related to interfacing trees with the WAL.