Open Roasbeef opened 2 years ago
Example of the new architectures that this would indirectly enable: https://github.com/lightningnetwork/lnd/pull/6262#issuecomment-1050358986
I propose to add high level storage interfaces exchanging data in structured types rather than bytes (repository pattern).
Currently the primary storage interface is key-value which is bbolt-like. Data types in the KV interface are already serialized. Some disadvantages of KV approach to storage interface were noted in https://github.com/lightningnetwork/lnd/issues/6176#issue-1106872434
Many packages use kvdb
package directly and encode/decode data types themselves. I propose to "hide" kvdb from high level packages. High level code could pass structs to methods of repositories, which could encode/decode and call kvdb. Alternative implementation may call something else instead of kvdb.
If all parts of DatabaseInstances become high level interfaces, it will simplify the following things:
Transition from KV to high level interfaces can be done in an incremental way and it is an internal code change, not requiring database migration; the KV implementation could expose new methods with high level types instead of []byte
.
During the 0.14 cycle, we did a lot of refactoring to extract much of the internal sub-system initialization from the
main()
function inlng.go
. This was a necessary step along the road to enabling things like the new remote signer operating mode, which from the PoV oflnd
is just the same existing wallet/signing interfaces used. In this operating mode,lnd
isn't explicitly aware that it's actually speaking with a signer over the RPC interface. This interaction is enabled by the fact that our internal wallet/signing interface closely mirrors the wallet/signing sub-servers.In a similar vein, if we start to surface more storage-level interfaces as part of the
DatabaseBuilder
abstraction, then we enable users to pass in custom implementations of app-specific storage engines like the invoice registry or channel graph.One easy win here on the road to creating more performance optimized storage implementations for various sub-systems is abstracting away usage of the "channel graph" across the codebase. Today most of the time, the graph is interact with via a direct
*channeldb.DB
pointer. If we add a layer of indirection here with another interface, then it enables more advancedlnd
operating modes like having a fleet of nodes share the same channel graph (no need for them all to download it again once a new node is added to the cluster, etc). Ignoring some of the database concurrency control implications for a second, this would allow new nodes which are part of shared infrastructure to more instantly be initialized, as they don't need to sync the graph from the p2p network. This also serves to de-duplicate information that can safely be shared across relatedlnd
nodes.Steps to Completion
*channeldb.DB
interface which is part of the main db/storage config with the interface created above.