lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.63k stars 2.07k forks source link

[Epic] channeldb+payment: create native SQL schema for payment storage #7920

Open yyforyongyu opened 1 year ago

yyforyongyu commented 1 year ago

Depends on #6288.

This is the master issue that tracks the progress of adding native SQL support for payment db, which is split into three steps.

Support native SQL config

The first step is to allow users to configure SQL backends. There are two approaches,

  1. use existing config flags.
  2. add new config flags.

Since we already support multiple database backends, current nodes running with SQL would already have tables filled with key-value data. Thus going with option 1 creates more complexity as now we need to deal with mixed data types in the same database. Option 2 allows managing the native SQL in a clean state. With a few flag tricks, we can easily separate the way the database is configured. The idea is expressed in

Use native SQL for payments

The second step is to build SQL queries for payment data using sqlc. We already have the interface ControlTower that covers db operations. Ideally, we just need to create a new package payments to hold the data structs and db operations, to avoid we end up having a big package like channeldb. The tracking PR is,

Going SQL means we are giving up the flexibility provided by TLV, any updates to the scheme now need a migration. Future payment format should define its own struct and save the data in a new table.

Migrate kvdb data into SQL

In #6469 we introduced optional migrations. For existing nodes, we'd need to migrate their current key-value data into SQL rows. This migration needs to be paid attention to OOM as payment data can be large.

Roasbeef commented 1 year ago

Going SQL means we are giving up the flexibility provided by TLV, any updates to the scheme now need a migration. Future payment format should define its own struct and save the data in a new table.

If relevant, we do have the ability to sort of mix things a bit, by having a column that actually stores TLV data. The upside of just using the normal migrations there though as we have to maintain less specific migration code for the various data structures.