[NEW BONUS GUIDE + REESTRUCTURE] PostgreSQL + other related

[twofaktor]

What

One of the reasons for using PostgreSQL in LND is to address the inconvenience caused by the current bbolt database. With bbolt, node operators need to restart the node to compact the database, which can take up to 3 hours, particularly in cases with a large number of active channels. This downtime negatively impacts node uptime. By switching to PostgreSQL, this issue can be mitigated, as it offers more efficient database management capabilities without the need for node restarts, ensuring better uptime and operational efficiency.

Regarding this, PostgreSQL will no longer be used in MiniBolt solely by NBXplorer and BTCPay Server, but also by LND and Nostr relay. Therefore, the installation/update/uninstallation process will be separated into a bonus guide, and the LND guide will be modified to include the use of PostgreSQL instead of bbolt and for Nostr relay, SQLite.

Why

General reasons to use PostgreSQL:

• Power and Functionality
• Scalability
• Reliability and Durability
• Extensibility and Customization

Several advantages to using PostgreSQL over the default bbolt backend on LND reasons:

• Better handling of large data sets
• On-the-fly database compaction (auto vacuum) speeding up the startup process
• Database replication
• Inspect data while LND is running (bbolt opens the database exclusively)
• Usage of industry-standard tools to manage the stored data, get performance metrics, etc.

Furthermore, the SQL platform opens up possibilities to improve lnd's performance in the future. Bbolt's single-writer model is a severe performance bottleneck, whereas Postgres offers a variety of locking models. Additionally, structured tables reduce the need for custom serialization/deserialization code in lnd, saving developer time and limiting the potential for bugs.

How

• Separate PostgreSQL installation to a separate bonus guide
• Modify the LND guide to use PostgreSQL instead of bbolt
• Modify the Nostr relay guide to use PostgreSQL instead of sqlite
• Modify the BTCPay Server bonus guide according to these changes
• Use the lndinit fork to add a bonus section to migrate an existing bbolt database to Postgres (experimental)
• Include the Golang bonus guide as a common language for lndinit compile
• LND v0.18 update
• Regenerate Tor & I2P addresses extra Bitcoin Core section
• Generate a full bitcoin.conf file extra Bitcoin Core section
• Some additional fixes

Scope

• [X] significant change to core configuration
• [X] independent bonus guide
• [ ] simple bug fix

Test & maintenance

All changes tested by me ✅

Refs: https://github.com/lightningnetwork/lnd/blob/master/docs/postgres.md https://github.com/lightningnetwork/lnd/pull/5366 https://github.com/lightningnetwork/lnd/blob/master/docs/release-notes/release-notes-0.14.0.md#postgres-database-support https://github.com/lightningnetwork/lnd/releases/tag/v0.18.0-beta

Thanks to @blckbx for clarifying the way with this amazing guide: https://blckbx.github.io/lnd_postgres/

[blckbx]

Linking LND's PR to unleash the power of concurrent access for PostgreSQL, aimed to be merged in next release 0.18: https://github.com/lightningnetwork/lnd/pull/8644

[twofaktor]

Linking LND's PR to unleash the power of concurrent access for PostgreSQL, aimed to be merged in next release 0.18: lightningnetwork/lnd#8644

Uhm...interesting...one more reason to use PostgreSQL as the database backend for LND, but I'm not sure they include it on v0.18, I can't see anything on the milestone or roadmap of v0.18, yet: https://github.com/orgs/lightningnetwork/projects/6

[blckbx]

You're right! The replaced PR was planned for 0.18: https://github.com/lightningnetwork/lnd/pull/7992

[twofaktor]

Ready for the merge! Let's go to the main!