Pruned bitcoind support

[Sjors]

Useful for e-commerce solutions since 200 GB of storage on an EC2 instance costs ~$20 / month. Giving access to a remote shared full node is more complicated and means more servers to keep secure.

Known issues:

• [x] #1283
• [x] #1296
• [x] #1502

Once achieved, the following can be removed from the README:

Notice that currently pruned nodes are not supported and may result in lightningd being unable to synchronize with the blockchain.

[Sjors]

Is it correct to say that a node that's manually pruned to block 504500 ( .when_lightning_became_cool) and not pruned after that, is safe to use?

[ZmnSCPxj]

In theory, if somebody evil gives us gossip that claims a channel was opened at blockheight 504499, we will have to go check it, and have to query the block at that height from the bitcoind. If I remember correctly however we do not crash immediately if bitcoind is unable to provide that data in this case.

[Sjors]

Shouldn't when_lightning_became_cool be enforced by gossipd? As in, it would simply ignore such gossip.

[NicolasDorier]

In theory, if somebody evil gives us gossip that claims a channel was opened at blockheight 504499, we will have to go check it, and have to query the block at that height from the bitcoind. If I remember correctly however we do not crash immediately if bitcoind is unable to provide that data in this case.

I don't know enough about the protocol but does this pruned node is impossible with the current protocol spec?

It could have been solved by adding a merkle branch to the channel announcement. :(

Shouldn't when_lightning_became_cool be enforced by gossipd? As in, it would simply ignore such gossip.

Would be cool indeed, it would probably decrease the cost of a server to 5 USD a month or so.

[Sjors]

Pruning down to block 504499 means storing only 20 GB at the moment, 80 GB should last until the end of the year, at which point hopefully the problem is solved :-)

Similar benefits to low-end physical devices; a one-off savings of 160 GB is quite useful.

Closer to $10 / month savings on EC2. Even if there are cheaper competitors, the more options the better. 40 GB on AWS on magnetic storage costs $2.40 per month, using their previous generation volume. 200 GB would be $10 more. Modern sc1 storage is cheaper per GB, but you have to rent 500 GB minimum.

What part of the protocol prohibits it? Assuming you're willing to ignore the existence of channels whose initial announcement you missed. Does that cause problems for nodes that need to receive money (i.e. the payee can't find a route, because the receiving node doesn't know about a channel the sending node can see)?

There's always the option of fetching an old block temporarily, something perhaps bitcoind could add support for. But that does come at some DOS risk; you'd probably only want to do that occasionally and/or if it's related to a payment you're about to make or receive.

[ZmnSCPxj]

What part of the protocol prohibits it? Assuming you're willing to ignore the existence of channels whose initial announcement you missed. Does that cause problems for nodes that need to receive money (i.e. the payee can't find a route, because the receiving node doesn't know about a channel the sending node can see)?

Only payers/senders need to find a route; payees do not. In theory a payee need not build a routemap at all: it could just blindly forward gossip and not request a gossip dump on connection. Payees will still know of their direct channels (obviously, they have partial control over their direct channels after all), so if a direct channel is private it can at least offer it as a r field in an invoice to help those paying it to route.

So the issue is that as a payer, you want to know of as many valid open channels as you can, to increase the chance you can successfully pay. As a payee, you can ignore the rest of the universe other than your direct peers (but you would have to keep 24/7 watch over the channels with them).

Possibly you could have a payee node that does not maintain a routemap, and has direct channels to exchanges or some service that trades onLightning for onchain funds. If the direct channels become full on the payee side, it transfers the money to onchain funds (if the channel is not to an Lightning-to-chain exchange service, it would have to close the channels). In this case you could probably run on a pruned node that continuously prunes over time, running in O(1) storage space.

However such a payee-only mode may be very complicated, though.

[Sjors]

Payee mode is probably also too limited this way. It’s not great privacy and cost wise to close all channels everytime a shop needs to move their funds. Partial blindness to the gossip would seem more practical.

Maybe it just adds a boolean to the channel with “unverified” and then only when it needs to make a payment and can’t find a route without it, does it go out and fetch the relevant block. Then, if I understand correctly, it needs to check if the opening UTXO is still there. If not, it still ignores the channel.

[ZmnSCPxj]

Well, currently "fetch a block" is done via bitcoind, and if it is pruned then bitcoind will be unable to provide the block. There are vague future plans of adding a Neutrino mode where lightningd will use the Bitcoin P2P protocol directly, but currently there is no Bitcoin P2P protocol implementation at all, so that will take even longer to implement.

It may be feasible to simply let old gossiped channels that can no longer be verified (because the backing bitcoind pruned it out) to simply be removed from the routemap, at least as a first step to get lightningd somewhat working with pruned bitcoind.

[NicolasDorier]

@ZmnSCPxj I think this first step would be enough. Any idea why the current LN protocol does not include a partial merkle tree + transaction when announcing a channel so LN nodes don't need to fetch the block?

[ZmnSCPxj]

Presumably to limit scope, I think. There are plans to include various proofs at later date, here is one mention, that is similar to yours: https://github.com/lightningnetwork/lightning-rfc/issues/301#issuecomment-349918731

[jb55]

Had a random idea today. I wonder if we could add a "dynamic" prune mode to bitcoin so we don't have to worry about falling out of prune range after being offline for awhile.

Basic idea:

new bitcoin RPC call:

• keepheight <height>

This could be called when lightningd closes. This signals bitcoind to keep all blocks above <height> until keepheight -1 is called, and then pruning resumes as normal.

[Sjors]

That could cause the node to run out of disk space if c-lightning never comes back. Perhaps a second parameter could specify the max additional storage allowed before pruning kicks in regardless, e.g.:

• keepheight <height> <maxspace> with current_height and 300 MB (~ 1 day) being the default.

[Sjors]

TIL https://github.com/bitcoin/bitcoin/pull/10794, which introduces requestblocks, lets a pruned node fetch arbitrary historical block data.

[NicolasDorier]

That could be good... but if I understand we need to request one block per channel? This seems a huge amount of block to download.

[Sjors]

Only for channels that were created before your lightning node (new channels are tracked normally). And only for channels that you're considering using for a specific outgoing payment. See mailinglist for what I have in mind.

[NicolasDorier]

I plan to keep pruning for around 2 years of blocks to keep storage requirement fixed for BTCPay.

It only means that for new deployment, channels older than 2 years will not be seen right?

[Sjors]

storage requirement fixed

Keep in mind that SegWit hasn't been active for two years yet and blocks are nowhere near capacity, so you should keep a (2x?) margin.

Though maybe at some point we'll be able to (safely) prune witness data?

for new deployment, channels older than 2 years will not be seen

That's my understanding as well, but in practice I've seen regular crashes when running in pruned mode, probably because of old gossip. Though in your case there's 1.5 years to fix that :-)

[NicolasDorier]

The crash I have seen is when bitcoind was pruning faster than lightning node could scan. I don't seem to have issues by running clightning or lnd after it is completely sync.

[Sjors]

A rather different approach is suggested by @TheBlueMatt here:

I think this use-case is significantly more simply implemented with gettxoutproof/verifytxoutproof/decoderawtransaction. If you've already discovered from somewhere that you want a transaction, that somewhere can just as easily provide a txoutproof, and verifytxoutproof will already (AFAIR) tell you if the proof corresponds to a transaction on your (fully-validated) chain.

Perhaps we could add protocol messages to Lightning, asking other nodes for proof that a transaction is included in a block, which the pruned node can then verify with verifytxoutproof. Similarly you would ask the other node for the full transaction hex. Downside would be more information to gossip around and the need to hold on to it for a while as a courtesy to other nodes.

We can also obtain this information via the Bitcoin P2P network, but that seems to me not a good idea privacy wise.

[jonasschnelli]

We can also obtain this information via the Bitcoin P2P network, but that seems to me not a good idea privacy wise.

Why?

I think we should add a mode/parameter to bitcoind/getblock that allows pruned peers to fetch the block via p2p and return it through the getblock RPC call. It may take 2,3 seconds. It would just need some sort of DOS protection I guess.

Also, since the pruned peer has validated that block already, it should be sane to load it again via p2p in terms of the trust model.

A downside is probably the network work (wasted effort since the block gets just "consumed" by the pruned peer).

[ZmnSCPxj]

I have read somewhere that using spruned has been shown to work with C-Lightning. Is that an acceptable solution to this?

[Sjors]

Afaik spruned downloads blocks on demand. That seems like a DoS risk. I thought the 1.1 spec added SPV proofs to channel gossip? That should remove the need to download any block.

[ZmnSCPxj]

I thought the 1.1 spec added SPV proofs to channel gossip?

It was accepted as a Gossip Change: https://github.com/lightningnetwork/lightning-rfc/wiki/Lightning-Specification-1.1-Proposal-States#gossip-changes

See "Forwarding Bitcoin data over the overlay".

Until this is wildly debated and specced out and then implemented, there is still a need to download arbitrary blocks.

[kroese]

Has there been any progress on this?

The main problem is that currently c-lightning does not seem to be able to ignore blocks it cannot fetch. It tries endlessly in a loop, instead of giving up after a few tries, and ignoring the channel. This should be easy to fix, and would allow using it with a pruned node.

Then the second step would be to use the getblockfrompeer RPC call that has been added to Bitcoin, to retrieve the block from a peer, instead of ignoring channels.

But if only the first step could be implemented, I'm already happy!

[cdecker]

There are two options to configure CLN with a pruned node as a backend:

• Run an alternative backend such as Sauron or bcli4j. These fetch blocks and metadata from block explorers meaning you don't need a bitcoind at all
• Use the btc-rpc-proxy, which can be configured to fetch missing blocks on demand from the Bitcoin P2P network

The corner trusts explorers, while the latter may be a bit slower than having the blocks locally but it gets CLN the block after a couple of seconds, thus allowing it to verify the gossip.

[luke-jr]

keepheight <height>

Note this is basically implemented in https://github.com/bitcoin/bitcoin/pull/19463 (to be reopened shortly), which has been part of Knots for some time now.

[cdecker]

We might be getting pruned node support really soon :tm:

https://github.com/ElementsProject/lightning/issues/7201