search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.72k stars 2.09k forks source link

BOLT12 #5594

Open era-coder opened 3 years ago

era-coder commented 3 years ago

Please add BOLT12 to LND he BOLT 11 invoice format has proven popular, but has several limitations:

The entangling of bech32 encoding makes it awkward to send in other forms (i.e. inside the lightning network itself).
The signature applying to the entire invoice makes it impossible to prove an invoice without revealing its entirety.
Fields cannot generally be extracted for external use: the h field was a boutique extraction of the d field, only.
The lack of 'it's OK to be odd' rule makes backwards compatibility harder.
The 'human-readable' idea of separating amounts proved fraught: p was often mishandled, and amounts in pico-bitcoin are harder than the modern satoshi-based counting.
The bech32 encoding was found to have an issue with extensions, which means we want to replace or discard it anyway.
The payment_secret designed to prevent probing by other nodes in the path was only useful if the invoice remained private between the payer and payee.
Invoices must be given per-user, and are actively dangerous if two payment attempts are made for the same user.

please Read BOLT12.org for details

drvnoo commented 2 years ago

BOLT12 would be epic!

JimmyMow commented 2 years ago

Is this the correct place to track LND's progress in supporting BOLT12?

We're beginning to look into the best way to support BOLT12 for various efforts at Strike

Roasbeef commented 2 years ago

Is this the correct place to track LND's progress in supporting BOLT12?

Yeah you can view this as a tracking/brain storming issue. Also important to keep in mind that BOLT 12 is still evolving as the design meets real world design.

lnd development priorities

Our current set of priorities in terms of lnd development (roughly in order) are: improving the stability+security+scalability of lnd (should just run w/ minimal baby sitting, easy to run in a secure config, and be able to scale to meet the new levels of adoptions we're seeing w/ LN around the world), integrating taproot initially from an on-chain perspective -- ultimately leading to the first set of taproot native channels as they really expand the types of things we can do off-chain with channels, and path finding/routing improvements (better algos, trampoline, adding a payment level ACK, packet switching stuff, etc).

Most of our development resources currently allocated to the above items, along with more ad-hoc pushes to fix serious/critical bugs. Given the large design space of both LN the protocol, as well as LN implementations, each team needs to make tradeoffs to work particular items, as given the amount of innovation/ideas/proposals, it's difficult for a team to implement+review+develop everything while also making sure an adequate amount of maintenance+stability is in place. It's also somewhat unavoidable that all implementations won't roll out new feature in lock step (as an example only this year did another node join lnd in using anchor channels by default). Even today there're several protocol extensions that only a single implementation has fully implemented + rolled out, and in many cases due to the extensibility of the protocol, they're able to use those new features without requiring the entire network to adopt them.

In the past we admittedly moved a bit too quickly on new features (both at the protocol and RPC/API level), which resulted in existing bugs and stability improvements not being properly prioritized. My aim is to start to reverse that trend this year, slowing down on new features and instead ensuring that lnd the software is ready to shine in primetime and won't buckle under the load and is easy to run securely.

bolt 12's surface area

Given the above, on the topic of BOLT 12, it admittedly isn't a very high priority for the project atm. What people colloquially refer to as BOLT 12 is actually the composition of several (possibly independent protocol features) including:

and much more. In short, there's a lot packed in there and at this point, no one implementation has actually implemented it all to my knowledge.

At the current cross roads, if we had to pick between BOLT 12 and taproot (alongside making lnd more stable/secure/scalable), we'd pick taproot, as partial solutions to everything BOLT 12 is trying to tackle already exist, but taproot allows entirely new interactions/protocols off-chain. I think that given BOLT 12 encompasses so many layers of the protocol stack (from internal routing messages all the way up to wallet/service payment negotiation), at times conversations about the proposal end up missing a lot of nuance w.r.t what it "competes" with in the wild. For example, it's entirely possible for LN-URL to actually serve BOLT 12 invoices in the confines of the existing protocol. A static AMP invoice can even be served over the BOLT 12 invoice negotiation protocol.

decomposing bolt 12 + iterative deployment

I think in order for the proposal to move forward, the community and implementations should adopt of piece-wise approach. The entire proposal all together is a massive under taking, but it can be broken up into initial phases like just adding that new invoice format (we get pure TLV, more extensible, no weird bech32 stuff -- all great stuff). Even just that may take some time as BOLT 11 is the current default across all wallets/services. Should wallets be ready to scan/display both invoices? Just switch to the new format? Convert between BOLT 11 and 12? I'm not sure what the best path forward is there. I also wouldn't underestimate the timeline to fully convert to the new format. With segwit it took years for services to update, and even now w/ taproot, even though the "new" address was just a constant change (bech32m), some services still aren't able to send to taproot addresses.

onion messaging concerns

The part of BOLT 12 that gives me personally the greatest pause is the onion messaging component. In theory, it sounds dope: any node can just send any other node a message tunneled over LN using our existing onion encryption stuff. In the past I even advocated for similar schemes, like streaming video of the network using HORNET. Several years later, I'm a bit more apprehensive to introducing generalized messaging as, then we'd need to still ensure LN is an acceptable payment network and an arbitrary messaging protocol. Can LN be LN, and also a better version of Tor? It seems like a lot to tackle at once, and getting DoS and rate limiting right isn't going to be a walk in the park.

Today with onion messaging the idea is you just sling out the message (maybe across concurrent paths) to any other node on the network. The lack of session creation means that any node can spam any other node, as you don't need to "accept" the circuit creation. As the message is meant to be forwarded on a best effort basis, it can be dropped or even prematurely rate limited across the entire network, meaning even just something like fetching an invoice can fail. Compare this to the situation today, where a user can reliably obtain an invoice and execute a payment immediately. If the BOLT 12 we talk about today just uses the inner invoice format as the default (fetching isn't always the default) we can achieve a similar flow and take advantage of the new bits of the invoice.

adding metered payments and explicit circuit creation to onion messaging

I've been working in the background on a payment metered way to allow explicit session creation, and also allow nodes to natively charge for any onion messages forwarded. The end protocol resembles HORNET, but fits pretty nicely into our existing protocol. IMO if given the choice to forward arbitrary messages in the network for free vs be paid for that service, I think most node operators would choose the latter. Given that we still dealing with all the various griefing vectors today in the protocol as a result of the initial design, we should be careful to roll our new similarly exploitable extensions without thoroughly considering how to enforce rate limiting spam, and incentive alignment in the first version.

does onion messaging actually enable direct p2p mobile payments?

One last thing I think is lost in the current discourse is that the onion messaging component of BOLT 12 is meant primarily to allow direct p2p payments. In theory, if your phone can just fetch an invoice directly from my phone (or w/e) then that's ideal. Ignoring the fact that you can just fetch the invoice if you're scanning the invoice from me directly (or another site), the missing link is a way to accept payments while remaining offline. If my phone goes to sleep right when you send, the payment will fail. What's needed is a way to wake up my phone to have it both respond to the invoice fetch request and also sign the updates needed to receive funds. If you lean towards things like Firebase or apple push notifications, well then you're relying on "servers" once again. From a protocol design perspective, imo it's pragmatic to lean on higher level infrastructure when needed vs trying to do everything within our desired medium. If node implementations all packaged tor onion services, then maybe we would just use that instead and be able to ignore all the implications of making our own generalized messaging system?

drvnoo commented 2 years ago

Relying on Google/Apple to wakeup the app is not optimal, but it's the only solution we got. The data sent in the notification will ofc be encrypted and you can further improve the privacy by mixing in random fake notifications.

It might also be possible to encrypt a timer in the notification for when the real payment is expected and have the wallet wake up itself.

It's not optimal, but users will be able to receive offers in sleep mode while Google only sees encrypted uncorrelateable crap.

kingonly commented 2 years ago

Relying on FCM to wakeup the app is unreliable. Behavior is unpredictable and dependent on the specific os. Also it doesn't provide a solution to other use cases (e.g. browser nodes). Offline receiving is one the problems we must tackle within Lightning if we want it to scale.

saubyk commented 2 years ago

@Roasbeef thanks for sharing your perspective on BOLT12, which I think was sorely needed. Couple of points to consider:

  1. Agree that BOLT12's scope is significant, but implementations can break it down and develop it gradually, you've remarked that as well. E.g. only supporting onion messaging is a good start, as it helps other implementations, without adding significant dev load on your team.
  2. Lack of good solution for mobile offline payments, shouldn't mean that BOLT12 implementation should be held off for the nodes as well. As it's a significant improvement of the payment functionality aiding tipping and donations campaigns run with nodes.
Ziya-Sadr commented 2 years ago

but it can be broken up into initial phases like just adding that new invoice format

wouldn't such an approach (breaking it into initial phases) mess up compatibility and confuse users and their experience, even more? (I mean in case BOLT 12 gets implemented by wallets and other LN implementations in the future)

joostjager commented 2 years ago

I think it should be possible to implement bolt12 externally? Use custom peer messages to handle the bolt12 traffic in a separate process.

Maybe the onion message routing is more complicated, but just implementing the end point behavior looks to be feasible.

Roasbeef commented 2 years ago

I've been working in the background on a payment metered way to allow explicit session creation, and also allow nodes to natively charge for any onion messages forwarded. The end protocol resembles HORNET, but fits pretty nicely into our existing protocol.

Hi y'all, I just posted a draft proposal to the ML detailing this idea: https://lists.linuxfoundation.org/pipermail/lightning-dev/2022-February/003498.html. If everything holds up, then this can add a way to allow nodes to be paid for forwarding onion messages, and also let senders accept/reject them.

@saubyk

Agree that BOLT12's scope is significant, but implementations can break it down and develop it gradually, you've remarked that as well

Yep that's what I suggest in my initial comment.

Lack of good solution for mobile offline payments, shouldn't mean that BOLT12 implementation should be held off for the nodes as well

Totally, I didn't entail that it should be. Instead I was trying to draw attention to the fact most proponents of onion message really like it for the mobile p2p use case, but it alone doesn't fully support that use case in isolation.

@Ziya-Sadr

wouldn't such an approach (breaking it into initial phases) mess up compatibility and confuse users and their experience, even more?

Such initial fragmentation is unavoidable IMO. Even if all the major implementations release new versions that support the bolt 12 invoice format tomorrow, there'll still be older nodes that are unable to scan/parse/pay those invoices.

carlaKC commented 2 years ago

Following up on some conversations in Oakland, I'd like to outline a proposal for adding the ability to pay BOLT12 offers in lnd. While this is only half of the work, my hope is that adding the ability to pay offers in lnd will get us on the road towards deprecating BOLT11, and a common invoice format.

I'd like to implement this in an experimental offers subserver, so that the code is (so far as possible) isolated to a single location. I'm aware that this is a non-trivial review and maintenance cost for the team, so would also be open to initially implementing this externally to lnd if preferred, and limiting the change set to those required to support offers using only the APIs (eg, a carve-out to allow custommessage to relay onion messages) if there would be more enthusiasm for those changes? Would like to get some direction on this so that I don't plunder on in the wrong direction 🙏

Order of Operations

  1. Support forwarding blinded routes
  2. Add ability to pay bolt 12 invoices
  3. Add ability to decode bolt 12 offers
  4. Direct Connection Offers
  5. Add support for paying blinded routes

Blinded Route Forwarding

Adding support for forwarding htlcs in blinded routes allows lnd nodes to support network payments to BOLT12 invoices with blinded routes. This is good for reliability, since having more nodes that support blinded routes increases your selection options.

Adding support here is also fairly lightweight, because all we need to do is grab an ephemeral key out of update_add_htlc and decode the encoded_data provided in the onion and use the fees/timelock suggested. This change would need to be internal to lnd, because we don't have incoming link htlc interception (and I don't think it would be valuable to add).

Pay BOLT12 Invoices

Add understanding of the new invoice structure, initially without support for blinded routes. There is quite a lot of functionality embedded in BOLT12 invoices, so I'll start with a minimal subset to keep the change small (excluding fields like currency, for example). While the typical offer user wouldn't directly interact with BOLT12 invoices directly (they scan an offer and the invoice is retrieved), this is a useful building block for breaking the change down into parts.

Add BOLT12 Offer Decoding

Add the ability to decode offers, implementing a minimal subset of TLVs required for basic invoice_request functionality.

Direct Connection Offers

When presented with an offer, add the ability to make an ad-hoc connection to retrieve an invoice and pay it. Once we have this functionality, we can always switch out the "direct connect" part with full onion messages in the background without any changes to the API.

  1. Directly connect to peer: Make an ad-hoc connection to the peer to relay single-hop onion messages.
  2. Create invoice request, handle response: Create an onion message with an invoice_request and handle the invoice_response.
  3. Pay BOLT 12 invoice: Pay the invoice returned, using the funcionality above.

Add support for blinded routes

Once offers are working, we can come back and add support for blinded routes. This change will require the following:

  1. Update BOLT12 supported TLVs: add blinded routes to the set of known even TLVs in our invoices.
  2. Update pathfinding to support blinded routes: pathfinding needs to be able to start with the aggregate CLTV and fees indicated by the blinded route.
  3. Update route construction to include blobs: payments to blinded routes need to include encrypted data and ephemeral keys for the blinded route.

Out of scope (for now)

I've omitted the ability to create/receive BOLT12 offers/invoices because the above is a large enough bite of the elephant already, and I'm mindful that LND has ongoing work in the invoice registry (#6288 / #6176, afaik) which needs to be completed before we make any more large changes there. (Will help with review where I can!)

My hope is that sender-support will provide wallets/applications with the opportunity to demonstrate their commitment to supporting offers through the "proof of work" of adding it to their application, and we can confidently proceed with the large changes required to add receiver-side functionality.

Roasbeef commented 2 years ago

Thanks for putting together this proposal!

Order of Operations

Wouldn't steps 1 and 2 be reversed? AFAIK, there're still a number of details re which privacy leaks are "acceptable" (fees, timelock, path length, etc) so blinded hop processing isn't something yet that's fully defined? Also one would need to concurrently implement blinded route selection/construction as well to make sure everythign actually worked e2e.

There is quite a lot of functionality embedded in BOLT12 invoices, so I'll start with a minimal subset to keep the change small (excluding fields like currency, for example)

Does this include a step to first extract out the invoice format (including test vectors, etc) into a separate extension (?) BOLT document?

I'd like to implement this in an experimental offers subserver, so that the code is (so far as possible) isolated to a single location

I'm not sure it's possible to actually isolate all the code to a single location given the new invoice format would have rippling effects on: internal invoice tracking and the invoice persistence itself. IMO re-implementing the entirety of the invoice registry all over again for a slightly different invoice format isn't a very good use of time, and is bound to re-introduces existing bugs we've already fixed in the area. IMO we should first complete the first few steps of this issue which would give us a more solid base to work off of, and also allow the invoice registry to be instantiated completely outside of lnd, which would be a boon for projects like lnmux. I also really would't underestimate the scope of #6288, which IMO is a perquisite for any further work in the invoice registry.

Add BOLT12 Offer Decoding

If the invoice format is extracted, then why is offer decoding necessary? At least from the PoV of this simplified scope.

Directly connect to peer: Make an ad-hoc connection to the peer to relay single-hop onion messages.

The order of operations above doesn't mention onion message at all, I assume this was just an oversight? On the topic of onion messages itself, what type of rate limiting and DoS prevention would be implemented?

Similar questions above: if the invoice format is extracted, then why would nodes need to implement the extra step of connecting out to fetch another (?) invoice. I've never really been convinced that the direction connection doesn't lead to the exact same IP leaks the whole onion messaging thing is meant to resolve.

Update route construction to include blobs: payments to blinded routes need to include encrypted data and ephemeral keys for the blinded route.

I think we'd want some upfront design/heuristics before this step w.r.t: how to select/place blinded routes in practice to actually maximize any added privacy benefits, while also minimizing any reachability degradation as a result of requiring a more constrained path to the node.

Another step we can do before any thing outlined here is: accept prior fee policies at the forwarding level for a grace period. Otherwise blinded paths are pretty fragile since a single fee update may mean that none of the paths are actually workable.

As a short cut to skipping to feeling out the above (wherein IMO lies the greatest set of unknowns), we could add basic blinded paths support to BOLT 11 itself. Re QR codes it can be minimal, and in areas where LN-URL is used, there're no encoding issues.

I've omitted the ability to create/receive BOLT12 offers/invoices because the above is a large enough bite of the elephant already,

AFAICT, this is described in the "direction connection offers" section?

This change would need to be internal to lnd, because we don't have incoming link htlc interception (and I don't think it would be valuable to add).

Why don't you think this would be valuable? AFAICT, if this is added, then this would really speed up all the work outlined above. I think the main incision point would be here: https://github.com/lightningnetwork/lnd/blob/master/htlcswitch/link.go#L146-L152. This is what would be able to detect blinded paths usage, and transparently handle all the new logic w/o the link needing to be aware of what's going on. To be able to swap this out in when building an lnd instance, we'd add an interface here, then allow it to be specified here when building an lnd instance. IMO this is independently useful, as it allows lnd to be instantiated with a fully external onion payload processor.

Roasbeef commented 2 years ago

Re route blinding: @ellemouton has some code she put together during some initial investigation that'll be made into a proper PR 🔜

carlaKC commented 2 years ago

Thanks for the detailed response @roasbeef! Really appreciate your time looking into different ways to cut this up and alternative approaches. On your suggestion, I’ve spent some time looking into the changes LND would need to implement paying bolt 12 offers externally:

This is not including forwarding htlcs in blinded routes, or creating offers so that we can be paid, since both are much more involved. I’ll spin up separate issues on what we’d need to externally support those once I’ve investigated more thoroughly?

ellemouton commented 2 years ago

Re route blinding: @ellemouton has some code she put together during some initial investigation that'll be made into a proper PR 🔜

here it is! https://github.com/lightningnetwork/lightning-onion/pull/57 cc @carlaKC

l0k18 commented 2 years ago

I just would like to make it known that I 100% agree with the following, as I am working specifically on a source routed dVPN that has a little similarities to HORNET and the sphinx-style BOLT4 onion routed payments, but is focusing on being a network transport. Here: Indranet

onion messaging concerns

The part of BOLT 12 that gives me personally the greatest pause is the onion messaging component. In theory, it sounds dope: any node can just send any other node a message tunneled over LN using our existing onion encryption stuff. In the past I even advocated for similar schemes, like streaming video of the network using HORNET. Several years later, I'm a bit more apprehensive to introducing generalized messaging as, then we'd need to still ensure LN is an acceptable payment network and an arbitrary messaging protocol. Can LN be LN, and also a better version of Tor? It seems like a lot to tackle at once, and getting DoS and rate limiting right isn't going to be a walk in the park.

Today with onion messaging the idea is you just sling out the message (maybe across concurrent paths) to any other node on the network. The lack of session creation means that any node can spam any other node, as you don't need to "accept" the circuit creation. As the message is meant to be forwarded on a best effort basis, it can be dropped or even prematurely rate limited across the entire network, meaning even just something like fetching an invoice can fail. Compare this to the situation today, where a user can reliably obtain an invoice and execute a payment immediately. If the BOLT 12 we talk about today just uses the inner invoice format as the default (fetching isn't always the default) we can achieve a similar flow and take advantage of the new bits of the invoice.

adding metered payments and explicit circuit creation to onion messaging

I've been working in the background on a payment metered way to allow explicit session creation, and also allow nodes to natively charge for any onion messages forwarded. The end protocol resembles HORNET, but fits pretty nicely into our existing protocol. IMO if given the choice to forward arbitrary messages in the network for free vs be paid for that service, I think most node operators would choose the latter. Given that we still dealing with all the various griefing vectors today in the protocol as a result of the initial design, we should be careful to roll our new similarly exploitable extensions without thoroughly considering how to enforce rate limiting spam, and incentive alignment in the first version.

does onion messaging actually enable direct p2p mobile payments?

One last thing I think is lost in the current discourse is that the onion messaging component of BOLT 12 is meant primarily to allow direct p2p payments. In theory, if your phone can just fetch an invoice directly from my phone (or w/e) then that's ideal. Ignoring the fact that you can just fetch the invoice if you're scanning the invoice from me directly (or another site), the missing link is a way to accept payments while remaining offline. If my phone goes to sleep right when you send, the payment will fail. What's needed is a way to wake up my phone to have it both respond to the invoice fetch request and also sign the updates needed to receive funds. If you lean towards things like Firebase or apple push notifications, well then you're relying on "servers" once again. From a protocol design perspective, imo it's pragmatic to lean on higher level infrastructure when needed vs trying to do everything within our desired medium. If node implementations all packaged tor onion services, then maybe we would just use that instead and be able to ignore all the implications of making our own generalized messaging system?

What this commentary has prompted me to respond about is, is there some way I can avoid duplicating effort with my work, because it seems really obvious to me that this onion routing for messages is a whole project in itself and I would love to collaborate on getting the best solution together and out there.

The laggy timeouts and high general latency of Tor make it quite a bad fit with both bitcoin P2P - indeed, long lived connections of any kind, and that naturally includes Lightning channels as well. Tor is a solution specifically tailored to carry TCP packets and work with the highly intermittent connections used by web services, and not for gossip or peer to peer connections.

Rereading, looking at the last paragraph I quoted, I had already envisioned a way of dealing with this problem where wifi was enabled but the user's phone didn't have internet over mobile network or the wifi. With open wifi, nodes can connect to it and scan for a router, and attach to it and be able to directly run the transaction, or at least use the routing to access the user's home base router and run the funding that way. Running a lightning node on mobile is not very practical. The best way I envision is you have your own home base, which has a full btcd and lnd node running on it, using neutrino on the mobile device and as well neutrino would allow chain data access over the hypothetical wifi only connection with a router on it.

Yes, it's important to restate: Indra will not be standalone, it will be running and managing a full node and LN node and relaying a lot of messages to and fro, it was my plan to start with the restriction of network "exit" only to mean accessing btcd and lnd with source connection privacy, and thus also enable these two servers to connect privately to peers instead of directly, thus enabling the concealment of the origin of bitcoin transactions as well as one side of lightning channels.

saubyk commented 1 year ago

To round out the discussion on this thread, here is the current plan of action for BOLT12 adoption on LND. This plan was presented in our last roadmap presentation. Adding the slides here for general visibility

Lightning Labs_ Roundtable Roadmap (February '23) BOLT12-1

Lightning Labs_ Roundtable Roadmap (February '23) BOLT12-2

carlaKC commented 1 year ago

External daemon repo here: https://github.com/carlaKC/lndk + write up of onion message architecture here: https://github.com/carlaKC/lndk/issues/2 for the intrigued!

augustresende commented 1 year ago

Any news?

carlaKC commented 1 year ago

Any news?

7267 merged: LND can send payments to blinded routes (requires deconstructing a bolt 12 invoice)

7376 in progress: will allow LND to forward blinded payments on behalf of other nodes.

MichaelAntonFischer commented 5 months ago

Hi,

do I understand it correctly that version 0.18.0 now fully implemented Bolt12?

guggero commented 5 months ago

No, just Phase II as outlined above (the version numbers aren't in sync anymore).

MichaelAntonFischer commented 5 months ago

Ah so sad. It would make my life sooooooo much easier.

prusnak commented 4 months ago

Can you confirm that the Phase II is delivered? What is the state of Phase III? Any WIP PRs you can link to?

saubyk commented 4 months ago

So Phase II will be complete with 18.3, when route blinding with bolt11 will be delivered. Related prs: https://github.com/lightningnetwork/lnd/pull/8752 https://github.com/lightningnetwork/lnd/pull/8735 https://github.com/lightningnetwork/lnd/pull/8764

One part missing in the above roadmap is onion messaging, which is planned to start with 0.19. Right now we are analyzing onion messaging changes along with mitigations for potential DoS vectors. No prs are up yet for that, but you should progress on that in the coming months. Phase III in all earnest will begin with onion messaging.

Also, this roadmap is now outdated, but we are making consistent progress towards BOLT12 with each lnd release.