Lightning integration

[artemii235]

• [x] Look for Rust libs/crates to work with Lightning.
• [x] Research whether HTLCs are possible.
• [x] Implement coin activation and balance display in MM2.
• [x] Implement external backup for lightning channels.
• [x] Implement payments and channels History.
• [x] Add SPV validation before confirming transactions in electrum mode.
• [x] Move BackgroundProcessor back inside LightningCoin
• [x] Release used port on coin deactivation
• [ ] Support notifications / streaming channel for LightningCoin
• [ ] Implement order matching and swaps for LightningCoin. BTC and Lightning-BTC should have different orderbooks as swapping BTC to lightning balance can be used to refill lightning channels outbound liquidity. Also instead of closing a channel the reverse swap from lightning to on-chain BTC can be done to get some of the channel liquidity on-chain.
• [ ] Compilation to WASM.
• [ ] Lightning derivation path https://github.com/KomodoPlatform/atomicDEX-API/issues/740#issuecomment-1329365708 https://github.com/KomodoPlatform/atomicDEX-API/pull/1512#discussion_r1010326620
• [ ] Cloud/Telegram/IPFS channels backup research.
• [ ] Taker fee implementation, also need to include proofs of swap in SwapStatus (Proof here https://github.com/KomodoPlatform/atomicDEX-API/issues/1045#issuecomment-1410449770 can be reused fro dex fee proof)
• [ ] Add test_deserialize_lightning_swap_status similar to the iris one here https://github.com/KomodoPlatform/atomicDEX-API/pull/1608
• [ ] https://github.com/KomodoPlatform/atomicDEX-API/pull/1591#discussion_r1063540459
• [ ] https://github.com/KomodoPlatform/atomicDEX-API/pull/1339#discussion_r921307635
• [ ] https://github.com/KomodoPlatform/atomicDEX-API/pull/1452#discussion_r968295686
• [ ] Add P2P transaction helper broadcast for lightning-related transactions.
• [ ] Implement rapid gossip sync mode to be used for mobile
• [ ] Implement native mode for lightning
• [ ] Implement Onion addresses.
• [ ] Research how to use watchtowers with our implementation of lightning.
• [ ] If more lightning coins to be supported in the future (e.g LTC). Swaps and locktimes need to support swaps between 2 lightning coins. Also research if dynamic locktime for lightning swaps is needed so that taker can use more than 3 hops.

@shamardy Please post the documentation links, notable libraries, and intermediate milestones reports to this issue comments.

[shamardy]

LDK/Rust-Lightning is the only good rust lightning network library/crate that exists, there are other libraries that implement LN in Rust but only partially. It also states in the about section that it's designed for integrating lightning with custom features such as our own chain sync/key management/data storage/backup logic.

[shamardy]

Update on whether HTLCs are possible on LN. HTLCs are native to the lightning network as they enable payments between 2 lightning addresses that don't have a direct payment channel open between them. this is done through trustless routing of HTLC payments via a network of channels to open a path for payment between the 2 addresses. Also, atomic swaps between lightning network and on-chain blockchain addresses already exist and are called submarine swaps. Submarine swaps are mainly used to fund lightning payment channels directly with BTC instead of closing and opening the channel with more funds which cost more in transaction fees. The submarine swaps concept can be extended to swap lightning BTC with other chains and has been implemented before with LTC and BCH.

[shamardy]

A few Issues we should take into consideration when Integrating lightning to mm2.

• In lightning, the parties involved in a transaction must be online during the transaction to prevent fraudulent behavior. If a party goes offline for an extended period of time for any reason an execution fork attack can be carried out. An execution fork can revert a payment channel’s history, potentially causing financial damage to a party that is innocent except for having crashed. To mitigate this watchtowers have been proposed. Watchtowers are third-party services that enable such parties to delegate the cancelation of execution forks on their behalf. I think we should have our own watchtower server/service to prevent such attacks when a user of mm2 goes offline for whatever reason while using the lightning network for a swap.

• Another issue that is also solved by third-party services is channel capacity. Presumably when a swap using LN and another chain needs to occur both the sender and the receiver need to open a channel (BTC multisig wallet) between them. Opening a channel is costly since it involves an on-chain transaction. This is solved by using multihop payments which involves forwarding payments through channels that already exist if a route can be found between the sender and the receiver. But the problem is that the amount that can be sent is capped by the minimum amount available between the hops that the payment can be routed through, this is where the third-party service comes into play as they provide a channel with big capacity that LN users can open a channel with and can find each other through it. I don't know if we can provide such service or if we integrate with such services so that when an mm2 user opens a lightning channel for the first time uses it providing easy routes for all mm2 users otherwise swaps on LN can be capped to very low amounts.

• The last issue is not really an issue but a proposal of how to chose between transacting BTC on LN or on-chain. The lightning network fees grow linearly with the amount transacted, so LN swaps are perfect for small swaps. But if the amount is big enough the on-chain transaction fees can be lower than LN fees, this is not taking transaction speed into consideration of course as LN transactions are a lot faster. So when implementing LN fees in mm2 in the future we should take this into consideration.

[shamardy]

@artemii235 The next step will be to implement an HTLC PoC on the testnet following this example https://github.com/KomodoPlatform/atomicDEX-API/blob/mm2.1/mm2src/coins/eth/eth_tests.rs#L196 as we discussed.

[shamardy]

The following is the best technical book on the lightning network (Maybe the only one) https://github.com/lnbook/lnbook it's equivalent to the "Programming Bitcoin" book but for LN. It's not officially released yet but should be in Jan of 2022. They released the work on progress and now the final production version on Github.

[artemii235]

We should consider researching the Joule in regards to Lightning web wallet support: https://lightningjoule.com/ https://github.com/joule-labs/joule-extension cc @ca333 @tonymorony

[shamardy]

Just opened the lightning payments PR.

After the review process and merging to dev, the next step should be persisting lightning payments and channels history (closed channels) to storage (SQLite) and implementing channels monitors external backup.

Payments history is analogous to transactions history for other coins, as for monitors backup, this is required because lightning channels can't be reconstructed from the seed only unlike other coins so backup to external drive or cloud services is required.

Once both of these functionalities are implemented minimal lightning wallet functionalities can be considered ready.

I updated the above checklist with the upcoming steps. The ordering of these steps can be changed based on priorities.

[shamardy]

After a swap P.O.C with lightning coin as taker was implemented, the next step would be to implement locktimes correctly for lightning. This will let us continue the implementation of maker swaps and swaps refunds for lightning coin. Here are some notes on my research about how swap locktimes can be implemented for lightning:

Assumptions

• After some discussion with @artemii235 we agreed that it's better for lightning locktimes to stay constant same as other coins. I discuss in the upcoming notes what can be a safe value for such locktimes.
• Locktimes in lightning payments are block height based not time based, we agreed to use an estimated time using average block times similar to how it's already done in tendermint coin. https://github.com/KomodoPlatform/atomicDEX-API/blob/2c1524dfdcaad68ccabbefc7caa3dfd73e8926d5/mm2src/coins/tendermint/tendermint_coin.rs#L655

  Payments HTLCs

• Timelocks are a part of any lightning payment, it allows a lightning node to rollback/refund a payment if the payment recipient doesn't claim it for any reason.
• This rollback/refund is insured by an exchange of new commitment transactions between every pair of any channel in the payment path/s when a payment is routed through this path.
• In this new commitment transactions a new HTLC is added for the new payment going through this channel.
• A delayed spending path is added in the payment HTLC back to the side of the channel that offered the HTLC for the purpose of refund.
• The other spending path depends on the receiving party knowing the payment preimage, meaning the payment reached it's intended recipient.
• If a payment is not claimed by the recipient, the parties update the commitment transactions by removing the HTLCs and revoking the old commitment transactions.
• If a counterparty is offline or refuses to update the commitment transactions for a certain amount of time, the un-updated commitment transaction is broadcasted and the HTLC can be spent on-chain by one of the two paths.
• This depends on if the payment is claimed or not by the recipient, if claimed, the receiving side of the HTLC will have the preimage.
• The locktime cltv_expiry_delta in the HTLC is chosen to give the receiving party a window to spend the first path using the preimage.
• If the locktime has passed, the sending party can spend the delayed spending path on-chain.
• cltv_expiry_delta was chosen to be large enough (at least 34 blocks) to give the routing parties the chance to appear back online and claim the payment on-chain if the channel between them was closed.

  cltv_expiry_delta

• cltv_expiry_delta is negotiated by the counterparties when opening a channel between them and remains the same value unless the channel is updated. If the channel was updated, the HTLCs in the commitment transactions are not updated with new locktime values of course. Only new HTLCs for new payments use the new value.
• Common values for cltv_expiry_delta:
  • Most nodes use a cltv_expiry_delta of 40 blocks. We can call this value MEDIAN_HOP_CLTV_EXPIRY_DELTA. This is the value that will be used in all the assumptions to calculate a safe locktime for swap payments.
  • The minimum allowed cltv_expiry_delta value for channels opened with rust-lightning is 42 blocks.
  • The default value that is currently used in mm2 lightning implementation is 72, I guess it can be reduced to 42 to be similar to most nodes.
• MEDIAN_HOP_CLTV_EXPIRY_DELTA is the locktime for only 1 hop:
  • For a 1 hop payment path consisting of 3 nodes (a sender, a routing node and a receiver) there are 2 channels, meaning there are 2 HTLCs:
    • The first HTLC is between the sender and the routing node with a cltv_expiry_delta that is equal to the MEDIAN_HOP_CLTV_EXPIRY_DELTA
    • The second HTLC is related to the last hop and has a different cltv called final_cltv_expiry. final_cltv_expiry must be greater than a value determined by the receiving node in the invoice. This value is called min_final_cltv_expiry.
    • min_final_cltv_expiry has a minimum allowed value of 24 blocks in rust-lightning. This is the value that is currently used in swaps implementations.
    • The final_cltv_expiry is increased by a random value to obfuscate who is receiving this payment to the routing nodes, this is required since lightning payments should be private. This obfuscating is done by a concept called shadow routing, this consists of adding a cltv value of a random chosen real route after the last hop consisting of 1 to 3 more hops.
    • final_cltv_expiry = min_final_cltv_expiry + shadow_route_cltv_expiry or for the maximum value that we will use for swap locktimes calculations is min_final_cltv_expiry + 3 * MEDIAN_HOP_CLTV_EXPIRY_DELTA
    • In general, the median total payment expiry time for a multi hop payment can be calculated as following: median_total_cltv_expiry = (no_of_hops + 3) * MEDIAN_HOP_CLTV_EXPIRY_DELTA + min_final_cltv_expiry
    • Here are a few examples of routing to demonstrate the above.

      Maker Swap Locktime if LightningCoin is Taker

• A good maximum number of hops needs to be determined to calculate a safe Maker Swap Locktime.
• In my opinion 3 hops are good enough given that users should have channels open to good connected nodes. If there is no route between the 2 nodes making the swap their orders shouldn't be matched anyways.
• For a maximum of 3 hops the median_total_cltv_expiry for the taker payment will be 264 blocks. This means that the maker payment in the other coin should have a locktime of 528 blocks which is about 3 days and 16 hours. If something goes wrong in the swap the maker will need a lot of time to refund his payment, by reducing the number of hops for swap payments to 3 we reduce the probability of the lightning payment swap to fail too. We can reduce the maker locktime by not making it double the taker locktime in this case but I am not sure if this is the right call. What do you think @artemii235?

  Taker Swap Locktime if LightningCoin is Maker

• Current used locktimes for swaps in mm2 can be used for taker non-lightning payments Locktimes if LightningCoin is Maker. The only value of importance here for the taker is min_final_cltv_expiry which should be double the taker payment locktime.

[artemii235]

@shamardy Thanks for very detailed research! 🙂

The median_total_cltv_expiry for the maker payment will also be 264 blocks. In this case the taker swap locktime will be 22 hours. I believe this is acceptable.

Yes, this seems acceptable. Though,

This means that the maker payment in the other coin should have a locktime of 528 blocks which is about 3 days and 16 hours.

is a bit too much 🙂

We can reduce the maker locktime by not making it double the taker locktime in this case but I am not sure if this is the right call.

We can try it - with such a long lock duration, even 1,5x will give taker enough time to act even if his payment is spent right before locktime expiration.

Also, a couple of questions: examples demonstrate the node having cltv_expiry_delta=10. Why rust-lightning uses 42 as default value and why it can't be decreased? Lowering it to minimum recommended 34 value could give us a noticeable outcome.

(no_of_hops + 3) - I couldn't find an explanation on why 3 is added to number of hops? 🙂 Can this constant be decreased?

[shamardy]

The median_total_cltv_expiry for the maker payment will also be 264 blocks. In this case the taker swap locktime will be 22 hours. I believe this is acceptable.

Yes, this seems acceptable. Though,

First, Just adding a note about this to not forget :) The taker has to make sure that the final_cltv_expiry (last hop cltv) for the maker lightning payment is at least double his (other coin) swap locktime. I will add this info to the above research comment to have a complete picture in one comment.

(no_of_hops + 3) - I couldn't find an explanation on why 3 is added to number of hops? 🙂 Can this constant be decreased?

The 3 is related to the maximum shadow route hops. As shown by the below quote from the my original comment.

The final_cltv_expiry is increased by a random value to obfuscate who is receiving this payment to the routing nodes, this is required since lightning payments should be private. This obfuscating is done by a concept called shadow routing, this consists of adding a cltv value of a random chosen real route after the last hop consisting of 1 to 3 more hops.

I believe it can be decreased, or even be zero in some cases :) The max_total_cltv_expiry_delta that can be specified in the route parameters when finding a route for the payment can be set to a value of no_of_hops * MEDIAN_HOP_CLTV_EXPIRY_DELTA + min_final_cltv_expiry which in case of 3 hops equal to 24 hours / 1 day. If the route/path found has a total cltv of exactly 24 hours (path_total_cltv_expiry_delta), no shadow route will be added. If path_total_cltv_expiry_delta is less than 24 hours then a cltv less than or equal to the remainder from 24 hours will be added. In this case maker locktime can be 1.5 days if 1.5x is used. In general we can specify any constant maker locktime we want and the taker will have to find a route that matches the constraints, but for 3 hops as stated 1.5 days can be a good compromise, if the locktime is increased more routes can be found.

examples demonstrate the node having cltv_expiry_delta=10.

This is for demonstration purposes only.

Why rust-lightning uses 42 as default value and why it can't be decreased? Lowering it to minimum recommended 34 value could give us a noticeable outcome.

Most nodes use cltv_expiry_delta = 40 in their configs. It can be verified by checking the channels of the most connected nodes here, 40 and 144are the most used numbers. I believe 40 is chosen because it's the default number in LND sample config, it gives a bit more safety than the recommended minimum of 34. I am not sure why 42 was chosen by rust-lightning, they maybe rounded the number of hours to 7 hours, or added 2 blocks to give the node better safety against chain reorganizations / transaction to appear on-chain. Also 42 is the answer to life the universe and everything 😁

[shamardy]

First, Just adding a note about this to not forget :) The maker has to make sure that the final_cltv_expiry (last hop cltv) is less than the taker swap locktime (Assumed 22 hours for now). I will add this info to the above https://github.com/KomodoPlatform/atomicDEX-API/issues/1045#issuecomment-1310604403 to have a complete picture in one comment.

final_cltv_expiry (last hop cltv) should be more than taker swap locktime (at least double like other swaps to give the taker enough time to claim the lightning payment), not less. I miscalculated this, will update this in both comments now. Taker swap locktime (for other non-lightning coins) when Maker is lightning can be just the current implemented swap locktimes in mm2, not 22 hours.

[shamardy]

This is a diagram to demonstrate swap locktimes if taker is LightningCoin as explained in https://github.com/KomodoPlatform/atomicDEX-API/issues/1045#issuecomment-1310604403 https://github.com/KomodoPlatform/atomicDEX-API/issues/1045#issuecomment-1311755066 This is another diagram to demonstrate swap locktimes if maker is LightningCoin as explained in https://github.com/KomodoPlatform/atomicDEX-API/issues/1045#issuecomment-1310604403 https://github.com/KomodoPlatform/atomicDEX-API/issues/1045#issuecomment-1312062816

[shamardy]

@artemii235 In this comment, I would like to propose a solution to how lightning taker fees can be implemented in AtomicDEX so that we can discuss this solution before proceeding with implementing this part of the lightning network integration.

Problems

• To receive a lightning fee from a taker through the lightning network, Komodo would need to run its own lightning node that the taker can pay the fee to. Which brings about the following problems:
  • Centralization: Komodo's lightning node will be a single point of failure for lightning swaps.
  • Security: Lightning nodes are hot wallets.
• Lightning payments are private in nature, there is no single source of truth like how it is for on-chain transactions. Because of this, a method of verification is needed for the maker to verify that the taker has paid the fee.

Proposed Solution

• We can avoid running a lightning node for the fees altogether and make the taker pay the fee on-chain.
  • This solves the centralization, security and verification problems.
  • But this also defeats the purpose of integrating the lightning network by increasing the fees and swap times.
  • This solution can be used as a backup to the next solution that requires running a lightning node for the fees, if the fees node is down for any reason, trading can resume using on-chain fees until the node is operational again.
• A lightning node for fees is used.
  • lightning offers are used to solve the verification problem. Lightning offers will be described in details in the next section.
  • Phantom Nodes are used to solve the single point of failure problem. The article linked provides the necessary information about phantom nodes. I still need to do more research on this topic to be sure that it completely fits the requirements.
  • Validating Lightning Signer is used to solve the hot wallet problem. Merchant policy control can be used to ensure that no funds can be sent from the fees lightning node without external signing. I still need to do more research on this topic to be sure that it completely fits the requirements.

Lightning offers

• Lightning offers are mostly implemented in core lightning and currently being implemented in rust-lightning.
• It shouldn't take long for offers to be an official lightning bolt (it's next in line actually, currently there are 11 official bolts, offers is bolt12), which means all lightning node implementations should implement it.
  • An offer by komodo's lightning node for fees will be hardcoded similar to the dex fee address public key. An offer can live forever unlike invoices which require an expiration timestamp to be set.
  • The taker should start with an invoice request to pay the fees by paying the invoice that he will receive in response to this request. This request is routed through the lightning network to reach the fees lightning node.
  • In the invoice request, the taker should set invreq_payer_note to the swap's secret hash.
  • The fees lightning node will send the taker through the lightning network a signed invoice, it will have the same invreq_payer_note set by the taker which is equal to the swap's secret hash.
  • The taker will pay the fee by paying the invoice, receiving the preimage for the invoice payment hash.
  • The taker will send the maker the signed invoice and the preimage in the TakerFee msg.
  • To verify that the taker paid the fee, the maker will verify the following:
  • The invoice is signed by the fees lightning node public key, this is the node's public key not the dex fee address public key. The node's public key should be hardcoded in mm2 code too since it doesn't change.
  • The signed invoice invreq_payer_note field is equal to the swap's secret hash.
  • The hash of the preimage is equal to the invoice payment hash.

[shamardy]

@smk762 after this PR https://github.com/KomodoPlatform/atomicDEX-API/pull/1592, new events that should be documented were added to MAKER_SUCCESS_EVENTS, MAKER_ERROR_EVENTS, TAKER_SUCCESS_EVENTS, TAKER_ERROR_EVENTS. These events are:

• MakerPaymentInstructionsReceived added to MAKER_SUCCESS_EVENTS: This event is triggered alongside TakerFeeValidated event (it's before it in order, if both are successful, both are triggered. If one of them fails TakerFeeValidateFailed is triggered instead). It indicates that maker received inside the TakerFee swap message instructions on how to send their payment and validated these instructions successfully. For protocols other than lightning the event success depends on taker fee validation success since no payment instructions are sent.
• TakerPaymentInstructionsReceived added to TAKER_SUCCESS_EVENTS: This event is triggered alongside MakerPaymentReceived, MakerPaymentWaitConfirmStarted events (it's before them in order, if all are successful, all are triggered. If one of them fails MakerPaymentValidateFailed is triggered instead). It indicates that taker received inside the MakerPayment swap message instructions on how to send their payment and validated these instructions successfully. For protocols other than lightning the event success depends on maker payment validation success since no payment instructions are sent.
• MakerPaymentRefundStarted added to MAKER_ERROR_EVENTS: This event is triggered after MakerPaymentWaitRefundStarted event. It indicates that the refund process for maker payment has started successfully and any process required before it was successful.
• TakerPaymentRefundStarted added to TAKER_ERROR_EVENTS: This event is triggered after TakerPaymentWaitRefundStarted event. It indicates that the refund process for taker payment has started successfully and any process required before it was successful.
• MakerPaymentRefundFinished added to MAKER_ERROR_EVENTS: This event is triggered after MakerPaymentRefunded event. It indicates that any process needed after maker payment was refunded was successful.
• TakerPaymentRefundFinished added to TAKER_ERROR_EVENTS: This event is triggered after TakerPaymentRefunded event. It indicates that any process needed after taker payment was refunded was successful.

Full list of success and error events after this change

Maker success events

"success_events": [
    "Started",
    "Negotiated",
    "MakerPaymentInstructionsReceived",
    "TakerFeeValidated",
    "MakerPaymentSent",
    "TakerPaymentReceived",
    "TakerPaymentWaitConfirmStarted",
    "TakerPaymentValidatedAndConfirmed",
    "TakerPaymentSpent",
    "TakerPaymentSpendConfirmStarted",
    "TakerPaymentSpendConfirmed",
    "Finished",
]

Taker success events

"success_events": [
    "Started",
    "Negotiated",
    "TakerFeeSent",
    "TakerPaymentInstructionsReceived",
    "MakerPaymentReceived",
    "MakerPaymentWaitConfirmStarted",
    "MakerPaymentValidatedAndConfirmed",
    "TakerPaymentSent",
    "TakerPaymentSpent",
    "MakerPaymentSpent",
    "Finished",
]

Maker error events

"error_events": [
    "StartFailed",
    "NegotiateFailed",
    "TakerFeeValidateFailed",
    "MakerPaymentTransactionFailed",
    "MakerPaymentDataSendFailed",
    "MakerPaymentWaitConfirmFailed",
    "TakerPaymentValidateFailed",
    "TakerPaymentWaitConfirmFailed",
    "TakerPaymentSpendFailed",
    "TakerPaymentSpendConfirmFailed",
    "MakerPaymentWaitRefundStarted",
    "MakerPaymentRefundStarted",
    "MakerPaymentRefunded",
    "MakerPaymentRefundFailed",
    "MakerPaymentRefundFinished",
]

Taker error events

"error_events": [
    "StartFailed",
    "NegotiateFailed",
    "TakerFeeSendFailed",
    "MakerPaymentValidateFailed",
    "MakerPaymentWaitConfirmFailed",
    "TakerPaymentTransactionFailed",
    "TakerPaymentWaitConfirmFailed",
    "TakerPaymentDataSendFailed",
    "TakerPaymentWaitForSpendFailed",
    "MakerPaymentSpendFailed",
    "TakerPaymentWaitRefundStarted",
    "TakerPaymentRefundStarted",
    "TakerPaymentRefunded",
    "TakerPaymentRefundFailed",
    "TakerPaymentRefundFinished",
]