Relai: Payment with no status and wrong amount sent

[kavofa]

• node_id=02281d4861674eb5020b9d27f095a9c92c613b06e07509d3ff76a1a2c195de7c28

I used Relai to pay for a lightning invoice of 149000000 SATS. My balance was reduced but the recipient never received any SATS. Upon connecting to my node via gl-client, I noticed in the "paylist" logs that the payment has no status and the amount sent doesn't match the amount of the invoice. The Invoice has already expired by now.

pays {
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  destination: "\002\213\020\276\220\335\373N\361X\302R\026>O4\014\212\t;\356\342\004'\323\232\t\206\253\254X\245^"
  created_at: 1716286803
  bolt11: "lnbc1490u1pnycug4pp5hkdcxk9w7crfwvq6qhf5g9ylh5l2qvt2gs0z9p0uw5aflg5z035qdqqcqzzsxqyz5tlsp539r7dk8jrr9k0kcwwudq9ytfl27qa4s4py434xf2q4zvnl36mafs9qyyssq5ewtx9q944kry40zl0hmxnc68rjla2nz98zkyj6ju254u7wx37788qu7yz8swl535tcwxay35pcvy0emu5q0hpzfq6fx023dnm56f8sp5qamy9"
  description: ""
  amount_msat {
    msat: 126267684
  }
  amount_sent_msat {
    msat: 126783461
  }
  number_of_parts: 10
}

So two issues here, firstly the amount sent is wrong. Secondly the payment has no status but the balance (spendable_msat from list_peers) was reduced. Is there a way to recover those funds?

[cdecker]

There are a number of cases that can lead to this situation, but I think the most likely is the following:

• You tried to pay an invoice for 149000000 msats (you wrote sats, but that'd be >1BTC in value, which I doubt)
• The payment was attempted, and had to be split into multiple parts (see the number_of_parts entry there)
• The payment ultimately failed, with only a partial amount reaching the destination
• The destination held the HTLCs in the hopes of completing the payment, but ultimately it had to return them
• The HTLCs are not yet all returned, causing the balance discrepancy.

You see, when you send a payment, we have to set aside the payment amount from your balance, and allocate them to HTLCs, as that's how multi-hop payments are performed. The balance will only return to the starting situation once all HTLCs are settled. So the spendable_msat key in listpeerchannels is just that, the amount you can spend immediately without having to wait for things like HTLCs to settle. It's also worth pointing out that counting HTLCs to your balance is ultimately wrong, as the HTLC is in a state that cannot be decided (it could either be settled or timeout, which determines who owns the funds).

Please allow some time for the funds to be fully returned, at which point you will see that the payment failed. Given you have the gl-client installed, you could look into list_peer_channels to see which and how many HTLCs are pending, along with the timeouts that delineate their validity.

[kavofa]

Thanks for your response. I checked with list_peers and figured that all HTLCs are expired by now. Current Block as of writing is 844739.

htlcs {
  direction: OUT
  id: 49
  amount_msat {
    msat: 75249437
  }
  expiry: 844723
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 64
  amount_msat {
    msat: 1895106
  }
  expiry: 844723
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 65
  amount_msat {
    msat: 2029320
  }
  expiry: 844723
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 55
  amount_msat {
    msat: 1873692
  }
  expiry: 844723
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 57
  amount_msat {
    msat: 9284735
  }
  expiry: 844619
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentRemoveRevocation
}
htlcs {
  direction: OUT
  id: 58
  amount_msat {
    msat: 4101783
  }
  expiry: 844619
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 60
  amount_msat {
    msat: 4823519
  }
  expiry: 844619
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 59
  amount_msat {
    msat: 4058217
  }
  expiry: 844619
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 62
  amount_msat {
    msat: 2290245
  }
  expiry: 844619
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: RcvdRemoveHtlc
}
htlcs {
  direction: OUT
  id: 66
  amount_msat {
    msat: 1265341
  }
  expiry: 844619
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: RcvdRemoveHtlc
}
htlcs {
  direction: OUT
  id: 63
  amount_msat {
    msat: 21258254
  }
  expiry: 844723
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}
htlcs {
  direction: OUT
  id: 61
  amount_msat {
    msat: 2209398
  }
  expiry: 844723
  payment_hash: "\275\233\203X\256\366\006\2270\032\005\323D\024\237\275>\2401jD\036"\205\374u:\237\242\202|h"
  state: SentAddAckRevocation
}

In the list_peers response I also find some weird status messages.

status: "ONCHAIN:Tracking our own unilateral close"
status: "ONCHAIN:16 outputs unresolved: in 4294967176 blocks will spend OUR_HTLC (00c4619fbcc059107504d53d1138dab98d7ab710477b9da37c7bd6f764cc8205:2) using OUR_HTLC_TIMEOUT_TX"

So in summary the HTLC with the latest expiry has the expiry of block 844723, this has already happened ~16 blocks ago. Shouldn't the funds be returned by now, or am I misunderstanding something here.

Cheers

[cdecker]

I see, yes, that is an issue with the fees that are attached to those sweep transactions. Given the high-fee environment, we only slowly raise the feerate in order not to waste a user's funds on sweeping funds, but paying high fees to do so. It just takes a while to sweep.

16 blocks is still rather recent, let's see if the sweeps eventually go through. If not you might have a case of https://github.com/Blockstream/greenlight/issues/439, which we are actively debugging (restarts messing up the onchaind state machine).

[kavofa]

@cdecker to this day (2 months) the sats are still stuck. Apparently v24.02gl1 doesn't fix this. The node still gives the following status:

status: "ONCHAIN:Tracking our own unilateral close"
status: "ONCHAIN:16 outputs unresolved: in 4294967119 blocks will spend OUR_HTLC (00c4619fbcc059107504d53d1138dab98d7ab710477b9da37c7bd6f764cc8205:2) using OUR_HTLC_TIMEOUT_TX"

It seems that the calculation of the number of blocks until the sweep transaction will be published is miscalculated (4294967119 blocks).

Do you have an update on this issue or an idea on how to get my sats back out of this state?

[cdecker]

Yes, the numeric overflow is just a representational issue. I'll check the status of this node asap and see what's going on here 🤔

[kavofa]

@cdecker It's been one month since you wanted to "check the status of this node asap". I would really appreciate you actively helping me out to recover my sats. It should be around $70 (current price) that are in this weird stage.

[cdecker]

Apologies, my bad for dropping the ball on this one. I just checked the node right now, and looking at the 3 user sessions today, I can see that it is indeed attempting to sweep the funds from the HLTCs. It is lacking the funds to do so however:

0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e835-chan#1: We want to bump HTLC fee, but no funds!

This means that we're attempting to RBF the sweeps, but do not have on-chain funds to do so. This is either because the node does not have an on-chain output it could use, and the emergency funds have been set to 546sat (min-emergency-msat=msat config option) in response to Breez asking for the ability to fully allocate funds to channels, and make them fully drainable.

There is little we can do from our side, we'll have to wait for these sweeps to go through, or we'll have to add sufficient on-chain funds to pay for the sweeps.

[kavofa]

If it is possible, I would like to add sufficient on-chain funds to pay for the sweeps. But I am wondering how many sats will be swept and if it is even economical to do so. To my knowledge there should be around 0.00149 BTC (which are mostly locked in this weird state where there are not reflected in my lightning balance nor marked as sent) that should be swept. Can you give me the exact amount of BTC I would receive if I add sufficient on-chain funds to pay for the sweeps?

Next question: How many BTC do I have to transfer in order for the sweeps to go through and to which address?

Cheers