lncli sendpayment/payinvoice hangs with no response when capacity is insufficient

sangaman commented 6 years ago

I've observed this behavior on multiple nodes so it isn't isolated. Calling lncli payinvoice or lncli sendpayment --pay_req when channel capacity is insufficient causes the commands to hang.

From the receiving node:

$ lncli addinvoice --value 100000
{
        "r_hash": "2a5630d4f1fdd01b45ade6194930a43d9ae0c4658993e998d970b8dc43066468",
        "pay_req": "lntb1m1pdydtu8pp59ftrp483lhgpk3dducv5jv9y8kdwp3r93xf7nxxewzudcscxv35qdqqcqzys9q062arzhw759uv6twpndw0dscrc3u7m8vxhdzf62gcacv3kdq2kqyha9tcleghxkt20yszlvp4efw44fv4fr6r6yj98d79pg7sjfqgqwjefzx"
}

Then trying to pay the invoice on the sending node using one of the commands above then causes the console to hang. I grabbed the following from the lnd output:

2017-12-29 21:12:47.722 [ERR] CRTR: Attempt to send payment 2a5630d4f1fdd01b45ade6194930a43d9ae0c4658993e998d970b8dc43066468 failed: TemporaryChannelFailure: insufficient capacity in available outgoing links: need 100002200 mSAT, max available is 52796000 mSAT
2017-12-29 21:12:49.450 [ERR] CRTR: Attempt to send payment 2a5630d4f1fdd01b45ade6194930a43d9ae0c4658993e998d970b8dc43066468 failed: TemporaryChannelFailure

Roasbeef commented 6 years ago

If you look at the logs for Mission Control, do you see any lingering activity? Atm, we don't have an upper bound on our number of attempts, which itself is a bug. If your machine is slow, or is doing a ton of other activities, then with the way our timeouts are considered, it may prune an edge, attempt another payment, then by the time the payment fails, that edge is unpruned again.

sangaman commented 6 years ago

I don't see Mission Control in the logs but I do see multiple lines of Attempt to send payment over ~2 minutes or so, I think that's about how long I waited before killing the command . One machine I observed this on was slow, the other was using modern hardware. If I see this again I'll let it run for a while and keep a close eye on the logs.

MrHash commented 6 years ago

I have been getting sending/routing hangs as well. I hadn't correlated it to a funding issue, but for example i sent small payments to htlc.me which worked then shortly after a larger payment which hangs. Subsequent attempts with lower amounts are also hanging until a service restart. queryroutes often hangs as well even with small amounts. I'm running this on a relatively slow machine also.

MrHash commented 6 years ago

Update: queryroutes did work with small amounts but took half an hour and returned 100s of results!

halseth commented 6 years ago

queryroutes and sendpayment are doing path finding a bit differently, #379 should fix a lot of the slowness seen in queryroutes.

sendpayment is using MissionControl, which should show up in the logs with prefix CRTR: if you are using debug logs :)

sangaman commented 6 years ago

@halseth good to know. I do see all the Attempt to send payment messages are prefaced with CRTR:. I haven't encountered this issue again yet but I'll update if I do.