After opening the channel, the node balance is lost

Background

After closing some channels, we attempted to establish multiple channels with three nodes. After the establishment was completed, there was a discrepancy between the balance query on the node and the balance on the chain. The following is the specific situation

Actual behaviour

At the beginning, we closed the opening of six channels with LNBiG Hub

./lncli closechannel --funding_txid=3bdbaf268a743727b23ec4df241d3a5b70601605621e8c7f4ec31faf4e59c2df --output_index=1 { "closing_txid": "a1710f294c65a628d34a00f3bd42675e038770bdb2422da7396235a2b71cbb0f" } [root@eb7e346f7884 lnd]# ./lncli closechannel --funding_txid=d2935b0d2958beb4d876a2c5f39faf24bcd74f1bdb6584dcaf6638df234755ca --output_index=1 { "closing_txid": "8397d5c2821a32085b503d93788f02a48e91aa63cdac3d182e80884dc99e66ae" } [root@eb7e346f7884 lnd]# ./lncli closechannel --funding_txid=43b4aabec2749fb54a6534985eec30c1cd33e7bb6c4f10f04bedfafad05fde10 --output_index=1 { "closing_txid": "62e6472be147d5f8ba1aae6bccdb7aeef7a06d4822b30d8d4584cbb5062db207" } [root@eb7e346f7884 lnd]# ./lncli closechannel --funding_txid=231e7290c9fbd7248207c2984564a34c4bcc4f91fa5fbfd5a1d5f51fd5d2ad4e --output_index=1 { "closing_txid": "80f3a78317f97a894e3d0cfd6089adbb62ce6528213d58c7ccf60ae4192bc99b" } [root@eb7e346f7884 lnd]# ./lncli closechannel --funding_txid=7ad753f3321b3105be779bf4ceb50fe405b0295dd392e312bdf71eb93f54302f --output_index=1 { "closing_txid": "754886ad752ef9e157e28a71282cc210e6a9c393b0696c6e83cf8d5ca66948cd" }

Then we tried to establish a channel. But when executing this command, we still get no result, so we forcibly interrupted it, and then restarted the node

./lncli openchannel --local_amt=500000000 --node_key=034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5

After restarting the node, we tried to execute the above command again. it still got no result, so we forcibly interrupted it again Then we tried to establish channels with the other two nodes. The specific implementation results are as follows

./lncli openchannel --private --local_amt=500000000 --node_key=03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d { "funding_txid": "c588e86f2d29b3c4e0951e321b53716ddc84cd77c611b05e74496e6cc4f7ff1a" } [root@eb7e346f7884 lnd]# ./lncli openchannel --private --local_amt=500000000 --node_key=03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d ./lncli openchannel --private --local_amt=500000000 --node_key=03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d ./lncli openchannel --private --local_amt=500000000 --node_key=03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d ./lncli openchannel --private --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 ./lncli openchannel --private --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 ./lncli openchannel --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 [lncli] rpc error: code = Unknown desc = received funding error from 03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d: chan_id=e74ab98a556eba61bc987c9061714727bc83f4bf666e9789155a43965ef1faef, err=Number of pending channels exceed maximum [root@eb7e346f7884 lnd]# ./lncli openchannel --private --local_amt=500000000 --node_key=03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d ./lncli openchannel --private --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3[lncli] rpc error: code = Unknown desc = received funding error from 03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d: chan_id=d7aae0430698288c23c80c8948da0daf79d76af6c36668b755e2db9c9324be68, err=Number of pending channels exceed maximum [root@eb7e346f7884 lnd]# ./lncli openchannel --private --local_amt=500000000 --node_key=03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d [lncli] rpc error: code = Unknown desc = received funding error from 03cabeaa1223b83d56954e944e83493d20c6b2c9b45e47b62a899ab9d7d31cb56d: chan_id=1ae2aa5d4670628baaba29371e7df5610766557acfb0b10d3d8fe31782243f10, err=Number of pending channels exceed maximum [root@eb7e346f7884 lnd]# ./lncli openchannel --private --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 { "funding_txid": "8504e0f8e59dfb21115d020a842e7f502d08e4948682a7f7091569667e3d768b" } [root@eb7e346f7884 lnd]# ./lncli openchannel --private --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 { "funding_txid": "ce76e3bb1dab2994a5c2f77ccbba22e56f9ba98683dbad0d3ab751024e3c1275" } [root@eb7e346f7884 lnd]# ./lncli openchannel --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 { "funding_txid": "538f2cae82461d2be43a646b3fdcde37d09955fa7c1d387ae416d29e9f1739bd" } [root@eb7e346f7884 lnd]# ./lncli openchannel --private --local_amt=500000000 --node_key=030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3 [lncli] rpc error: code = Unknown desc = received funding error from 030a58b8653d32b99200a2334cfe913e51dc7d155aa0116c176657a4f1722677a3: chan_id=bff3a46baa209f48fd207b9580134315dd7bd393e39ee1207a1fa827929af656, err=Number of pending channels exceed maximum

Sometimes the command can work, but sometimes it has always been blocked, so we force the interrupt of this command every time According to the above execution results, It take us 20 btc to open channel but we found our wallet amount has decreased by 40 btc. We don't know where 20 BTC left off, This is wallet balance now, just 17.3 btc, but before we execute that command ,we have 51 btc. And channel balance is right, now we have 174 btc, before execute amount is 152 btc.

We checked the balance of all our addresses on the chain and on the node. These addresses are different.

address: bc1qdjr4hu53fkknq68m0vd3823gxhzrefhhkykedz balanceOnNode: 0, balanceOnChain: 2049667 address: bc1pujzttnxrrvdpw9kvpf5v3fwnum5gr65qnf5sem4stnvm0pwy05kqlgvmrr balance on node: 0, balance on chain: 197944196 address: bc1p6hjlgjey376hk2ue9hjlmt6x0q0spte7z58gjm8ecj3xmm8updfqctwvem balance on node: 0, balance on chain: 977800 address: bc1pnhaulw39786q5mn2yx7d82epjdaue5n2xgqqs4dn6drz67frgqmshm4f0r balance on node: 0, balance on chain: 494897660 address: bc1pctzseenm2pmrlhv6wud9807f5smq5g92ecqxxyczxmqegltc97kqqhzrk7 balance on node: 0, balance on chain: 7917819 address: bc1p2lhmy7qxtmzuzrjf6r2p5z9awtzn4y842u2clk8l7jkukw7tgussqgdaq6 balance on node: 0, balance on chain: 492734454 address: bc1pp759mhsd64ecnxjuennlu3m4e5gmxpukfpwrgxt7ujnp5guspgwsvwuz39 balance on node: 0, balance on chain: 44535128 address: bc1pgymnykx5rwf7rkger6qvac4f563zq7tad4ha3009r9ytwf9xwf5syj0a4e balance on node: 0, balance on chain: 4945808 address: bc1p24wcv388604frgnwrlnm7z5ughp5kqnfxuey6n0gglkuzfs4d9qq0rmuke balance on node: 0, balance on chain: 494868539 address: bc1pkrsna3504jf8t3wvescd5stzymcaepyh47x5s60a3drjg9njhmystrw4lk balance on node: 0, balance on chain: 494385275

address: bc1pkarchx83dyuzncus48w7mghv03p6y9rgfmppkdj0szhnlznq7fmq5ecqru balance on node: 5931994, balanceOnChain: 0 address: bc1p5jsu6j9jq4zyfxeccal5fvuvxlm2ak6w2s6l638qmxf83nrr2fcseyh3nr balance on node: 2047257, balance on chain: 0 address: bc1p3ltxu26p0jshtlftvq4e4x7c5facv4clhumt22xv3agnlha94eyqgdvtf6 balance on node: 4943463, balance on chain: 0 address: bc1pf85p270fm4d7f0lxv9tjx5qz604g4arx3zwzln9xs3w49ttqayusqmyqej balance on node: 44509285, balance on chain: 0

Please help us find out what the problem is, and help us find the lost 20 btc or fix the balance on node, And the reason for getting no result also needs to be found Thank you so much.

So your funds are definitely not lost but we need to make your lnd wallet aware of them again.

Generally it's a bad idea to interrupt important lnd rpc calls (for example openchannel and trying it again without finding the root cause of the problem why this cmd is somehow stuck).

Could you try to broadcast the rawtransaction via the lncli wallet publishtx for all the transactions you are missing the funds. For example:

address: bc1pnhaulw39786q5mn2yx7d82epjdaue5n2xgqqs4dn6drz67frgqmshm4f0r balance on node: 0, balance on chain: 494897660

Do:

lncli wallet publishtx 020000000001021fae4d1ef4d3e6177ea97f361054aac0bc491843939dd3142f0eb1299bc1a817000000000000000000dc43d7336aaca9c4013107a2ad2fee97a8b377061f8055eb06b3dfacaa2238eb01000000000000000001fc897f1d000000002251209dfbcfba25f1f40a6e6a21bcd3ab21937bccd26a32000855b3d3462d79234037024830450221009a1289b846bfee5cfff1bbf9a1a1a19310fbe873cc635d9154aa2135e291d69402201f18214fad85de6a4ca3ef26f2f1c5dafecb994945fe536f5761e8b9eb35be170128210276c4d3567a91c230c96c0abc26a95336bbd09efb2cb69ffec625ac2687c30897ac736460b2680140afa5ec980518ab60d8fccd73fd009a8ed04487b288feeffec6ac01c1db82045dc38f725e2266a2d30fef6f0ab78a860d42925bcc8c804a9f6050ec975331482a70820c00

See whether the following utxo shows up in the lncli listunspent | grep a9cf392047a5b946f9e1b5a835815157860ddb4b663c49044ecaf4fb5614d6cd:0

In addition it would be very helpful if you could provide your logs via lnd slack or another private way why lnd was struggling opening the channel.

Can you fill out the rest of the issue template? eg: which lnd version are you running?

Do you see any pending open channels in lncli pendingchannels?

I'm not sure this is related, but wanted to share my experience with something similar: Running v0.17.2-beta, during my last few attempts to open channels, in every case it seems that LND ended up mis-counting my onchain balance.

Yesterday, I tried a lncli batchopenchannel with about 6 or 8 nodes... after waiting a bit, I saw that lncli pendingchannels didn't show any pending channels, and the batchopenchannel command was just "hanging" (not returning) in the console and I force-quit it after an hour or so. At the beginning of the opening, I did seem messages in the LND logs like:

lnd-lnd-1  | 2023-12-15 23:19:48.281 [ERR] FNDG: received funding error from [redacted]: chan_id=[redacted], err=Number of pending channels exceed maximum

My understanding is that one channel failure like this will stop the entire BatchOpenChannel process (as is documented here: https://lightning.engineering/api-docs/api/lnd/lightning/batch-open-channel/index.html ). So after an hour I assumed that the BatchOpen failed.

I then wanted to check-in on my onchain balance, so I ran

:/$ lncli walletbalance
{
    "total_balance": "0",
    "confirmed_balance": "0",
    "unconfirmed_balance": "0",
    "locked_balance": "0",
    "reserved_balance_anchor_chan": "100000",
    "account_balance": {
        "default": {
            "confirmed_balance": "0",
            "unconfirmed_balance": "0"
        }
    }
}

Which was somewhat scary. So then I restarted LND, and everything reverted and my on-chain balance came back.

I then decided that there were too many unknowns with the BatchOpenChannel command, so I opened a few channels one-by-one. The channel openings succeeded, I found them as pending, but then I looked again at lncli walletbalance , and once again this showed 0 total balance.

And then again, when I restarted LND, the balances came back.

I have a similar issue with openchannel. It is hanging, the channel doesn't show up in lncli pendingchannels, the balance is 0 in walletbalance.

The whole story, including some details, which may be important.

Originally I had this.

$ lncli walletbalance                                                                                                                                                         
{                                                                                                                                                                                             
    "total_balance":  "1381855",              
    "confirmed_balance":  "735444",
    "unconfirmed_balance":  "646411",         
    "locked_balance":  "0",                 
    "reserved_balance_anchor_chan":  "100000",
    "account_balance":  {             
        "default":  {                  
            "confirmed_balance":  "735444",
            "unconfirmed_balance":  "646411"
        }
    }                                
}

unconfirmed_balance is caused by an old anchor sweep tx which spends anchor output and one of wallet's UTXOs and produces another wallet utxo. By the way, the transaction itself reduced my wallet balance and was economically meaningless. Its fee rate was quite high, but it didn't confirm in time and went below purging limit according to mempool.space.

Later I restarted the node and saw, that the balance is fully confirmed:

$ lncli walletbalance
{
    "total_balance":  "1384283",
    "confirmed_balance":  "1384283",
    "unconfirmed_balance":  "0",
    "locked_balance":  "0",
    "reserved_balance_anchor_chan":  "100000",
    "account_balance":  {
        "default":  {
            "confirmed_balance":  "1384283",
            "unconfirmed_balance":  "0"
        }
    }
}

Probably it forgot about the anchor sweep transaction, which is a good thing.

I decided to open a channel and openchannel got stuck. It is still stuck. Here is the command and relevant parts of the log.

$ lncli openchannel --fee_rate_ppm **** --sat_per_vbyte 70 --fundmax ***

2024-01-02 04:37:11.802 [INF] FNDG: Initiating fundingRequest(local_amt=0 BTC (subtract_fees=false), push_amt=0 mSAT, chain_hash=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce
26f, peer=***, min_confs=1)
2024-01-02 04:37:11.933 [INF] CHFD: Performing funding tx coin selection using 17500 sat/kw as fee rate
2024-01-02 04:37:12.103 [WRN] CHFD: Unable to find funding output for shim intent: unable to create witness script, no funding keys
2024-01-02 04:37:12.387 [INF] FNDG: Target commit tx sat/kw for pendingID(47f419bcab6ce0da34dd453fb9afa0fc48e15da19c574815acfcda8557f772e2): 2500
2024-01-02 04:37:12.387 [INF] FNDG: Dust limit for pendingID(47f419bcab6ce0da34dd453fb9afa0fc48e15da19c574815acfcda8557f772e2): 0.00000354 BTC
2024-01-02 04:37:12.387 [INF] FNDG: Starting funding workflow with ***:9735 for pending_id(47f419bcab6ce0da34dd453fb9afa0fc48e15da19c574815acfcda8557f772e2), committype=anchors-zero-fee-second-level
2024-01-02 04:47:39.600 [WRN] FNDG: reservation timed out waiting for peer (peer_id:***, chan_id:47f419bcab6ce0da34dd453fb9afa0fc48e15da19c574815acfcda8557f772e2)
2024-01-02 04:47:42.972 [INF] CRTR: Examining channel graph for zombie channels
2024-01-02 04:47:43.204 [INF] CRTR: Pruning 1 zombie channels
2024-01-02 04:47:43.255 [INF] CHDB: Pruned unconnected node yyy [unrelated node] from channel graph
2024-01-02 04:47:43.256 [INF] CHDB: Pruned 1 unconnected nodes from the channel graph
2024-01-02 04:47:46.441 [INF] CRTR: Processed channels=0 updates=91 nodes=27 in last 59.999399041s

Channel opening is still stuck, wallet balance is 0.

$ lncli walletbalance
{
    "total_balance":  "0",
    "confirmed_balance":  "0",
    "unconfirmed_balance":  "0",
    "locked_balance":  "0",
    "reserved_balance_anchor_chan":  "100000",
    "account_balance":  {
        "default":  {
            "confirmed_balance":  "0",
            "unconfirmed_balance":  "0"
        }
    }
}

Maybe if I restart the node, it will recover the balance, but I better wait.

Update. lncli openchannel exited:

[lncli] rpc error: code = Unknown desc = peer disconnected

and the wallet balance recovered!

The openchannel and "funds disappearing" issue might be related to #8362.

I think we can close this as a duplicate of: https://github.com/lightningnetwork/lnd/issues/7868

It might be, but I would say this issue here ties more to the fix here: https://github.com/lightningnetwork/lnd/pull/8367

Because I think the main problem here is that the peer does disrupt the communication whereas in https://github.com/lightningnetwork/lnd/issues/7868 the funding flow works fine but it only fails because the funding tx cannot be broadcasted on our end.

So I would keep those two issues distinct but maybe @hieblmi can verify ?

@ziggie1984 shouldn't the coins be cleaned up either way due to the defer?

https://github.com/lightningnetwork/lnd/blob/6d3f359091168418c3c143d0618ca488aebd9ab9/funding/batch.go#L295-L299

I think we can help confirm if any of the other commenters in this list saw the coins in lncli wallet listleases.

lightningnetwork / lnd

After opening the channel, the node balance is lost #8251

Background

Actual behaviour