Still cannot force close inactive channels

kiaraho commented 5 years ago

Background

Follow up to issue #2391. Now running 0.5.1. Most of the channels have closed and I can see the funds on-chain when I run lncli walletbalance. However, there are still some funds off-chain in one pending_force_closing_channels and two waiting_close_channels. Those channels have been in limbo for several days now.

Your environment

lncli version 0.5.1-beta commit= Linux hostname 4.15.0-43-generic #46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Bitcoin Core Daemon version v0.17.0.0-ge1ed37edaedc85b8c3468bd9a726046344036243

Steps to reproduce

Ran lncli closeallchannels --force to close inactive channels. Got the following message:

{ "remote_pub_key": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b", "channel_point": "825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1", "closing_txid": "cb118298c5526ca09576dd54f8be48b06994fafb790738879069ef50d8e156e6", "error": "" } { "remote_pub_key": "032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada", "channel_point": "fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1", "closing_txid": "e2e7c83691ad21deb37a42bcbb22886225a88c0b67a57be94e0060992d8e0b3c", "error": "" }

Expected behaviour

Inactive channels should close.

Actual behaviour

Three channels still stuck in limbo. The two closing_txid above do not seem to have been broadcast. Running lncli closeallchannels --force again returns a "[lncli] no open channels to close" message. Output from lncli pendingchannels command:

{ "total_limbo_balance": "11783124", "pending_open_channels": [ ], "pending_closing_channels": [ ], "pending_force_closing_channels": [ { "channel": { "remote_node_pub": "029afc726a18abc8dc75ef6c9ed34354c275261086597d98067dd972bd72965943", "channel_point": "cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0", "capacity": "2062186", "local_balance": "2054765", "remote_balance": "0" }, "closing_txid": "297735f112539d8c1f7325863468b8a67c20f3584be06bb4f9634c10c458df36", "limbo_balance": "0", "maturity_height": 0, "blocks_til_maturity": 0, "recovered_balance": "0", "pending_htlcs": [ ] } ], "waiting_close_channels": [ { "channel": { "remote_node_pub": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b", "channel_point": "825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1", "capacity": "7119858", "local_balance": "6947834", "remote_balance": "0" }, "limbo_balance": "6947834" }, { "channel": { "remote_node_pub": "032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada", "channel_point": "fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1", "capacity": "4842711", "local_balance": "4835290", "remote_balance": "0" }, "limbo_balance": "4835290" } ] }

Roasbeef commented 5 years ago

That first channel has been fully swept on chain. Do you have start up logs for that? Similarly, do you have logs for the latter two?

kiaraho commented 5 years ago

By start up logs, are you referring to the entries in lnd.log after running lncli unlock? If so, which log entries in particular?

Roasbeef commented 5 years ago

In lnd.log grep for either of those channel point or txid's.

kiaraho commented 5 years ago

Here are the log entries I have immediately after running lncli pendingchannels.

grep cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666 ~/.lnd/logs/bitcoin/mainnet/lnd.log 2019-01-15 09:44:04.884 [INF] UTXN: NurseryReport: building nursery report for channel cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0

grep 825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e ~/.lnd/logs/bitcoin/mainnet/lnd.log 2019-01-15 09:43:37.082 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 10m0s.

grep fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4 ~/.lnd/logs/bitcoin/mainnet/lnd.log 2019-01-15 09:43:37.117 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 10m0s.

halseth commented 5 years ago

Aah... looks like you restored from an outdated backup?

If you're unlucky, you force-closed the channels (which is a breach when your backup is outdated), and the remote took the funds. If you're lucky, the remote force closed and has data-loss protection enabled. If that's the case, you can try to connect to the peer (02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b and 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada) and see what happens. Please post debug logs!

kiaraho commented 5 years ago

I was already connected to both peers.

///

lncli listpeers | grep -A 7 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b "pub_key": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b", "address": "167.99.50.31:9735", "bytes_sent": "6728289", "bytes_recv": "10243259", "sat_sent": "0", "sat_recv": "0", "inbound": false, "ping_time": "2025706"

lncli listpeers | grep -A 7 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada "pub_key": "032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada", "address": "5.189.141.242:9735", "bytes_sent": "6171020", "bytes_recv": "10269721", "sat_sent": "0", "sat_recv": "0", "inbound": false, "ping_time": "234567"

///

Should I disconnect then reconnect? If so, what debuglevel should I set prior to doing so and what entries should I grep for?

halseth commented 5 years ago

Set debuglevel debug and reconnect. Interesting logs you can grep for are the channel points (825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e and fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4).

kiaraho commented 5 years ago

So now I seem to be stuck in an LND roach motel, as lncli disconnect won't let me check out! Output from lncli pendingchannels is the same as before, with one pending_force_closing_channels and two waiting_close_channels. Limbo balance is unchanged.

///

lncli disconnect --node_key 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b [lncli] rpc error: code = Unknown desc = cannot disconnect from peer(02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b), all active channels with the peer need to be closed first

lncli disconnect --node_key 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada [lncli] rpc error: code = Unknown desc = cannot disconnect from peer(032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada), all active channels with the peer need to be closed first

lncli listchannels { "channels": [ ] }

grep 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b ~/.lnd/logs/bitcoin/mainnet/lnd.log | tail -n 5 2019-01-15 19:15:13.309 [INF] CRTR: New channel discovered! Link connects 02282cfc83d4e4ac088a654184be62d7de0be6298e83af39431f9598c8c5631c09 and 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b with ChannelPoint(c8fdd4f1e9e57f68ada64b8d00a0c4bb3b8642a4d61161efafdfd010ec0865a2:0): chan_id=612600600140906496, capacity=0.00317161 BTC 2019-01-15 19:15:13.315 [INF] CRTR: New channel discovered! Link connects 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b and 02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0 with ChannelPoint(9b7d587ef135daabbc1039e401b86ff9a063252427c9c5bd035fb0589635e84a:1): chan_id=612440071376011265, capacity=0.003 BTC 2019-01-15 19:15:13.318 [INF] CRTR: New channel discovered! Link connects 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b and 03cfb64a81ad9b94e6f3fa2e34218c9242606890bbfb65a0bb57d603158e6c590c with ChannelPoint(cc88267b03a05a0d6059550f6ad8aaec1c196b9a7f72dc68fb7012930ebc47ce:0): chan_id=612615993258999808, capacity=0.00353752 BTC 2019-01-15 19:18:15.906 [DBG] RPCS: [disconnectpeer] from peer(02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b) 2019-01-15 19:30:50.100 [DBG] RPCS: [disconnectpeer] from peer(02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b)

grep 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada ~/.lnd/logs/bitcoin/mainnet/lnd.log | tail -n 5 2019-01-15 19:01:03.497 [INF] CRTR: New channel discovered! Link connects 024b8019795ec753fcdb1eb21ecd319bfab6a8b05e017f357dc5f052a181fb8c91 and 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada with ChannelPoint(18871533aec34e712053d661f7266c1f13f35f0e4b3bc8eda66cfe62ee747011:0): chan_id=582232088920064000, capacity=0.0002 BTC 2019-01-15 19:28:23.864 [DBG] RPCS: [disconnectpeer] from peer(032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada) 2019-01-15 19:30:51.862 [DBG] RPCS: [disconnectpeer] from peer(032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada) 2019-01-15 19:31:03.732 [INF] CRTR: New channel discovered! Link connects 02f3069a342ae2883a6f29e275f06f28a56a6ea2e2d96f5888a3266444dcf542b6 and 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada with ChannelPoint(2eb868dc242ce74f072abe9fab06c3a1823ce99b5b4ef156a3abf0dfc5109414:1): chan_id=592900650213703681, capacity=0.0002 BTC 2019-01-15 19:31:03.758 [INF] CRTR: New channel discovered! Link connects 02f3069a342ae2883a6f29e275f06f28a56a6ea2e2d96f5888a3266444dcf542b6 and 032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada with ChannelPoint(fcbf2a617b8165fb9d9c2d885bb0db24286f884acde2171bff05ec690f8e83e5:1): chan_id=605291046758973441, capacity=0.0002 BTC

grep 825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e ~/.lnd/logs/bitcoin/mainnet/lnd.log | tail -n 1 2019-01-15 19:33:37.095 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 10m0s.

grep fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4 ~/.lnd/logs/bitcoin/mainnet/lnd.log | tail -n 1 2019-01-15 19:33:37.129 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 10m0s.

halseth commented 5 years ago

Try restarting lnd, and post the grepped logs after restart (up until the "no commit point").

kiaraho commented 5 years ago

Stopped lnd, moved the existing lnd.log to a backup location, then restarted lnd so it creates a fresh lnd.log. Looks like older entries in the current lnd.log got overwritten, as I do not see what is usually the first line, Active chain: Bitcoin (network=mainnet). I've uploaded the contents of the current lnd.log up until the "no commit point", though I'm guessing lots of crucial debug messages got overwritten.

lnd-log_before_no-commit-point-found.txt

halseth commented 5 years ago

Yeah, basically what I want to look for is after you connect to the peers, the channels will attempt to sync up. Here I expect the peer to not respond since they have closed the channel. If they respond with the commitment point then we might be able to recover the funds.

You can try increasing the saved logs size (take a look at lnd -h) so they don't get overwritten!

kiaraho commented 5 years ago

Used the --maxlogfilesize=100 option to increase the log size so it won't get overwritten so quickly. Here is the grepped output for the three channel points (last 30 lines).

grep cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666 lnd.log | tail -n 30

2019-01-18 20:13:19.957 [DBG] CNCT: Starting ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0), htlc_set=(contractcourt.htlcSet) {
2019-01-18 20:13:19.957 [INF] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): starting state=StateWaitingFullResolution
2019-01-18 20:13:19.960 [INF] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): still awaiting contract resolution
2019-01-18 20:13:19.960 [INF] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): relaunching 1 contract resolvers
2019-01-18 20:13:19.961 [DBG] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): attempting to resolve *contractcourt.commitSweepResolver
2019-01-18 20:13:19.961 [DBG] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): contract *contractcourt.commitSweepResolver not yet resolved
2019-01-18 20:13:19.961 [DBG] CNCT: *contractcourt.commitSweepResolver(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): waiting for commit tx to confirm
2019-01-18 20:13:20.150 [INF] CNCT: *contractcourt.commitSweepResolver(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): waiting for commit sweep txid=3b6abf124da45c1a11f4811b21444586c2a3d720d1edcfdb1d8d03867da0788e conf
2019-01-18 20:15:04.452 [INF] UTXN: NurseryReport: building nursery report for channel cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0
2019-01-18 20:26:48.613 [INF] UTXN: NurseryReport: building nursery report for channel cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0

grep 825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e lnd.log | tail -n 30

2019-01-18 20:13:19.940 [DBG] CNCT: New ChainEventSubscription(id=0) for ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1)
2019-01-18 20:13:19.951 [DBG] CNCT: Starting chain watcher for ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1)
2019-01-18 20:13:19.951 [DBG] NTFN: Using height hint 537372 retrieved from cache for 825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1
2019-01-18 20:13:19.951 [INF] NTFN: New spend subscription: spend_id=2, outpoint=825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1, height_hint=537372
2019-01-18 20:13:19.957 [INF] CNCT: Close observer for ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) active
2019-01-18 20:13:19.960 [DBG] CNCT: Starting ChannelArbitrator(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1), htlc_set=(contractcourt.htlcSet) {
2019-01-18 20:13:19.961 [INF] CNCT: ChannelArbitrator(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1): starting state=StateCommitmentBroadcasted
2019-01-18 20:13:20.088 [INF] NTFN: Dispatching spend notification for outpoint=825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1 at height=559059
2019-01-18 20:13:20.088 [WRN] CNCT: Unprompted commitment broadcast for ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) 
2019-01-18 20:13:20.088 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 1s.
2019-01-18 20:13:20.107 [INF] CNCT: ChannelArbitrator(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1): noop trigger chainTrigger
2019-01-18 20:13:21.064 [DBG] LNWL: ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1), starting local commitment: (*lnwallet.commitment)(0xc0114b6000)({
    PreviousOutPoint: (wire.OutPoint) 825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1,
2019-01-18 20:13:21.066 [DBG] LNWL: ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1), starting remote commitment: (*lnwallet.commitment)(0xc010f3c000)({
    PreviousOutPoint: (wire.OutPoint) 825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1,
2019-01-18 20:13:21.066 [INF] PEER: NodeKey(02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b) loading ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1)
2019-01-18 20:13:21.066 [WRN] PEER: ChannelPoint(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) has status CommitmentBroadcasted, won't start.
2019-01-18 20:13:21.090 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 2s.
2019-01-18 20:13:23.090 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 4s.
2019-01-18 20:13:27.090 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 8s.
2019-01-18 20:13:35.093 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 16s.
2019-01-18 20:13:51.094 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 32s.
2019-01-18 20:14:23.097 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 1m4s.
2019-01-18 20:15:27.103 [ERR] CNCT: Unable to retrieve commitment point for channel(825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1) with lost state: no commit point found. Retrying in 2m8s.

grep fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4 lnd.log | tail -n 30

2019-01-18 20:13:19.940 [DBG] CNCT: New ChainEventSubscription(id=0) for ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1)
2019-01-18 20:13:19.951 [DBG] CNCT: Starting chain watcher for ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1)
2019-01-18 20:13:19.951 [DBG] NTFN: Using height hint 538170 retrieved from cache for fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1
2019-01-18 20:13:19.951 [INF] NTFN: New spend subscription: spend_id=1, outpoint=fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1, height_hint=538170
2019-01-18 20:13:19.957 [INF] CNCT: Close observer for ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) active
2019-01-18 20:13:20.107 [DBG] CNCT: Starting ChannelArbitrator(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1), htlc_set=(contractcourt.htlcSet) {
2019-01-18 20:13:20.107 [INF] CNCT: ChannelArbitrator(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1): starting state=StateCommitmentBroadcasted
2019-01-18 20:13:20.110 [INF] CNCT: ChannelArbitrator(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1): noop trigger chainTrigger
2019-01-18 20:13:20.160 [INF] NTFN: Dispatching spend notification for outpoint=fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1 at height=559059
2019-01-18 20:13:20.161 [WRN] CNCT: Unprompted commitment broadcast for ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) 
2019-01-18 20:13:20.162 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 1s.
2019-01-18 20:13:20.873 [DBG] LNWL: ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1), starting local commitment: (*lnwallet.commitment)(0xc010a44180)({
    PreviousOutPoint: (wire.OutPoint) fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1,
2019-01-18 20:13:20.875 [DBG] LNWL: ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1), starting remote commitment: (*lnwallet.commitment)(0xc010f22000)({
    PreviousOutPoint: (wire.OutPoint) fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1,
2019-01-18 20:13:20.875 [INF] PEER: NodeKey(032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada) loading ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1)
2019-01-18 20:13:20.876 [WRN] PEER: ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) has status CommitmentBroadcasted, won't start.
2019-01-18 20:13:21.165 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 2s.
2019-01-18 20:13:23.165 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 4s.
2019-01-18 20:13:27.169 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 8s.
2019-01-18 20:13:35.170 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 16s.
2019-01-18 20:13:51.169 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 32s.
2019-01-18 20:14:23.171 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 1m4s.
2019-01-18 20:15:27.174 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 2m8s.
2019-01-18 20:17:35.172 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 4m16s.

Roasbeef commented 5 years ago

If you restored from an old backup (which you should never do!), and the peers you were connected to either didn't support DLP (data loss protection), or the channel was closed before they were running 0.5.1, then they won't re-send the commitment point you need to sweep those funds. This was fixed in #1937. It's also the case that if you started with a stale backup, and broadcasted a revoked commitment, then the other party swept the funds. The channel (https://www.smartbit.com.au/tx/fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4) has been swept into the wallet, if you upgrade to master then if this hasn't confirmed yet we'll rebroadcast it with up to date fees.

kiaraho commented 5 years ago

I had no choice but to restore from a backup due to disk failure on the affected node. I have two nodes running in a part of the world where power outages are not uncommon. Is #1937 merged in 0.5.2 because both my nodes are running 0.5.1? I'm assuming that is what you meant by upgrading to master.

molxyz commented 5 years ago

@kiaraho

Is #1937 merged in 0.5.2 because both my nodes are running 0.5.1? I'm assuming that is what you meant by upgrading to master.

No, that's not what he meant by upgrading to master. We don't have v0.5.2 and master is a different branch where the code is most uptodate and it might be able to fix your issue. This is how you do to get master on your terminal screen:

cd $GOPATH/src/github.com/lightningnetwork/lnd
git fetch
git checkout origin/master
make && make install

Restart LND.

kiaraho commented 5 years ago

@molxyz, thanks for the clarification. I downloaded the LND binaries from https://github.com/lightningnetwork/lnd/releases/ instead of using Git. I'm assuming the most up to date code will eventually make its way into the next official release. If that is true, I can wait until those binaries are released.

halseth commented 5 years ago

New version of LND is now released.

halseth commented 5 years ago

Note that it is your channel peer that needs to update, and it probably won't work for old channels.

Roasbeef commented 5 years ago

I think this can be closed now as it's the role of DLP now (if possible) to remedy the situation?

kiaraho commented 5 years ago

Updated to v0.5.2 and waited for the node to get synched to chain. Limbo balance remains unchanged. The two channels below still show up under waiting_close_channels. Will wait to see what happens.

grep cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666 lnd.log | tail -n 30

2019-02-08 14:41:36.444 [INF] UTXN: NurseryReport: building nursery report for channel cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0
2019-02-08 14:45:46.430 [DBG] CNCT: Starting ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0), htlc_set=(contractcourt.htlcSet) {
2019-02-08 14:45:46.430 [INF] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): starting state=StateWaitingFullResolution
2019-02-08 14:45:46.432 [INF] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): still awaiting contract resolution
2019-02-08 14:45:46.432 [INF] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): relaunching 1 contract resolvers
2019-02-08 14:45:46.433 [DBG] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): attempting to resolve *contractcourt.commitSweepResolver
2019-02-08 14:45:46.433 [DBG] CNCT: ChannelArbitrator(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): contract *contractcourt.commitSweepResolver not yet resolved
2019-02-08 14:45:46.433 [DBG] CNCT: *contractcourt.commitSweepResolver(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): waiting for commit tx to confirm
2019-02-08 14:45:46.570 [INF] CNCT: *contractcourt.commitSweepResolver(cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0): waiting for commit sweep txid=3b6abf124da45c1a11f4811b21444586c2a3d720d1edcfdb1d8d03867da0788e conf
2019-02-08 16:21:50.202 [INF] UTXN: NurseryReport: building nursery report for channel cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0

grep fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4 lnd.log | tail -n 30

2019-02-08 14:45:46.240 [DBG] CNCT: Starting ChannelArbitrator(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1), htlc_set=(contractcourt.htlcSet) {
2019-02-08 14:45:46.241 [INF] CNCT: ChannelArbitrator(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1): starting state=StateCommitmentBroadcasted
2019-02-08 14:45:46.430 [INF] CNCT: ChannelArbitrator(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1): noop trigger chainTrigger
2019-02-08 14:45:46.504 [INF] NTFN: Dispatching spend notification for outpoint=fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1 at height=562158
2019-02-08 14:45:46.504 [WRN] CNCT: Unprompted commitment broadcast for ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) 
2019-02-08 14:45:46.504 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 1s.
2019-02-08 14:45:47.504 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 2s.
2019-02-08 14:45:49.505 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 4s.
2019-02-08 14:45:53.505 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 8s.
2019-02-08 14:46:01.505 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 16s.
2019-02-08 14:46:17.505 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 32s.
2019-02-08 14:46:49.506 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 1m4s.
2019-02-08 14:47:53.506 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 2m8s.
2019-02-08 14:49:37.293 [DBG] LNWL: ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1), starting local commitment: (*lnwallet.commitment)(0xc000da9ec0)({
    PreviousOutPoint: (wire.OutPoint) fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1,
2019-02-08 14:49:37.298 [DBG] LNWL: ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1), starting remote commitment: (*lnwallet.commitment)(0xc000b06840)({
    PreviousOutPoint: (wire.OutPoint) fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1,
2019-02-08 14:49:37.299 [INF] PEER: NodeKey(032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada) loading ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1)
2019-02-08 14:49:37.299 [WRN] PEER: ChannelPoint(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) has status CommitmentBroadcasted, won't start.
2019-02-08 14:50:01.506 [ERR] CNCT: Unable to retrieve commitment point for channel(fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1) with lost state: no commit point found. Retrying in 4m16s.

Roasbeef commented 5 years ago

@kiaraho you lost data. Therefore your node is now waiting on the remote peer to connnect and give you data you need to recover the channel state. Depending on which version they're running (or even implementation), they may never give you this special data. The latest version of lnd will always give you this data.

Roasbeef commented 5 years ago

See this comment: https://github.com/lightningnetwork/lnd/issues/2468#issuecomment-454306391

kiaraho commented 5 years ago

To all the channels we lost...

MykelSIlver commented 5 years ago

So it makes no sense to make backups. When something happens, you always restore an old state (backups are at least some hours old). When nodes close channels when your node is offline, and this is the case when you restore the backup, then those funds are lost.

MasterNeuron commented 5 years ago

I have the same problem with more than 3 BTC missing and "If you restored from an old backup (which you should never do!)" sounds like the most irrational thing, because:

Every backup is old and that's the essence of it.
LND have an option to create static channel backups...

Then why the f*ck I'm making those backups if I should never use them?!

Roasbeef commented 5 years ago

Static Channel Backups are static in that they never need to be updated. They're a snapshot of the static data in a channel that never changes. We've provided a safe way to do this for several releases now.

If you're attempting to backup the dynamic per state update data, then you'll need a proper strongly consistent backup system. Backing up state very N seconds and hoping things are consistent is not a valid way to do this.

MasterNeuron commented 5 years ago

Static Channel Backups are static in that they never need to be updated. They're a snapshot of the static data in a channel that never changes. We've provided a safe way to do this for several releases now.

If you're attempting to backup the dynamic per state update data, then you'll need a proper strongly consistent backup system. Backing up state very N seconds and hoping things are consistent is not a valid way to do this.

I'm confused, because there are two types of SCBs and both of them are files.

JSON ones which looks like that (have multi channel backup at the bottom):

And the other type have the following structure:

The command will accept backups in one of three forms:

 * A single channel packed SCB, which can be obtained from
   exportchanbackup. This should be passed in hex encoded format.

 * A packed multi-channel SCB, which couples several individual
   static channel backups in single blob.

 * A file path which points to a packed multi-channel backup within a
   file, using the same format that lnd does in its channels.backup
   file.

OPTIONS: --single_backup value a hex encoded single channel backup obtained from exportchanbackup --multi_backup value a hex encoded multi-channel backup obtained from exportchanbackup --multi_file value the path to a multi-channel back up file

How to import the JSON one into 0.7.1-beta commit=v0.7.1-beta-231-g1f92b7587ce031628bfb5efe2b60e0edf6648968-dirty ?

Tried to verify the json one and unable to do it:

Also tried to verify the other type of backup and the response is unknown?

Then tried this command after deleting the channel.db file from the LND wallet:

Logs are showing that something is happening:

Will keep you updated.

Roasbeef commented 5 years ago

That JSON one is just the list of channels the backup contains. The actual backup is a binary format.

On Wed, Oct 9, 2019, 6:11 PM MasterNeuron notifications@github.com wrote:

Static Channel Backups are static in that they never need to be updated. They're a snapshot of the static data in a channel that never changes. We've provided a safe way to do this for several releases now.

If you're attempting to backup the dynamic per state update data, then you'll need a proper strongly consistent backup system. Backing up state very N seconds and hoping things are consistent is not a valid way to do this.

I'm confused, because there are two types of SCBs and both of them are files.

JSON ones which looks like that: [image: https://i.postimg.cc/jjJ1CFBM/Screenshot-2019-10-10-at-4-04-17.png] https://camo.githubusercontent.com/cc5394170c408cd1bf6cf01ab11651bb66dff0d9/68747470733a2f2f692e706f7374696d672e63632f6a6a4a314346424d2f53637265656e73686f742d323031392d31302d31302d61742d342d30342d31372e706e67

And the other type have the following structure: [image: https://i.postimg.cc/bw5MyhVG/Screenshot-2019-10-10-at-4-04-06.png] https://camo.githubusercontent.com/5481d7528eafb562b142d59f3e77df5c4c964a93/68747470733a2f2f692e706f7374696d672e63632f6277354d796856472f53637265656e73686f742d323031392d31302d31302d61742d342d30342d30362e706e67

The command will accept backups in one of three forms:

A single channel packed SCB, which can be obtained from exportchanbackup. This should be passed in hex encoded format.

A packed multi-channel SCB, which couples several individual static channel backups in single blob.

A file path which points to a packed multi-channel backup within a file, using the same format that lnd does in its channels.backup file.

OPTIONS: --single_backup value a hex encoded single channel backup obtained from exportchanbackup --multi_backup value a hex encoded multi-channel backup obtained from exportchanbackup --multi_file value the path to a multi-channel back up file

How to import the JSON one into 0.7.1-beta commit=v0.7.1-beta-231-g1f92b7587ce031628bfb5efe2b60e0edf6648968-dirty ?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/lightningnetwork/lnd/issues/2468?email_source=notifications&email_token=AAHTWLX5A5SILBTQPZY54VLQNZ6KNA5CNFSM4GPVWNDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAZ7TZA#issuecomment-540277220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHTWLXUEKOMXU7UOD76YT3QNZ6KNANCNFSM4GPVWNDA .

MasterNeuron commented 5 years ago

That JSON one is just the list of channels the backup contains. The actual backup is a binary format.

Thank you, I've just started some recovery process using that command: lncli restorechanbackup --multi_file fullbackup-file

where fullbackup-file is the binary one

btw if the actual backups are in binary format, does that mean that the JSON files are not actual(unuseful) backups? because the previous LND versions are providing only such SCB JSON export option. If the newest LND versions are migrating the graph database version without providing an option to recover from JSON, then everybody who had a JSON backup may loose his channel states if there is some issue... because the database graph version will be upgraded and then usable only by the newest LND version which doesn't support JSON import. Or I am wrong?

Also will be good to have an option to view the mnemonic phases after unlocking the wallet.

update: looks like the "lncli restorechanbackup --multi_file fullbackup-file" command is stuck... and at the moment I lost ~3-4 BTC after force closing channels while having backups of everything, it's hard to explain how it happened.

Before few months I sold my house for that coins and put them on LND, because I trust that technology. Probably I'm doing something wrong and just need a little bit of support to recover my funds.

Roasbeef commented 5 years ago

lnd doesn't export any JSON over the main RPC interface. Only in lncli are things converted to JSON. We have RPC calls that return the SCB file, which is in raw binary, and not JSON.

Also will be good to have an option to view the mnemonic phases after unlocking the wallet.

The phrase itself isn't stored anywhere, it's the user's job to ensure that they've written down their seed, just as with any other wallet.

If you just forced closed the channels, then you don't need to recover anything. If you have that many channels, then it may take some time for all the information to be processed and for recovery to begin. I'd keep watching the logs for progress. Also the verifychanbackup will show an error if the backup was invalid.

Roasbeef commented 5 years ago

Then tried this command after deleting the channel.db file from the LND wallet:

No need to delete your channel.db....particularly if you didn't have any other copies and this was an existing node. We have documentation on how to recovery properly, please read these docs if you haven't already: https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md

MasterNeuron commented 5 years ago

Still stuck on that phase, what else should I do, waited more than 3 days and 4.26 BTC are locked there:

guggero commented 5 years ago

This means that lnd is trying to connect to the remote peer and ask it for the information it needs to recover the funds from the backup. Maybe the peers of the channels listed aren't currently online. Can you check if you can connect to them? Pick a channel from the list, find out what peer it is (maybe with the help of 1ml.com) and try lncli connect.

MasterNeuron commented 5 years ago

This means that lnd is trying to connect to the remote peer and ask it for the information it needs to recover the funds from the backup. Maybe the peers of the channels listed aren't currently online. Can you check if you can connect to them? Pick a channel from the list, find out what peer it is (maybe with the help of 1ml.com) and try lncli connect.

Thank you, will try that. Do I need to connect to all of them to make it works? Or is it possible to forgot/delete some channels?

update: looks like the backend is still syncing and I'm unable to connect

Also all of my pendingchannels don't have maturity height:

Roasbeef commented 5 years ago

You have very old channels which are likely still rescanning. Check the logs of your connected full node (on a higher log level) to monitor activity. You can also increase the logging level on lnd as well.

MasterNeuron commented 5 years ago

After one week of loading it's still stuck, cannot get my coins:

MasterNeuron commented 5 years ago

To all the channels we lost...

I have more than 4 BTC there which is ~30K $... that's so unserious. I've just force-closed my channels and lost them!!!

guggero commented 5 years ago

@MasterNeuron can you DM me at the community Slack? I have a branch up for fund recovery. I'm going to try to help you get our BTC back.

molxyz commented 5 years ago

@guggero I tested v0.7.1 and 0.8.0 if you try to run an older invalid channel state, LND closes all channels, you get your coins back to your wallet like any normal closing txs. You don't even have to pay a penalty, nobody takes your coins. I don't think @MasterNeuron lost his 4 BTC as he claims. Wondering if he screwed up his node in some other way but he can try to do onchain recovery by creating a new node and following the recovery guides https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md

Here's my log when I ran an old invalid channel state that had 2 channels, you can see LND closed them:

2019-10-25 14:10:38.027 [INF] CRTR: Block 000000003abe602685f4a1de50caa162d2e571e118c6fc109a91ee62c4b574a2 (height=1583986) closed 2 channels
2019-10-25 14:10:38.107 [INF] NTFN: New block: height=1583986, sha=000000003abe602685f4a1de50caa162d2e571e118c6fc109a91ee62c4b574a2
2019-10-25 14:10:38.107 [INF] NTFN: Dispatching confirmed spend notification for outpoint=e83c056fbd99cf0a9724f20241032d1b731bf87c3cdec44dd79710517e68caaf:0 at height=1583986
2019-10-25 14:10:38.107 [INF] UTXN: Attempting to graduate height=1583986: num_kids=0, num_babies=0
2019-10-25 14:10:38.108 [INF] NTFN: Dispatching confirmed spend notification for outpoint=0b0ee5b93babd482c707f6ef42f6fadd3df149cd569b92d1fa6762199bda0f0c:0 at height=1583986
2019-10-25 14:10:38.159 [WRN] CNCT: Unprompted commitment broadcast for ChannelPoint(e83c056fbd99cf0a9724f20241032d1b731bf87c3cdec44dd79710517e68caaf:0) 
2019-10-25 14:10:38.159 [WRN] CNCT: Remote node broadcast state #2, which is more than 1 beyond best known state #0!!! Attempting recovery...
2019-10-25 14:10:38.159 [WRN] CNCT: Unprompted commitment broadcast for ChannelPoint(0b0ee5b93babd482c707f6ef42f6fadd3df149cd569b92d1fa6762199bda0f0c:0) 
2019-10-25 14:10:38.160 [WRN] CNCT: Remote node broadcast state #2, which is more than 1 beyond best known state #0!!! Attempting recovery...
2019-10-25 14:10:38.160 [INF] CNCT: Recovered commit point(03564ef81fd391fd8e94c0b0205ebe069f194a6a4bf4e583c05e1abcffbdce426f) for channel(e83c056fbd99cf0a9724f20241032d1b731bf87c3cdec44dd79710517e68caaf:0)! Now attempting to use it to sweep our funds...
2019-10-25 14:10:38.160 [INF] CNCT: Recovered commit point(02c1f940e576723203235244a6541d11a593613d0d9aa9638c3b5cb78e19cdb0ae) for channel(0b0ee5b93babd482c707f6ef42f6fadd3df149cd569b92d1fa6762199bda0f0c:0)! Now attempting to use it to sweep our funds...
2019-10-25 14:10:38.160 [INF] CNCT: Unilateral close of ChannelPoint(0b0ee5b93babd482c707f6ef42f6fadd3df149cd569b92d1fa6762199bda0f0c:0) detected
2019-10-25 14:10:38.160 [INF] CNCT: Unilateral close of ChannelPoint(e83c056fbd99cf0a9724f20241032d1b731bf87c3cdec44dd79710517e68caaf:0) detected
2019-10-25 14:10:38.199 [INF] CNCT: ChannelArbitrator(0b0ee5b93babd482c707f6ef42f6fadd3df149cd569b92d1fa6762199bda0f0c:0): remote party has closed channel out on-chain
2019-10-25 14:10:38.199 [INF] CNCT: ChannelArbitrator(e83c056fbd99cf0a9724f20241032d1b731bf87c3cdec44dd79710517e68caaf:0): remote party has closed channel out on-chain

Roasbeef commented 5 years ago

Yeah I don't think all their funds are gone. In their past screenshot they have 0 peers, which means there's something else going on there. Things don't quite add up, and if we could get some more information or logs @MasterNeuron we'll actually be able to help.

MasterNeuron commented 5 years ago

Yeah I don't think all their funds are gone. In their past screenshot they have 0 peers, which means there's something else going on there. Things don't quite add up, and if we could get some more information or logs @MasterNeuron we'll actually be able to help.

Thank you!

It's my fault that LND got broken because of the power outage...

Currently I've recovered the node from a paper backup on another server with bitcoind instead of btcd. I've made a copy of all files when bicoind and lnd are stopped, will try to import all channels from a static SCB file and also via channels.backup file.

@guggero is helping me on slack and I'm really grateful.

Will keep you updated.

Roasbeef commented 5 years ago

@MasterNeuron in that screen shot above, your node has zero peers, why? Was it able to boot up properly, what information was/is there in the logs? If you're doing a full rescan, then you may need to extend the recovery window as is mentioned in recovery.md.

molxyz commented 5 years ago

@MasterNeuron

It's my fault that LND got broken because of the power outage...

I don't think it's productive to play this "fault" game. The power outage didn't destroy your logs did it? How about provide the logs at roasbeef's request and answer his questions so they can get some info of how to help you out? We all would like to help you recover your money and would like to see your cooperation if this money is so important to you. Volunteers spending their time and not getting paid, at least they should be appreciated by your co-operation in solving this case and not dragging it out too long or seeing another post on reddit to fuel up more fuds when actually nobody has seen any proof of a breach to show you lost anything.

So, Please come clean and get this mess explained and cleaned up. Thank you.

MasterNeuron commented 5 years ago

@MasterNeuron

It's my fault that LND got broken because of the power outage...

I don't think it's productive to play this "fault" game. The power outage didn't destroy your logs did it? How about provide the logs at roasbeef's request and answer his questions so they can get some info of how to help you out? We all would like to help you recover your money and would like to see your cooperation if this money is so important to you. Volunteers spending their time and not getting paid, at least they should be appreciated by your co-operation in solving this case and not dragging it out too long or seeing another post on reddit to fuel up more fuds when actually nobody has seen any proof of a breach to show you lost anything.

So, Please come clean and get this mess explained and cleaned up. Thank you.

Hi Molly, I'm not playing a game, just trying to bring back my coins. Let me explain everything. This is my on-chain BTC wallet address, everything from my 2 nodes is going there:

https://www.blockchain.com/btc/address/384UPF8UVTV8b8rA8Binvx6WmHxTQkYDPY

The current final balance is 2.62369167 BTC (0.24745529 of them are sent previously, wasn't received from LND nodes)

This is my 1st (small) node linked to my website functionality: https://1ml.com/node/0356c02ffe265f8ff38471e901785528d3da0668cd0a590ed67ee809d929c9bfd4

All channels from that small node are already closed and funds sent to: 384UPF8UVTV8b8rA8Binvx6WmHxTQkYDPY

This is my 2nd (big) node which had many channels: https://1ml.com/node/02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520

Those are my balances, I've checked and typed them every few days: LND BALANCES - LND BALANCES.pdf same uploaded here: http://lightning-casino.com/balances.pdf

Latest final balance from both of them is 618142594 satoshis (field J:73), 2.62369167-0.24745529 = 2,37623638 BTC totally received from both nodes, which means that still need to unlock 6.18142594 - 2.37623638 = 3,80518956 BTC still missing.

Let me explain the chronological way:

In the beginning of September after reading that tweet: https://twitter.com/rusty_twit/status/1167371549575303169 decided to upgrade both nodes, date was 09.09.2019. I've stopped the LND node, downloaded the latest release from here https://github.com/lightningnetwork/lnd/releases, uploaded and placed both lnd and lncli in the bin Go folder. I've copied the whole .lnd folder when my node was stopped.

Also upgraded the btcd same way (my LNDs were using btcd instead of bitcoind)

After starting it with the newer LND 0.7.1 version, some database migration/convertion process started and the "synced_to_graph" was false while "synced_to_chain" was true. I've just left that process and after few days everything was fine, both of those values was "true" in the middle of September, I don't remember the exact date. My server is based on Ubuntu 16.04 LTS and there is a background veeam service which is backing up all files at midnight every day. Also every few weeks I'm stopping the LND manually to copy .lnd folder manually to my own PC just in case. Same way I've made few SBCs, first versions provided them in JSON format, next ones was binary files. I'm able to provide log files from any* previous date, @molxyz if you need a log from an exact date just let me know and I will provide it.

In the end of September when I logged into my big node again (don't have issues with the small node called NEURON), realised that it was synced only to chain without any sync to graph. Then decided that something is wrong and wanted to take back my coins after calling "lncli closeallchannels --force" (yes, I've already know that it was a stupid idea, but still wondering why LND is allowing that while it's not synced to graph) and after running that function for few minutes it wasn't completed, like lncli stuck on running it. Then pressed Ctrl + C and decided to close my channels from some previous state (which as I already know is not a good idea too), then loaded some older backups, waited to sync and force-closed them same way. After that I was calm, because as I know should wait up to 2 weeks and just waited from 2nd of October to ~18.10, after that realised that my coins are not back into the walledbalance and the panic began... Then I've tried to recover from many different backups and states, before LND upgrade, after that, before force-closing, after that... and started to wondering what happened, remembered that my electricity stopped 2 or 3 times in the end of September, told my girlfriend and she checked the local facebook group where all girls are sharing "interesting" events like local power outages etc... and the 2 dates were 01.10.2019 18:41 and 25.09.2019 00:12 GMT+2 that's why I suppose that my LND graph was broken because of those power outages. And after that mess... I have many renamed .lnd folder without deleting anything*, just renaming and copying the folders + veeam backups from midnight, few SCBs and paper backup.

Just need to tell me the exact date and I will provide the logs from .lnd folder

Currently on my small node called NEURON, I've switched from btcd to bitcoind, placed the latest LND "version": "0.8.0-beta commit=v0.8.0-beta" and recovered the node from fresh paper backup. Now it's totally clean and showed as syncronised:

It's synced from few hours, but still not updated in 1ml.com, I've changed the name: https://1ml.com/node/02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520

Do you have some suggestions how to proceed from that state ;?

vegardengen commented 5 years ago

@MasterNeuron - great, now you are half way there!

This is the place where you follow https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md#recovering-using-scbs

Remember - there's probably be a lot of mess because channels are closed in the meantime, the important thing here is patience. Wait it out some days, and watch what happens in your lncli pendingchannels as your channel peers that still exist close the channels. Your nodes should also now find and be able to sweep the funds that belongs to you that are just sitting on the end of the timelock, waiting for you to grab them and move them to your onchain wallet.

vegardengen commented 5 years ago

There will not be any gossiping of channels etc as long as you just connect it.

You should do the restore. The 15 channels you see on 1ml.com is probably channels that were not force-closed, and should be recoverable.

vegardengen commented 5 years ago

One other thing - that looks like your big node? Be sure not to mix up which restore you restore. Nothing bad will happen except you will be more confused and frustrated, as your node will not accept it as a valid backup.

The backup is encrypted with the private key of your node.

molxyz commented 5 years ago

@vegardengen @MasterNeuron I'd rather wait for the devs, @Roasbeef @guggero @wpaulino to take a look at his last post here to see what he should do next or which logs they need to diagnose this problem. It seems quite a mess to be honest, since he got things tangled up, but let's wait for the devs to see what he should do next.

guggero commented 5 years ago

@MasterNeuron: your "small" node is now ready to begin the SCB recovery. Please run lncli restorechanbackup --multi_file path/to/channel.backup where channel.backup is the binary backup file that is created by lnd in the .lnd/data/chain/bitcoin/mainnet folder. Use the most recent that you have.

It is normal that the name of the node doesn't update as long as you don't have any channels. Without any channels, no broadcasts happen and your node is basically "invisible" to the network.

MasterNeuron commented 5 years ago

@MasterNeuron: your "small" node is now ready to begin the SCB recovery. Please run lncli restorechanbackup --multi_file path/to/channel.backup where channel.backup is the binary backup file that is created by lnd in the .lnd/data/chain/bitcoin/mainnet folder. Use the most recent that you have.

It is normal that the name of the node doesn't update as long as you don't have any channels. Without any channels, no broadcasts happen and your node is basically "invisible" to the network.

Thank you, will try that and keep you updated.

lightningnetwork / lnd