search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.7k stars 2.09k forks source link

Still cannot force close inactive channels #2468

Closed kiaraho closed 4 years ago

kiaraho commented 5 years ago

Background

Follow up to issue #2391. Now running 0.5.1. Most of the channels have closed and I can see the funds on-chain when I run lncli walletbalance. However, there are still some funds off-chain in one pending_force_closing_channels and two waiting_close_channels. Those channels have been in limbo for several days now.

Your environment

lncli version 0.5.1-beta commit= Linux hostname 4.15.0-43-generic #46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Bitcoin Core Daemon version v0.17.0.0-ge1ed37edaedc85b8c3468bd9a726046344036243

Steps to reproduce

Ran lncli closeallchannels --force to close inactive channels. Got the following message:

{ "remote_pub_key": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b", "channel_point": "825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1", "closing_txid": "cb118298c5526ca09576dd54f8be48b06994fafb790738879069ef50d8e156e6", "error": "" } { "remote_pub_key": "032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada", "channel_point": "fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1", "closing_txid": "e2e7c83691ad21deb37a42bcbb22886225a88c0b67a57be94e0060992d8e0b3c", "error": "" }

Expected behaviour

Inactive channels should close.

Actual behaviour

Three channels still stuck in limbo. The two closing_txid above do not seem to have been broadcast. Running lncli closeallchannels --force again returns a "[lncli] no open channels to close" message. Output from lncli pendingchannels command:

{ "total_limbo_balance": "11783124", "pending_open_channels": [ ], "pending_closing_channels": [ ], "pending_force_closing_channels": [ { "channel": { "remote_node_pub": "029afc726a18abc8dc75ef6c9ed34354c275261086597d98067dd972bd72965943", "channel_point": "cbb1d6bdfa2d1197b13167e83c3b7e18feb410e8cbff02d0b5488e14cd1be666:0", "capacity": "2062186", "local_balance": "2054765", "remote_balance": "0" }, "closing_txid": "297735f112539d8c1f7325863468b8a67c20f3584be06bb4f9634c10c458df36", "limbo_balance": "0", "maturity_height": 0, "blocks_til_maturity": 0, "recovered_balance": "0", "pending_htlcs": [ ] } ], "waiting_close_channels": [ { "channel": { "remote_node_pub": "02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b", "channel_point": "825f7a52a6ac12b40f5ae110baf00441a080bea290abcf403d2172eebe90126e:1", "capacity": "7119858", "local_balance": "6947834", "remote_balance": "0" }, "limbo_balance": "6947834" }, { "channel": { "remote_node_pub": "032b2b3f4abda9677bb9563e226c068d3a2456fb8b036635a81c9bcaa1671d1ada", "channel_point": "fe49e1cc7f146578a73a42bb371355e087d1b4e14764ca1cd485085d3e3665b4:1", "capacity": "4842711", "local_balance": "4835290", "remote_balance": "0" }, "limbo_balance": "4835290" } ] }

MasterNeuron commented 5 years ago

@guggero I've tried that and it's returning an error:

ubuntu@ip-172-31-29-90:~> /home/ubuntu/gocode/bin/lncli restorechanbackup --multi_file /home/ubuntu/channel.backup [lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: a height hint greater than 0 must be provided

looks like a bug related to that github issue: https://github.com/lightningnetwork/lnd/issues/3444

guggero commented 5 years ago

It seems like that specific version of the SCB is in a strange state. But we need to fix that error anyway. In the meantime, can you please try an older version of the channel.backup file? For this particular type of file it is safe to use any old version. The only risk is that a channel is not yet contained in it. But we can work around that once #3444 is fixed.

MasterNeuron commented 5 years ago

It seems like that specific version of the SCB is in a strange state. But we need to fix that error anyway. In the meantime, can you please try an older version of the channel.backup file? For this particular type of file it is safe to use any old version. The only risk is that a channel is not yet contained in it. But we can work around that once #3444 is fixed.

After switching from "LND 0.8.0-beta commit=v0.8.0-beta" to a new freshly synchronised "LND 0.7.1-beta commit=v0.7.1-beta", tried few older channel.backup files and another error appeared:

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: chacha20poly1305: message authentication failed

vegardengen commented 5 years ago

Are you sure you are using the backup from the correct node?

vegardengen commented 5 years ago

I'm a bit confused. 3 days ago, you posted a long mail, claiming that your small node had nodeID: 0356c02ffe265f8ff38471e901785528d3da0668cd0a590ed67ee809d929c9bfd4

and the large one: 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520

Then you post a picture of something you claim is your upgraded small node, but which have the node ID you previously claimed is the large one.

That's why I am asking.

MasterNeuron commented 5 years ago

It's little bit confusing.

I was able to bring back all coins from my small node and after that decided to link LND with bitcoind instead of btcd. To achieve that I've recovered the big node seed on my small node server, deleted btcd, installed and linked bitcoind on its place, currently dealing only with it. That's why at the moment they have the same IP addresses. Currently trying to import my channels somehow, will keep you updated.

vegardengen commented 5 years ago

instead of restorechanbackup, you can also use verifychanbackup .

And as for which SCB I'd use - I think I'd prefer to use the last good one before october 2nd .

MasterNeuron commented 5 years ago

After recovering from seed and syncing, currently importing a channel.backup with LND 0.7.1 from 30.09.2019, it's still loading and I hope that after few hours all channels from that date will be back.

There are log lines like that which looks promising: [INF] LTND: Inserting 1 SCB channel shells into DB

Screenshot 2019-10-29 at 23 05 20
vegardengen commented 5 years ago

Here is where you need patience. Look at your pendingchannels and wait it out

kiaraho commented 5 years ago

Glad to see somebody is making progress in recovering their funds.

As an update to the original post in this thread, I was able to close and recover my funds from my node with public key 0249c30c3aedc95d87e430f4b6c076049714c5b7ea536d4e4e1863c5ec8f73821c, but not the one with public key 0312729bfe8c02a1189a94bf609be3f44e7e30f581ada3429925221915d5d6cfb3. Given my nodes were running LND 0.5.1 prior to DLP, I'm assuming the second node's channel state is lost for good. Would be happy to be proven wrong.

guggero commented 5 years ago

@MasterNeuron good to see you are making progress. As @vegardengen said, this is probably going to take some time.

@kiaraho if you still have the seed, you could try with my "manual" recovery mode: https://medium.com/@guggero/did-you-lose-funds-on-the-lightning-network-because-of-a-disk-crash-8971b6a92494 It requires the remote peer to be online and still know about the channel. So it's not a guarantee but might give you another chance.

molxyz commented 5 years ago

@kiaraho

Given my nodes were running LND 0.5.1 prior to DLP, I'm assuming the second node's channel state is lost for good. Would be happy to be proven wrong.

@guggero wrote a blog how to recover funds like this, someone on LND slack successfully recovered his funds by following the guide, just don't throw away your data. EDIT: Ooops... lol forgot to refresh gh page, didn't see you already replied to his comment, @guggero

vegardengen commented 5 years ago

@MasterNeuron - I am a bit curious about the progress? Are you able to recover some money, by now?

MasterNeuron commented 5 years ago

@MasterNeuron - I am a bit curious about the progress? Are you able to recover some money, by now?

Hi @vegardengen, unfortunately not.

I've just woke up and checked my node, after ~2 days of loading it's over. That's the current node state after importing the channel.backup file from 30.09.2019 on LND 0.7.1, recovered from fresh paper wallet seed of my node:

Screenshot 2019-10-31 at 10 44 12

pendingchannels response is that:

Screenshot 2019-10-31 at 10 52 29

pendingchannels.txt

listchannels:

Screenshot 2019-10-31 at 11 03 26

I'm stuck... in limbo.

Regards, INWHY

vegardengen commented 5 years ago

Hi,

A restart is known to kick off something, sometimes. Can not hurt.

What you need now, is for your node to find the already submitted force-close transactions.

I can see that at least in the channel I have with you, your share of my channel with you is waiting for you to claim them and move them to your onchain balance.

Btw: Was there no onchain balance in your node? If there was, you might need to increase the lookahead count when rescanning. But for now, I think that can wait.

MasterNeuron commented 5 years ago

Hi,

A restart is known to kick off something, sometimes. Can not hurt.

What you need now, is for your node to find the already submitted force-close transactions.

Will restart it now. Also when the node is turned off will copy the .lnd folder and will switch to LND 0.8.0 to give it a try.

I can see that at least in the channel I have with you, your share of my channel with you is waiting for you to claim them and move them to your onchain balance.

That sounds promising : )

Btw: Was there no onchain balance in your node? If there was, you might need to increase the lookahead count when rescanning. But for now, I think that can wait.

384UPF8UVTV8b8rA8Binvx6WmHxTQkYDPY - That's my wallet address and before starting the channels import process I've sent on-chain transaction, probably that's why the current balance is zero, every time when there are some bitcoins I'm sending them no-chain.

Thank you!

Update: I've tried with LND 0.8.0 and unable to turn it on: "Unable to start server: a height hint greater than 0 must be provided"

Screenshot 2019-10-31 at 14 50 19

With LND 0.7.1 it starts well, but still showing zero balances, also there is a strange error:

2019/10/31 13:00:05 http: TLS handshake error from 127.0.0.1:59612: remote error: tls: unknown certificate authority

vegardengen commented 5 years ago

@guggero - any suggestion?

I actually have another suggestion, if the restore from SCB doesn't work out, but ideally I think restore from SCB should work.

Where do the spend subscriptions get their spends from? In our case, they all happened on october 2nd, so quite a few blocks ago....

@MasterNeuron what's becoming increasingly clear is that the fact that you started your node with an old database at october 2nd caused the other nodes (mine included) to use the SCB protocol and do a normal force close, where the funds are just waiting for you to claim them. In my understanding, the restorechanbackup have restored the keys which should enable you to do just that.

MasterNeuron commented 5 years ago

Is there any LND call to claim my funds or that should happen automatically after syncing?

vegardengen commented 5 years ago

That should happen automatically, the spend notifications should pick up those force-closes that has already happened. But it could be that you need more patience.

@guggero - how does this scan for old transactions happen? I guess it will have to scan from the height hint until it finds the transaction?

What is happening in the log files? Perhaps increasing some log level would reveal some activity.

MasterNeuron commented 5 years ago

@vegardengen Thank you about everything, I'm still trying to bring back my funds somehow. Found the best backup with logs from the days when all channels were closed, uploaded the log folder on google drive: https://drive.google.com/file/d/13AP_S9rNGaUzSIzIfCsY31ntHBnz2Cod/view?usp=sharing

Btw I found that article from @guggero https://medium.com/@guggero/did-you-lose-funds-on-the-lightning-network-because-of-a-disk-crash-8971b6a92494 and looks like he have a commission "I can also try to recover the funds for you but then I would need to charge 20% of the recovered satoshis because it’s quite some work" and may need to pay him ~ 7400$ for that service (~0.8 BTC)

Currently trying to make it works, will set the debug level to "trace" and keep you updated.

vegardengen commented 5 years ago

Do you have the logs from october 2nd? That's when all the channels were closed (at least those I have seen).

MasterNeuron commented 5 years ago

Do you have the logs from october 2nd? That's when all the channels were closed (at least those I have seen).

Sorry, I've provided wrong logs from 02.09.2019 instead of 02.10.2019. With zgrep found many logs from 2nd of October, zipped and uploaded all of them:

.NUCLEUS/LNDS/.lnd-save/logs/bitcoin/mainnet/lnd.log
.NUCLEUS/LNDS/.lnd-save/logs/bitcoin/mainnet/lnd.log.93.gz

.NUCLEUS/LNDS/.lnd-current2/logs/bitcoin/mainnet/lnd.log

.NUCLEUS/LNDS/.lnd-current3/logs/bitcoin/mainnet/lnd.log

.NUCLEUS/LNDS/.lnd-copy/logs/bitcoin/mainnet/lnd.log
.NUCLEUS/LNDS/.lnd-copy/logs/bitcoin/mainnet/lnd.log.93.gz
.NUCLEUS/LNDS/.lnd-copy/logs/bitcoin/mainnet/lnd.log.93

.NUCLEUS/LNDS/.lnd-havesomechance/logs/bitcoin/mainnet/lnd.log.241.gz
.NUCLEUS/LNDS/.lnd-havesomechance/logs/bitcoin/mainnet/lnd.log.240.gz
.NUCLEUS/LNDS/.lnd-havesomechance/logs/bitcoin/mainnet/lnd.log.242.gz

.NUCLEUS/LNDS/.lnd-buggy/logs/bitcoin/mainnet/lnd.log

.NUCLEUS/otherbackups/.lnd-current1/logs/bitcoin/mainnet/lnd.log
.NUCLEUS/otherbackups/.lnd-current1/logs/bitcoin/mainnet/lnd.log.241.gz
.NUCLEUS/otherbackups/.lnd-current1/logs/bitcoin/mainnet/lnd.log.240.gz

.NUCLEUS/otherbackups/.lnd/logs/bitcoin/mainnet/lnd.log

There are many folders with copies, because I wasn't aware of that "latest state" requirement.

https://drive.google.com/file/d/10gKEowvm3Hx2Z70uEOTsX8trT5Rv0yyD/view?usp=sharing

MasterNeuron commented 5 years ago

I'm currently running one of those backups and unable to sweep some coins which are showing in walletbalance:

Screenshot 2019-11-04 at 15 03 26

If the output is already spent, then why it's showing in walletbalance while synced_to_chain = true

vegardengen commented 5 years ago

LND onchain wallet can get out of sync like that, "pretty easily". I'd not really worry about that now, but if you would really like to, there are ways to rescan.

More importantly: Where is this channel balance you are trying to sweep coming from? If it coins you sent off from another backup, then they don't really exist in your copy.

But: Which backup are you now running? I was going to suggest, as a next step, to run one taken after having started your node on october 2nd, but before the restore on the 3rd. It's quite likely that that is the one with the most correct information.

But, let is hold off for someone elses opinion. @guggero ?

MasterNeuron commented 5 years ago

I'm still "fighting" with my node backups and wanna ask publicly @guggero for support, able to give him all files, wallet password and 24 words seed for 20% commission tax of recovered (if any) satoshis.

That's my own wallet address: 384UPF8UVTV8b8rA8Binvx6WmHxTQkYDPY

vegardengen commented 5 years ago

I have now looked through some logs. In your lnd-havesomechance/log/bitcoin/mainnnet/lnd.log.242.fz is where I can see some instances of:

2019-10-02 11:59:25.724 [DBG] PEER: Sending ChannelReestablish(next_local_height=1, remote_tail_height=0) t
o 0200dfda3cb0250b2d801d200155ec056958cf598a30ffd36c207340cb9f6645a3@76.20.210.129:9735
2019-10-02 11:59:25.909 [DBG] PEER: Received ChannelReestablish(next_local_height=501, remote_tail_height=5
00) from 0200dfda3cb0250b2d801d200155ec056958cf598a30ffd36c207340cb9f6645a3@76.20.210.129:9735
2019-10-02 11:59:25.911 [INF] HSWC: Received re-establishment message from remote side for channel(9785a3d3
58ed96c435656b3d50fb1e4871167fd4b2a086c530c8de8550976f93:1)
2019-10-02 11:59:25.911 [ERR] LNWL: ChannelPoint(9785a3d358ed96c435656b3d50fb1e4871167fd4b2a086c530c8de8550
976f93:1), sync failed with local data loss: remote believes our tail height is 500, while we have 0!
2019-10-02 11:59:25.912 [WRN] LNWL: ChannelPoint(9785a3d358ed96c435656b3d50fb1e4871167fd4b2a086c530c8de8550976f93:1): detected restored triggering DLP
2019-10-02 11:59:25.912 [WRN] HSWC: Error when syncing channel states: ChannelPoint(9785a3d358ed96c435656b3d50fb1e4871167fd4b2a086c530c8de8550976f93:1) with CommitPoint(03ba85737f49b8b949d75d2d58a551d46b4bbdc1be0a35c253e24da25595498bc6) had possible local commitment state data loss
2019-10-02 11:59:25.938 [ERR] HSWC: ChannelLink(571528:2517:1) Failing link: unable to synchronize channel states: ChannelPoint(9785a3d358ed96c435656b3d50fb1e4871167fd4b2a086c530c8de8550976f93:1) with CommitPoint(03ba85737f49b8b949d75d2d58a551d46b4bbdc1be0a35c253e24da25595498bc6) had possible local commitment state data loss

I can also see, down there, and in many logs:

2019-10-10 23:55:20.294 [DBG] NTFN: Dispatching historical spend rescan for outpoint=aebc8f6f39f0057aec4ac7dafc2c61fe8576f61e3b0dcb3458c3261b37ef3253:0

These are what ought to find the force-close funds from your channel partners force-closed channels. And you are sure that they never found anything?

vegardengen commented 5 years ago

I would now like to see the logs from your longest running restore of a channel.backup, after this incident...

Btw, more logs. You are not running with a pruned backend?

2019-10-10 16:06:14.745 [ERR] CNCT: Unable to retrieve commitment point for channel(ff77fc071c7ec3441830af465764d 0208c4028e93adcf6580bc2727bde2d8c04:0) with lost state: no commit point found. Retrying in 10m0s.

guggero commented 5 years ago

I haven't had time to look at the logs. So thanks @vegardengen for digging into them!

It looks like there are at least two problems that block you from getting all channels back:

  1. You initialized SCB, which in general is the right thing to do. But SCB relies on the remote peer to be available to give you the data you need to recover the funds. If a peer is offline for good, you won't be able to get your channel balance back with SCB. The error message CNCT: Unable to retrieve commitment point for means: "I'm trying to initialize DLP but the remote peer is not responding. Trying again later."

  2. Your chain backend (bitcoind or btcd?) might be overwhelmed with all the rescans it has to do. It's also a very valid question if you run a pruned node? It looks like some rescans are getting lost if the wallet thinks it has UTXOs that are in fact already spent.

Also, I really would have liked to see what would have happened with a full chain rescan while you still had the channel DB up, after you did the "foce close all" command. Because there you would have had a chance of getting funds from non-existing peers while the DB was still intact. But you went ahead and jumped right into SCB where we know for sure that offline peers will never help us.

So it's a huge steaming mess right now with over 400 channels that we have no idea what state they are in. I'm not sure if I even want to give this a try. There's so much code that needs to be written and so much rescanning to be done, it will take forever... This is not at all the situation that I described in my article.

MasterNeuron commented 5 years ago

Btw, more logs. You are not running with a pruned backend?

Sorry, I don't know what pruned backend means.

Your chain backend (bitcoind or btcd?)

Currently using bitcoind, previously btcd.

If a peer is offline for good, you won't be able to get your channel balance back with SCB.

The @vegardengen node is not offline but my funds are still locked.

And you are sure that they never found anything?

At the moment from both nodes I've recovered totally 2.4468972 BTC out of 6.18142594, still need to bring back ~3.73452874 BTC, some of the recovered coins was in my walletbalance even before "closing" those channels.

Also, I really would have liked to see what would have happened with a full chain rescan while you still had the channel DB up, after you did the "foce close all" command.

I've tried and nothing was found.

I would now like to see the logs from your longest running restore of a channel.backup, after this incident...

Let me provide the logs from that "recovery" process described above:

I've just woke up and checked my node, after ~2 days of loading it's over. That's the current node state after importing the channel.backup file from 30.09.2019 on LND 0.7.1, recovered from fresh paper wallet seed of my node:

Screenshot 2019-11-05 at 11 13 35

pending.txt

So it's a huge steaming mess right now with over 400 channels that we have no idea what state they are in. I'm not sure if I even want to give this a try.

Who should have an idea if not the software?, it must work flawlessly like an ATM machine.

After providing a backup file if something is wrong the software need to say "hey, dude, you f***ed up everything and I'm recovering only 20 channels out of 400, those are your logs!"

That would be the main purpose of our backups, to sort out everything no matter how many channels are there. Ok, if 400 are too many, then set some limitations...

It should work like a PLUG & PLAY device. flawlessly. If channel balances are still owned by my node, then what's the issue? - I'm not a programmer, but it's sure that for one reason or another, the software just fails to sweep them back.

I thought that it's enough to have a paper seed backup + static channel backup + file backups + passwords to recover them...

... like every normal software

vegardengen commented 5 years ago

@MasterNeuron - LND will never be an end-user-friendly wallet, it will probably always require some knowledge. There exists more end-user-options from Lightning Labs, the lightning-app, which is available both in desktop, android and mobile version.

That said, everyone agrees that it has a way to go, still, which is everyone is always providing disclaimers about not putting in more than you are willing to risk.

Right now, I would like to see logs from the longest-running, most detailed restore from seed plus channel.backup.

In case it is still unclear, you should then start with a blank state: No wallet.db and no channel.db, it should all be provided by seed and channel.backup.

Do you still have such a restore running? As previously mentioned, this may definitely take time.

MasterNeuron commented 5 years ago

@MasterNeuron - LND will never be an end-user-friendly wallet, it will probably always require some knowledge. There exists more end-user-options from Lightning Labs, the lightning-app, which is available both in desktop, android and mobile version.

There are some differences between end-user-friendly and stable. If the coins are still locked in a limbo state between my node and your node, and if the software was stable, then everything should be fine, but it's not.

That said, everyone agrees that it has a way to go, still, which is everyone is always providing disclaimers about not putting in more than you are willing to risk.

You are right about that.

Right now, I would like to see logs from the longest-running, most detailed restore from seed plus channel.backup.

In case it is still unclear, you should then start with a blank state: No wallet.db and no channel.db, it should all be provided by seed and channel.backup.

Do you still have such a restore running? As previously mentioned, this may definitely take time.

Here is it: lnd.log

vegardengen commented 5 years ago

It looks to me like things are progressing, gradually. Do you get any new incoming funds in lncli listchaintxns?

Do you get any new closed channels in lncli closedchannels

Is the number of entries in pendingchannels decreasing?

MasterNeuron commented 5 years ago

I don't have any closedchannels, pendingchannels are still the same amount without any change:

ubuntu@ip-172-31-29-90:~> /home/ubuntu/gocode/bin/lncli closedchannels { "channels": [ ] }

I'm still trying to fix some SCB import errors:

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: chacha20poly1305: message authentication failed

and another one

Unable to start server: a height hint greater than 0 must be provided

It's full of errors. Many people blame me that it's only my fault, but I have to say that the software isn't less buggy. lightning-casino is my second blockchain project, the first one was created in early 2014, co-founded one of the first bitcoin exchanges in my country, that's me from 2016: https://www.bloombergtv.bg/investbook/2016-06-28/kriptovalutata-se-razviva-po-barzo-ot-tehnologiite-za-sigurnost

guggero commented 5 years ago

It's full of errors.

Yes, you are right. That's why there still is a -beta suffix in each version that we release.

But I am working on fixing those SCB errors. The a height hint greater than 0 must be provided should be an easy fix. I think it happens when there is a channel in the SCB file that was not confirmed yet at the time of the file update. We should just skip this channel instead of failing the whole file.

The message authentication failed is harder to pin down. It either means that you are trying to use a channel backup file that was not created with the node/seed you are trying to restore. But it could also mean that the SCB file is corrupt for some reason.

I'm going to write a script that goes through all your 400 channels and looks at the TX outputs if there is still something to be claimed. This should also give us a final account of how much was actually lost because of justice transactions (if any).

guggero commented 5 years ago

Looking at the log file, you need to let your node where you started the rescue run for a few days. When you shut it down on October 31, there were a lot of messages like:

Rescan to determine the spend details of outpoint=6a60b1f49b94f31dc4409f210d86aac32454d344466ae3d92f4477c87b7a318d:1 within range 570450-601598 failed: chain notifier shutting down

Which means, the node was still rescanning channels from the chain backend (bitcoind). This can take a long time!

MasterNeuron commented 5 years ago

@guggero, Thank you, If I recover some funds will give you a percentage, because without your help I'll be lost.

vegardengen commented 5 years ago

What I meant when I said, "looks like it is progressing nicely", is this:

2019-10-31 12:32:27.931 [INF] LTND: Received interrupt 2019-10-31 12:32:27.932 [INF] LTND: Shutting down... 2019-10-31 12:32:27.932 [INF] LTND: Gracefully shutting down. 2019-10-31 12:32:28.150 [ERR] NTFN: Rescan to determine the spend details of outpoint=bbc18a3bda9d24f99b74bdc89b4 d52671365dd9265f3cb770eb27d742a527e5c:0 within range 566116-601590 failed: chain notifier shutting down 2019-10-31 12:32:30.009 [ERR] NTFN: Rescan to determine the spend details of outpoint=30a6add708ccc85aa38c5015029 b23487f6b46d3adf21d974b87260387362097:1 within range 568399-601613 failed: chain notifier shutting down

If you let these run, there's all reason to believe that your node will find spend details of those outpoints, and it has the keys to sweep them.

I believe this is likely to bring back at least quite a lot of your funds, @guggero has promised to write a script which is likely to tell us to which extent.

The reason why your restore of the static channel backup doesn't work is something else which we don't fully understand. Maybe if you could show us the exact steps you tried when restoring it?

For the record, the starting procedure for using a static channel backup should, in your case, be an empty node. No wallet, no channel database, just your 24 word seed and the static channel backup.

But I believe the best way forward is to continue the attempt which you seemed to stop at november 5th. Let it run - for a week or two if it needs to. There's a lot of scanning your node has to do.

And watch your logs, your listchaintxns and closedchannels and see what happens, if you like. Because every now and then, it should find a closing transaction that it need to sweep funds from.

vwoo commented 5 years ago

The logs contain commit points, such as in vegardengen's comment https://github.com/lightningnetwork/lnd/issues/2468#issuecomment-549677751

As a last resort, could we derive the private key for each utxo with MasterNeuron's lnd aezeed private key and the commit point (if found in the log)? The channel point is right there as well so we can find the relevant utxo with MasterNeuron's funds.

MasterNeuron commented 5 years ago

@vwoo Thanks for your support, I'm chatting with @guggero on lightning community slack channel and he will try to recover some funds for me with 20% commission which is acceptable. Will send everything to him based on trust, looks like he helped other people before and I hope that there is a chance for my funds. Currently typing an encrypted email with node seed, files, ssh server access etc. That's our conversation:

slack.txt

MasterNeuron commented 4 years ago

To keep everybody updated, @guggero unlocked more than 1 BTC of those "lost" coins:

https://www.blockchain.com/btc/tx/1478c8d3e95043f77a18a50e1f51822930c03c1ba60662a623118eae7e94b3a9 https://www.blockchain.com/btc/tx/ed16c57d711061575495ac730ff4d55c13dde5756487a5e3e762986e11030b06

He explained to me every step of that process which is still ongoing.

guggero commented 4 years ago

I'm finally closing this issue as the remaining 3.7 BTC have been restored. This is the tool I created to assist in the recovery, combined with SCB of course.

These are the most important things we learned from this issue (for people coming across this when searching):