Partially recovered off-chain balance

robclark56 commented 5 years ago

Background

Following a power failure, LND now crashes/panics just after wallet is unlocked

Your environment

version of lnd Version: 0.7.1-beta commit=v0.7.1-beta, build=production, logging=default
which operating system (uname -a on *Nix) Linux RaspiBolt 4.14.30-v7+ #1102 SMP Mon Mar 26 16:45:49 BST 2018 armv7l GNU/Linux
version of btcd, bitcoind, or other backend Bitcoin Core RPC client version v0.17.1
any other relevant environment details All had been running very stable for a few months (since approx end July). Was only the power failure that caused the problem.

BTW - I do have SCB files (Static Channel Backups) back for months. (I save them offsite with time stamps)

Steps to reproduce

Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.

Start lnd
Unlock wallet

Expected behaviour

Tell us what should happen

Should not crash

Actual behaviour

Tell us what happens instead

bitcoin@RaspiBolt:/home/admin $ lnd
2019-10-07 06:39:56.519 [INF] LTND: Version: 0.7.1-beta commit=v0.7.1-beta, build=production, logging=default
2019-10-07 06:39:56.519 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-10-07 06:39:56.530 [INF] CHDB: Checking for schema update: latest_version=9, db_version=9
2019-10-07 06:39:56.536 [INF] RPCS: password RPC server listening on 127.0.0.1:10009
2019-10-07 06:39:56.536 [INF] RPCS: password RPC server listening on 10.1.1.141:10009
2019-10-07 06:39:56.537 [INF] RPCS: password gRPC proxy started at 10.1.1.141:8082
2019-10-07 06:39:56.537 [INF] RPCS: password gRPC proxy started at 127.0.0.1:8082
2019-10-07 06:39:56.542 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2019-10-07 06:41:50.684 [INF] LNWL: Opened wallet
2019-10-07 06:41:51.137 [INF] LTND: Primary chain is set to: bitcoin
2019-10-07 06:41:51.156 [INF] LTND: Initializing bitcoind backed fee estimator
2019-10-07 06:41:51.157 [INF] LNWL: Started listening for bitcoind block notifications via ZMQ on 10.1.1.141:29001
2019-10-07 06:41:51.157 [INF] LNWL: Started listening for bitcoind transaction notifications via ZMQ on 10.1.1.141:28333
2019-10-07 06:41:51.160 [DBG] LNWL: Using minimum fee rate of 253 sat/kw
2019-10-07 06:41:57.165 [INF] LNWL: The wallet has been unlocked without a time limit
2019-10-07 06:41:57.174 [INF] LTND: LightningWallet opened
2019-10-07 06:41:57.174 [DBG] LNWL: Birthday block has already been verified: height=519682, hash=00000000000000000037b601cbce953b984360b958f8b4e044ec5155fe2745b1
2019-10-07 06:41:57.174 [DBG] LNWL: Waiting for chain backend to sync to tip
2019-10-07 06:41:57.209 [INF] HSWC: Restoring in-memory circuit state from disk
2019-10-07 06:41:58.182 [DBG] LNWL: Chain backend synced to tip!
2019-10-07 06:41:58.575 [INF] HSWC: Payment circuits loaded: num_pending=423, num_open=16
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=566251:963:0, start_htlc_id=5
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=558008:2135:0, start_htlc_id=2
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=532124:1726:0, start_htlc_id=2
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=580139:2541:1, start_htlc_id=0
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=567284:715:0, start_htlc_id=0
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=565286:2043:0, start_htlc_id=16
2019-10-07 06:41:58.728 [INF] HSWC: Trimming open circuits for chan_id=566629:2218:0, start_htlc_id=26
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=578358:390:0, start_htlc_id=290
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=529278:1647:0, start_htlc_id=0
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=586705:1992:0, start_htlc_id=0
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=578180:1001:0, start_htlc_id=0
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=592730:1935:1, start_htlc_id=0
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=545523:741:0, start_htlc_id=7
2019-10-07 06:41:58.729 [INF] HSWC: Trimming open circuits for chan_id=596476:1049:0, start_htlc_id=0
2019-10-07 06:41:58.748 [INF] LTND: Shutdown complete
panic: page 9501 already freed

goroutine 1 [running]:
github.com/coreos/bbolt.(*freelist).free(0x2620080, 0x28732aa, 0x0, 0x6102f000)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/freelist.go:175 +0x448
github.com/coreos/bbolt.(*node).spill(0x248ee40, 0x2b2aec0, 0x2)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/node.go:363 +0x204
github.com/coreos/bbolt.(*node).spill(0x248ea40, 0x2b2aeb0, 0x2)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/node.go:350 +0xa0
github.com/coreos/bbolt.(*node).spill(0x248ea00, 0x2611074, 0x17)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/node.go:350 +0xa0
github.com/coreos/bbolt.(*Bucket).spill(0x2d1c2c0, 0x2d1c200, 0x2611244)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/bucket.go:568 +0x360
github.com/coreos/bbolt.(*Bucket).spill(0x2498a0c, 0xbf5eb94d, 0x82738dc7)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/bucket.go:535 +0x2d8
github.com/coreos/bbolt.(*Tx).Commit(0x2498a00, 0x0, 0x0)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/tx.go:160 +0x90
github.com/coreos/bbolt.(*DB).Update(0x259a3c0, 0x261134c, 0x0, 0x0)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/db.go:695 +0xd4
github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).SetSourceNode(0x24912f0, 0x2498980, 0x46, 0x76eaaa49)
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/channeldb/graph.go:422 +0x5c
github.com/lightningnetwork/lnd.newServer(0x25bbad8, 0x1, 0x1, 0x240d4d0, 0x0, 0x2482930, 0x25ec6a0, 0x0, 0x0, 0x0, ...)
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/server.go:617 +0x19b0
github.com/lightningnetwork/lnd.Main(0x0, 0x0)
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/lnd.go:419 +0xd9c
main.main()
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/cmd/lnd/main.go:14 +0x14

robclark56 commented 5 years ago

Looks similar - except I am running newer versions.

robclark56 commented 5 years ago

Data storage medium: Spinning Hard disk (no SSD). HDD has 600GB free space (of 1TB)

wpaulino commented 5 years ago

@robclark56 could you try running with v0.8.0-beta-rc2 or the master branch? There are some database changes that might allow you to salvage the node. If that doesn't work, then you should use your SCBs following our recovery documentation.

robclark56 commented 5 years ago

Followed the recovery documentation. But did not get too far. I did NOT try v0.8.0-beta-rc2, using the logic of one thing at a time.

lnd is crashing on $ lncli create. Did not even get to try $ lncli restorechanbackup. See below the lncli dialog, and and lnd output.

CLI DIALOG

admin ~  ฿  lncli create
Input wallet password:
Confirm password:

Do you have an existing cipher seed mnemonic you want to use? (Enter y/n): y

Input your 24-word mnemonic separated by spaces: [24 word mnemnonic entered]

Input your cipher seed passphrase (press enter if your seed doesn't have a passphrase):
Input an optional address look-ahead used to scan for used keys (default 2500):

!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!

---------------BEGIN LND CIPHER SEED---------------
[ text deleted]
---------------END LND CIPHER SEED-----------------

!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!

lnd successfully initialized!

LND MESSAGES

bitcoin@RaspiBolt:/home/admin $ lnd
2019-10-08 12:37:53.580 [INF] LTND: Version: 0.7.1-beta commit=v0.7.1-beta, build=production, logging=default
2019-10-08 12:37:53.580 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-10-08 12:37:53.591 [INF] CHDB: Checking for schema update: latest_version=9, db_version=9
2019-10-08 12:37:53.597 [INF] RPCS: password RPC server listening on 127.0.0.1:10009
2019-10-08 12:37:53.598 [INF] RPCS: password RPC server listening on 10.1.1.141:10009
2019-10-08 12:37:53.598 [INF] RPCS: password gRPC proxy started at 10.1.1.141:8082
2019-10-08 12:37:53.598 [INF] RPCS: password gRPC proxy started at 127.0.0.1:8082
2019-10-08 12:37:53.600 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2019-10-08 12:41:08.031 [INF] LNWL: Opened wallet
2019-10-08 12:41:08.032 [INF] LTND: Wallet recovery mode enabled with address lookahead of 2500 addresses
2019-10-08 12:41:08.434 [INF] LTND: Primary chain is set to: bitcoin
2019-10-08 12:41:08.452 [INF] LNWL: Started listening for bitcoind block notifications via ZMQ on 10.1.1.141:29001
2019-10-08 12:41:08.453 [INF] LTND: Initializing bitcoind backed fee estimator
2019-10-08 12:41:08.453 [INF] LNWL: Started listening for bitcoind transaction notifications via ZMQ on 10.1.1.141:28333
2019-10-08 12:41:08.456 [DBG] LNWL: Using minimum fee rate of 253 sat/kw
2019-10-08 12:41:14.478 [INF] LNWL: The wallet has been unlocked without a time limit
2019-10-08 12:41:14.497 [INF] LTND: LightningWallet opened
2019-10-08 12:41:14.497 [DBG] LNWL: Waiting for chain backend to sync to tip
2019-10-08 12:41:14.525 [INF] HSWC: Restoring in-memory circuit state from disk
2019-10-08 12:41:15.510 [DBG] LNWL: Chain backend synced to tip!
2019-10-08 12:41:15.515 [DBG] LNWL: Locating suitable block for birthday 2019-04-25 02:15:05 +0800 AWST between blocks 0-598397
2019-10-08 12:41:15.520 [DBG] LNWL: Checking candidate block: height=299198, hash=00000000000000001918f7e8dc11ef804755d7f8de89d18d394f46728c05158f, timestamp=2014-05-05 20:09:20 +0800 AWST
2019-10-08 12:41:15.525 [DBG] LNWL: Checking candidate block: height=448797, hash=000000000000000001e736a31451132af67205db74ae2e5cf8dc612121b4e2b9, timestamp=2017-01-18 21:50:01 +0800 AWST
2019-10-08 12:41:15.530 [DBG] LNWL: Checking candidate block: height=523597, hash=00000000000000000002a54e8dcecd8bba38059a6028bb0196342970bc1326b4, timestamp=2018-05-21 04:34:47 +0800 AWST
2019-10-08 12:41:15.534 [DBG] LNWL: Checking candidate block: height=560997, hash=000000000000000000064bd48eca9c8fb054a9d666cb978aecdbbbc729042493, timestamp=2019-02-01 10:49:56 +0800 AWST
2019-10-08 12:41:15.539 [DBG] LNWL: Checking candidate block: height=579697, hash=00000000000000000012102b00804ecb21fca1df0e7666b296cbadd774034ebd, timestamp=2019-06-08 01:02:16 +0800 AWST
2019-10-08 12:41:15.544 [DBG] LNWL: Checking candidate block: height=570347, hash=0000000000000000000658ece9be0ad55d64cace67481a87e1ae9ae7a2628403, timestamp=2019-04-06 00:01:02 +0800 AWST
2019-10-08 12:41:15.548 [DBG] LNWL: Checking candidate block: height=575022, hash=0000000000000000000acefdcf8bcd80aae308b56dd9832c276c2a2930f6f258, timestamp=2019-05-08 03:33:43 +0800 AWST
2019-10-08 12:41:15.554 [DBG] LNWL: Checking candidate block: height=572684, hash=0000000000000000000d3b28b10f5383a281bb5c8acf62f07c5069f071d53d22, timestamp=2019-04-22 07:36:58 +0800 AWST
2019-10-08 12:41:15.559 [DBG] LNWL: Checking candidate block: height=573853, hash=000000000000000000100cc66407b7ebf9dfb484ccac32c47d703723f7e21231, timestamp=2019-04-30 07:32:16 +0800 AWST
2019-10-08 12:41:15.564 [DBG] LNWL: Checking candidate block: height=573268, hash=0000000000000000000ee29451f0f96c1c5bbf3aae0e90af1e7cfcdcf133bc96, timestamp=2019-04-26 12:41:36 +0800 AWST
2019-10-08 12:41:15.569 [DBG] LNWL: Checking candidate block: height=572976, hash=0000000000000000001561678d75b89063e6306be7caa6cec1b1548db52edcb3, timestamp=2019-04-24 14:43:49 +0800 AWST
2019-10-08 12:41:15.574 [DBG] LNWL: Checking candidate block: height=573122, hash=000000000000000000191d39dd56e41246cac5c043a31fe528e7d9972ab2943b, timestamp=2019-04-25 12:44:02 +0800 AWST
2019-10-08 12:41:15.579 [DBG] LNWL: Checking candidate block: height=573049, hash=00000000000000000011a4aa538b938cd2e31c0bdb41ce080d22acfe7ad35f33, timestamp=2019-04-25 01:37:29 +0800 AWST
2019-10-08 12:41:15.579 [DBG] LNWL: Found birthday block: height=573049, hash=00000000000000000011a4aa538b938cd2e31c0bdb41ce080d22acfe7ad35f33, timestamp=2019-04-25 01:37:29 +0800 AWST
2019-10-08 12:41:15.586 [INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=2500
2019-10-08 12:41:15.612 [INF] LNWL: Seed birthday surpassed, starting recovery of wallet from height=573050 hash=000000000000000000200157dc7a04eb47b2d9d749e0bdb487cd9b93ff57b2f3 with recovery-window=2500
2019-10-08 12:41:15.987 [INF] HSWC: Payment circuits loaded: num_pending=423, num_open=16
2019-10-08 12:41:16.149 [INF] HSWC: Trimming open circuits for chan_id=566251:963:0, start_htlc_id=5
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=558008:2135:0, start_htlc_id=2
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=532124:1726:0, start_htlc_id=2
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=580139:2541:1, start_htlc_id=0
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=567284:715:0, start_htlc_id=0
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=565286:2043:0, start_htlc_id=16
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=566629:2218:0, start_htlc_id=26
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=578358:390:0, start_htlc_id=290
2019-10-08 12:41:16.150 [INF] HSWC: Trimming open circuits for chan_id=529278:1647:0, start_htlc_id=0
2019-10-08 12:41:16.151 [INF] HSWC: Trimming open circuits for chan_id=586705:1992:0, start_htlc_id=0
2019-10-08 12:41:16.151 [INF] HSWC: Trimming open circuits for chan_id=578180:1001:0, start_htlc_id=0
2019-10-08 12:41:16.151 [INF] HSWC: Trimming open circuits for chan_id=592730:1935:1, start_htlc_id=0
2019-10-08 12:41:16.151 [INF] HSWC: Trimming open circuits for chan_id=545523:741:0, start_htlc_id=7
2019-10-08 12:41:16.151 [INF] HSWC: Trimming open circuits for chan_id=596476:1049:0, start_htlc_id=0
2019-10-08 12:41:16.183 [INF] LTND: Shutdown complete
panic: assertion failed: write: circular dependency occurred

goroutine 1 [running]:
github.com/coreos/bbolt._assert(...)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/db.go:1162
github.com/coreos/bbolt.(*node).write(0x355e280, 0x359f000)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/node.go:226 +0x37c
github.com/coreos/bbolt.(*node).spill(0x355e280, 0x0, 0x0)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/node.go:378 +0x2f4
github.com/coreos/bbolt.(*node).spill(0x355e240, 0x2e9d074, 0x17)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/node.go:350 +0xa0
github.com/coreos/bbolt.(*Bucket).spill(0x350db40, 0x350da00, 0x2e9d244)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/bucket.go:568 +0x360
github.com/coreos/bbolt.(*Bucket).spill(0x2d4798c, 0xbf5f22bb, 0x39eba6cb)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/bucket.go:535 +0x2d8
github.com/coreos/bbolt.(*Tx).Commit(0x2d47980, 0x0, 0x0)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/tx.go:160 +0x90
github.com/coreos/bbolt.(*DB).Update(0x2dc63c0, 0x2e9d34c, 0x0, 0x0)
        /Users/roasbeef/gocode/pkg/mod/github.com/coreos/bbolt@v1.3.2/db.go:695 +0xd4
github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).SetSourceNode(0x3031230, 0x2d47900, 0x46, 0xa28f653b)
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/channeldb/graph.go:422 +0x5c
github.com/lightningnetwork/lnd.newServer(0x2c79a58, 0x1, 0x1, 0x2c0d490, 0x0, 0x2c820e0, 0x2ea4c00, 0x0, 0x0, 0x0, ...)
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/server.go:617 +0x19b0
github.com/lightningnetwork/lnd.Main(0x0, 0x0)
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/lnd.go:419 +0xd9c
main.main()
        /Users/roasbeef/gocode/src/github.com/lightningnetwork/lnd/cmd/lnd/main.go:14 +0x14

wpaulino commented 5 years ago

Can you try with v0.8.0-beta-rc2 to see if you can reproduce?

robclark56 commented 5 years ago

Here is my plan .. see if you agree:

I have a 2nd (different hardware) LND node with no on-chain or off-chain funds. (Just use it to check new FW releases before upgrading my 'main' node that is now sick. It is also running V0.7.1 (Both share the same bitcoind usinq ZMQ.)

On 2nd node: lncli create with my 24 word mnemonic. See if it crashes also. Yes: Upgrade to v0.8.0. Repeat. No crash: Try to restore channels

wpaulino commented 5 years ago

Any updates @robclark56?

robclark56 commented 5 years ago

Some success.

2nd node: Recovery of onchain balance seemed to go OK. (took 48hrs on my very low powered 2nd node)

Main node: Upgraded to V0.8.x. Tried to unlock wallet -> panic. So will now try to restore

Onchain, then
off chain

robclark56 commented 5 years ago

bitcoin@RaspiBolt:~ $ lncli --version
lncli version 0.8.0-beta commit=v0.8.0-beta-rc2

On-chain: recovered OK

Off-chain: Partial success

bitcoin@RaspiBolt:~ $ lncli restorechanbackup --multi_file /tmp/channel.backup
[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: unable to derive shachain root key: unable to derive private key

At this stage about 60% of my pre-crash local off-chain local balance has appeared back in my wallet.

bitcoin@RaspiBolt:~ $ lncli pendingchannels
{
    "total_limbo_balance": "0",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

But I have lost about 40% of my local off-chain balance.

Seems bugs still exit ... but not surviving a power failure, and then not being able to recover properly, is a major concern for me.

guggero commented 5 years ago

About the error you get:

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: unable to derive shachain root key: unable to derive private key

Are you trying to restore the channel backup with a different seed than was used to create the backup? The message indicates either that or that the backup is corrupt...

Backups are strictly tied to the seed they were created for/with. Because the private keys for the channels aren't in the backup and need to be derived from the seed. That's why the backups are encrypted with a key from the seed and therefore can only be opened with a wallet that has been initialized with the proper seed.

robclark56 commented 5 years ago

Oliver,

Thanks for the reply .

I ran that command only once. I did get that errror. But 60% of the channels were recovered OK...so doubt the backup file was created with a different seed.

Corrupt SCB file??? That would say the SCB file was partially corrupt.

I am guessing that it is of no benefit to retry lncli restorechanbackup with the same file a 2nd time.

Suggestions welcome .

thephizzz commented 4 years ago

I am having a similar problem. lnd went down due to power outage. Hung on startup. Had to restore blockchain data. Then - no funds in lnd wallet or channels. When restoring channel.backup, get the following error:

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: chacha20poly1305: message authentication failed

I have no idea where that text 'chacha20poly1305' comes from.

wpaulino commented 4 years ago

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: chacha20poly1305: message authentication failed

@thephizzz you can only use channel backups with the same seed with which they were created.

guggero commented 4 years ago

@robclark56 I think I found the problem for the unable to derive shachain root key: unable to derive private key error. See #3881. Am I correct in assuming that at least one of your channels was opened with a version prior to v0.4.0-beta?

robclark56 commented 4 years ago

To be honest .... can't remember when V0.4.0 was released.

But I started using that node April 2017

guggero commented 4 years ago

v0.4.0-beta was released on March 15, 2018. So that could definitely be it. You still weren't able to restore the channel.backup file and are still getting the unable to derive shachain root key error, is that correct?

robclark56 commented 4 years ago

Correct- I was never able to recover 40% of my offchain funds.

guggero commented 4 years ago

These channels are really ancient. Respect for keeping them for so long! Unfortunately these old channels aren't supported in SCB. There was a big change in key derivation in v0.4-beta and it was explicitly stated that channels should be closed and reopened: https://github.com/lightningnetwork/lnd/releases/tag/v0.4-beta

This means, to get the remaining 40% of your channel funds back, there will be some manual work necessary.

Can you please try the following steps:

Create a filtered channel.backup file that contains only the channels that are still not recovered (just to speed up the following steps) by using the filterbackup command of https://github.com/guggero/chantools
Run the fixoldbackup command on the filtered channel.backup.
Follow this guide: https://medium.com/@guggero/did-you-lose-funds-on-the-lightning-network-because-of-a-disk-crash-8971b6a92494 but instead of creating a fake channel.backup use the fixed backup-fixed-xyz.backup file from the previous command.
If you are lucky and the remote nodes are still around, you should be able to ask them to force-close and then can recover the private key that is necessary to sweep the funds.

Feel free to contact me on Slack if you need assistance.

I'm closing this issue for now as the original report was caused by data corruption of the channel.db and the follow-up error was caused by old, unsupported channels. For discussion around the manual recovery of pre-0.4 channels, please also follow #3881.

lightningnetwork / lnd