[ERR] LTND: unable to open channeldb: unable to update edge indexes: unable to fetch node:

bhstauffer commented 5 years ago

Background

The Lightning Daemon crashed on its own with no user input. I get the following errors in the log:

2019-02-19 16:50:53.747 [ERR] DISC: Unable to determine if node 022c699df736064b51a33017abfc4d577d133f7124ac117d3d9f9633b6297a3b6a is advertised: edge not found 2019-02-19 16:51:05.989 [ERR] DISC: Unable to determine if node 039f01ad62e5208940faff11d0bbc997582eafad7642aaf53de6a5f6551ab73400 is advertised: edge not found 2019-02-19 16:51:10.709 [ERR] DISC: Unable to determine if node 03465f79f840a1169e2b4cb42d684e1723af861b8a82336aa72f9766643a6ef4e4 is advertised: edge not found 2019-02-19 16:51:11.866 [ERR] DISC: Unable to determine if node 03472d3e488f6bd2e33b4992ce75c211393ba1ac66a4270cc26502a6fbac646c99 is advertised: edge not found 2019-02-19 17:14:49.556 [ERR] LTND: unable to open channeldb: unable to update edge indexes: unable to fetch node: 031cc35d487b8fae5f24df919e8ce5ad2a6bbf489f67694e5a929f10eee0e000a5, unable to find node

When I try to restart LND, the daemon immediately crashes and I get the following in the log:

2019-02-19 17:17:39.266 [INF] LTND: Version: 0.5.2-beta commit=v0.5.2-beta, build=production, logging=default 2019-02-19 17:17:39.267 [INF] LTND: Active chain: Bitcoin (network=mainnet) 2019-02-19 17:17:39.271 [INF] CHDB: Checking for schema update: latest_version=7, db_version=0 2019-02-19 17:17:39.271 [INF] CHDB: Performing database schema migration 2019-02-19 17:17:39.272 [INF] CHDB: Applying migration #1 2019-02-19 17:17:39.273 [INF] CHDB: Populating new node update index bucket 2019-02-19 17:17:39.373 [INF] CHDB: Populating new edge update index bucket 2019-02-19 17:17:39.381 [INF] CHDB: Unable to apply migration #1 2019-02-19 17:17:39.398 [ERR] LTND: unable to open channeldb: unable to update edge indexes: unable to fetch node: 031cc35d487b8fae5f24df919e8ce5ad2a6bbf489f67694e5a929f10eee0e000a5, unable to find node

It looks like a database migration is happening? I had already updated to version 0.5.2-beta from 0.5.1-beta when 0.5.2 was released and did not have this problem. Perhaps it is crashing because my node cannot connect to node 031cc35d487b8fae5f24df919e8ce5ad2a6bbf489f67694e5a929f10eee0e000a5. I don't think being unable to connect should cause a shutdown.

Your environment

I am using lnd version 0.5.2-beta, and bitcoind 0.17.1. I have tested this error on an Arch Linux desktop (kernel 4.20.10-arch1-1-ARCH) and a raspberry pi (kernel 4.19.23-1-ARCH). The same error ocurrs when I downgrade to 0.5.1-beta.

Steps to reproduce

run lnd

Expected behaviour

lnd should await the 'lncli unlock' command to open the wallet.

Actual behaviour

LND shuts down with the above error log.

Roasbeef commented 5 years ago

It looks like the migration is failing possibly due to a bug that was present in the master version of the prior release, but was fixed before the actual release as cut. It isn't due to it not being able to connect to a node.

bhstauffer commented 5 years ago

Is there a version I should run that would fix the migration, or is the channeldb unsalvageable at this point?

Roasbeef commented 5 years ago

Those first few errors are fine, just an issue with it trying to create invoices as you don't have the proper node state on disk.

You should be able to boot up with this patch:

diff --git a/channeldb/migrations.go b/channeldb/migrations.go
index f86e416b9..72a71c970 100644
--- a/channeldb/migrations.go
+++ b/channeldb/migrations.go
@@ -5,6 +5,7 @@ import (
    "crypto/sha256"
    "encoding/binary"
    "fmt"
+   "strings"

    "github.com/coreos/bbolt"
    "github.com/lightningnetwork/lnd/lnwire"
@@ -346,6 +347,10 @@ func migrateEdgePolicies(tx *bbolt.Tx) error {
            return nil
        }

+       if strings.Contains(err.Error(), "to fetch node") {
+           return nil
+       }
+
        return err
    }

bhstauffer commented 5 years ago

This patch has no effect. I still get the same error messages.

jfburdet commented 5 years ago

I've got the same issue since one week. I hope my channels will not be all closed. What can I do ?

$ lnd -d debug
Attempting automatic RPC configuration to bitcoind
Automatically obtained bitcoind's RPC credentials
2019-02-23 18:02:49.638 [INF] LTND: Version: 0.5.1-beta commit=v0.5.1-beta-728-g44b8cd6699db84d9d163a31efc038571afe6d2be, build=production, logging=default
2019-02-23 18:02:49.639 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-02-23 18:02:49.669 [INF] CHDB: Checking for schema update: latest_version=7, db_version=0
2019-02-23 18:02:49.669 [INF] CHDB: Performing database schema migration
2019-02-23 18:02:49.669 [INF] CHDB: Applying migration #1
2019-02-23 18:02:49.669 [INF] CHDB: Populating new node update index bucket
2019-02-23 18:02:49.691 [INF] CHDB: Populating new edge update index bucket
2019-02-23 18:02:49.702 [INF] CHDB: Unable to apply migration #1
2019-02-23 18:02:49.742 [ERR] LTND: unable to open channeldb: unable to update edge indexes: EOF
2019-02-23 18:02:49.742 [INF] LTND: Shutdown complete
unable to update edge indexes: EOF

Edit : I just realised that I had faulty memory ... so I believe it is the cause of corrupted file. Anyway, lnd should be able to discard bad data and start anyway.

wpaulino commented 5 years ago

It looks like a database migration is happening? I had already updated to version 0.5.2-beta from 0.5.1-beta when 0.5.2 was released and did not have this problem.

@bhstauffer do you remember what lnd version this node was created with? This migration is from v0.5, so if your node was created after that then this would point to some sort of database corruption.

bhstauffer commented 5 years ago

I remember initializing my node at version 0.4. I don't recall the subversion. When I upgraded to 0.5.1-beta, there was a successfull database migration, and I had no issues. I then went to 0.5.2-beta. No migrations or issues for that change. Then about two weeks ago LND crashed with the above log, and I can't restart it. I assume my channel database got corrupted somehow. I'd like to be able to salvage or at least close my channels if possible.

jfburdet commented 5 years ago

Hi folks, A bit of followup : I was able to dig into my backup (thanks restic and b2) ... and after a few tries, I was able to feed some old channel.db and sphinxreplay.db files that lnd was happy with. My node is back up and not too much channels where closed

wpaulino commented 5 years ago

I assume my channel database got corrupted somehow.

@bhstauffer yeah that seems to be the case.

I'd like to be able to salvage or at least close my channels if possible.

There won't be an easy way of recovering your channel funds until #2313 is in. The most you can do for now is just recover your on-chain funds with your seed.

bhstauffer commented 5 years ago

Alright. It's reassuring that a fix is in the pipeline. Thanks for the info.

lightningnetwork / lnd