search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.67k stars 2.07k forks source link

panic: runtime error: invalid memory address or nil pointer dereference #6541

Closed JanStr95 closed 2 years ago

JanStr95 commented 2 years ago

Background

LND-Process is shutting down

Your environment

Steps to reproduce

During operation, LND went offline and did not restart again. Flashing the SK card and updating to 0.14.3-beta didn't help either.

Expected behaviour

Start LND

Actual behaviour

LND did not start status: ● lnd.service - Lightning Network Daemon Loaded: loaded (/etc/systemd/system/lnd.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Sat 2022-05-14 22:45:19 BST; 51s ago Process: 232802 ExecStartPre=/home/admin/config.scripts/lnd.check.sh prestart mainnet (code=exited, status=0/SUCCESS) Process: 232956 ExecStart=/usr/local/bin/lnd --configfile=/home/bitcoin/.lnd/lnd.conf (code=exited, status=2) Main PID: 232956 (code=exited, status=2) CPU: 2.237s

May 14 22:45:19 raspberrypi systemd[1]: lnd.service: Failed with result 'exit-code'. May 14 22:45:19 raspberrypi systemd[1]: Failed to start Lightning Network Daemon. May 14 22:45:19 raspberrypi systemd[1]: lnd.service: Consumed 2.237s CPU time.

-------- LOG -------- 2022-05-14 21:17:39.464 [INF] LTND: Version: 0.14.2-beta commit=v0.14.2-beta, build=production, logging=default, debuglevel=debug 2022-05-14 21:17:39.464 [INF] LTND: Active chain: Bitcoin (network=mainnet) 2022-05-14 21:17:39.467 [DBG] WLKT: WalletKit RPC server successfully registered with root gRPC server 2022-05-14 21:17:39.467 [DBG] RRPC: Router RPC server successfully register with root gRPC server 2022-05-14 21:17:39.467 [DBG] ARPC: Autopilot RPC server successfully register with root gRPC server 2022-05-14 21:17:39.467 [DBG] VRPC: Versioner RPC server successfully registered with root gRPC server 2022-05-14 21:17:39.467 [DBG] SGNR: Signer RPC server successfully register with root gRPC server 2022-05-14 21:17:39.468 [DBG] NTFR: ChainNotifier RPC server successfully register with root gRPC server 2022-05-14 21:17:39.468 [DBG] IRPC: Invoices RPC server successfully registered with root gRPC server 2022-05-14 21:17:39.468 [INF] RPCS: RPC server listening on 0.0.0.0:10009 2022-05-14 21:17:39.476 [DBG] WLKT: WalletKit REST server successfully registered with root REST server 2022-05-14 21:17:39.476 [DBG] RRPC: Router REST server successfully registered with root REST server 2022-05-14 21:17:39.477 [DBG] ARPC: Autopilot REST server successfully registered with root REST server 2022-05-14 21:17:39.477 [DBG] VRPC: Versioner REST server successfully registered with root REST server 2022-05-14 21:17:39.477 [DBG] SGNR: Signer REST server successfully registered with root REST server 2022-05-14 21:17:39.478 [DBG] NTFR: ChainNotifier REST server successfully registered with root REST server 2022-05-14 21:17:39.478 [DBG] IRPC: Invoices REST server successfully registered with root REST server 2022-05-14 21:17:39.479 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080 2022-05-14 21:17:39.479 [INF] LTND: Opening the main database, this might take a few minutes... 2022-05-14 21:17:39.484 [INF] LTND: Opening bbolt database, sync_freelist=true, auto_compact=true 2022-05-14 21:17:39.484 [INF] CHDB: Compacting database file at /home/bitcoin/.lnd/data/graph/mainnet/channel.db 2022-05-14 21:17:39.484 [DBG] CHDB: Free disk space on compaction destination file system: 358233444352 bytes 2022-05-14 21:17:39.500 [DBG] CHDB: Compacting top level bucket 'byte' 2022-05-14 21:17:39.501 [DBG] CHDB: Compacting top level bucket 'channelOpeningState' 2022-05-14 21:17:39.501 [DBG] CHDB: Compacting top level bucket 'circuit-adds' 2022-05-14 21:17:39.502 [DBG] CHDB: Compacting top level bucket 'circuit-fwd-log' 2022-05-14 21:17:39.502 [DBG] CHDB: Compacting top level bucket 'circuit-keystones' 2022-05-14 21:17:39.502 [DBG] CHDB: Compacting top level bucket 'close-summaries' 2022-05-14 21:17:39.502 [DBG] CHDB: Compacting top level bucket 'closed-chan-bucket' 2022-05-14 21:17:39.502 [DBG] CHDB: Compacting top level bucket 'confirm-hints' 2022-05-14 21:17:39.503 [DBG] CHDB: Compacting top level bucket 'edge-index' 2022-05-14 21:17:39.503 [DBG] CHDB: Compacting top level bucket 'fwd-packages' 2022-05-14 21:17:39.503 [DBG] CHDB: Compacting top level bucket 'graph-edge' 2022-05-14 21:17:41.773 [INF] LTND: Shutdown complete

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xd18834]

goroutine 1 [running]: go.etcd.io/bbolt.(Bucket).Sequence(...) go.etcd.io/bbolt@v1.3.6/bucket.go:340 github.com/lightningnetwork/lnd/kvdb.(compacter).walkBucket.func1({0x7ea7b4dbfe, 0x5, 0x5}, {0x0, 0x0, 0x0}) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:277 +0xe4 go.etcd.io/bbolt.(Bucket).ForEach(0x400070bb80, 0x400094c750) go.etcd.io/bbolt@v1.3.6/bucket.go:390 +0x10c github.com/lightningnetwork/lnd/kvdb.(compacter).walkBucket(0x40005f0eb0, 0x400070bb80, {0x0, 0x0, 0x0}, {0x7eb31334b3, 0xa, 0xa}, {0x0, 0x0, ...}, ...) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:273 +0x184 github.com/lightningnetwork/lnd/kvdb.(compacter).walk.func1.1({0x7eb31334b3, 0xa, 0xa}, 0x400070bb80) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:226 +0x154 go.etcd.io/bbolt.(Tx).ForEach.func1({0x7eb31334b3, 0xa, 0xa}, {0x0, 0x0, 0x0}) go.etcd.io/bbolt@v1.3.6/tx.go:129 +0x7c go.etcd.io/bbolt.(Bucket).ForEach(0x40008640f8, 0x400094c930) go.etcd.io/bbolt@v1.3.6/bucket.go:390 +0x10c go.etcd.io/bbolt.(Tx).ForEach(0x40008640e0, 0x40005f0980) go.etcd.io/bbolt@v1.3.6/tx.go:128 +0x58 github.com/lightningnetwork/lnd/kvdb.(compacter).walk.func1(0x40008640e0) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:220 +0x54 go.etcd.io/bbolt.(DB).View(0x4000386480, 0x40005f0a40) go.etcd.io/bbolt@v1.3.6/db.go:772 +0x84 github.com/lightningnetwork/lnd/kvdb.(compacter).walk(0x40005f0eb0, 0x4000386480, 0x40005f0ab8) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:219 +0x54 github.com/lightningnetwork/lnd/kvdb.(compacter).compact(0x40005f0eb0, 0x4000386900, 0x4000386480) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:147 +0xd4 github.com/lightningnetwork/lnd/kvdb.(compacter).execute(0x400094ceb0) github.com/lightningnetwork/lnd/kvdb@v1.3.0/bolt_compact.go:117 +0x500 github.com/lightningnetwork/lnd/kvdb.compactAndSwap(0x40005f12d0) github.com/lightningnetwork/lnd/kvdb@v1.3.0/backend.go:179 +0x71c github.com/lightningnetwork/lnd/kvdb.GetBoltBackend(0x400094d2d0) github.com/lightningnetwork/lnd/kvdb@v1.3.0/backend.go:101 +0x98 github.com/lightningnetwork/lnd/lncfg.(DB).GetBackends(0x40000b2280, {0x1a54510, 0x40000b2900}, {0x400043b7a0, 0x25}, {0x400043ab70, 0x2d}, {0x400088a3c0, 0x32}, 0x1, ...) github.com/lightningnetwork/lnd/lncfg/db.go:392 +0x20b8 github.com/lightningnetwork/lnd.(*DefaultDatabaseBuilder).BuildDatabase(0x400026f110, {0x1a54510, 0x40000b2900}) github.com/lightningnetwork/lnd/config_builder.go:810 +0x324 github.com/lightningnetwork/lnd.Main(0x40000f2f00, {{0x0, 0x0, 0x0}}, 0x4000343020, {0x4000342ae0, 0x4000096660, 0x40000966c0, 0x4000096720, {0x0}}) github.com/lightningnetwork/lnd/lnd.go:353 +0x15d0 main.main() github.com/lightningnetwork/lnd/cmd/lnd/main.go:38 +0x258

alexbosworth commented 2 years ago

OS is 32 bit or 64 bit?

JanStr95 commented 2 years ago

OS is 32 bit or 64 bit?

64 bit channel.db was "compressed" from 2.9G to 2.9G

guggero commented 2 years ago

What do you mean by "compressed"? Did you run the compaction manually with an external tool? Because it looks like the automatic compaction of lnd is failing here. Which unfortunately does sound like data corruption of the channel.db file.

JanStr95 commented 2 years ago

I performed the compression in the raspiblitz menu after LND failed without warning during operation.

JanStr95 commented 2 years ago

What do you mean by "compressed"? Did you run the compaction manually with an external tool? Because it looks like the automatic compaction of lnd is failing here. Which unfortunately does sound like data corruption of the channel.db file.

I hope that there is a way to repair the channel.db to at least get back to the sats?!

guggero commented 2 years ago

I performed the compression in the raspiblitz menu after LND failed without warning during operation.

I'm not sure what that command in the menu does in the background, but I assume it's the compaction. Did that run successfully? And did you run that before you got that restart problem or afterwards to try and fix it?

I hope that there is a way to repair the channel.db to at least get back to the sats?!

It depends on the type of data corruption. Often when the data gets corrupted because of an unclean shutdown (power outage or unplugging the power supply by accident), there is no way to repair the DB.

JanStr95 commented 2 years ago

Exactly, I mean the compaction.

The error occurred without warning or indication. She was walking normally. There were no power outages or anything else.

so.. Last week when I looked at my node I saw that LND is not working.

Unfortunately, that didn't bring any improvement either

That didn't work either.

guggero commented 2 years ago

It looks like the corruption might be in the network graph data which can be re-synced from other nodes. So you can try running chantools dropchannelgraph, though that is risky and not well tested. The alternative is to run through the recovery process by using the seed and SCB file (channel.backup).