search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.67k stars 2.07k forks source link

Crash after ~13 days offline #6276

Closed PeerZetZzZzZ closed 2 years ago

PeerZetZzZzZ commented 2 years ago

Background

My raspiblitz was offline for about 8 days. Restarted it and got this error:

Feb 18 14:16:47 raspberrypi blitz.systemd.sh[10379]: /home/admin/config.scripts/blitz.systemd.sh: line 18: /home/admin/systemd.lightning.log: Permission denied
Feb 18 14:16:47 raspberrypi systemd[1]: Started LND Lightning Daemon.
Feb 18 14:17:11 raspberrypi lnd[10381]: panic: runtime error: slice bounds out of range [::3149120636] with length 2147483647
Feb 18 14:17:11 raspberrypi lnd[10381]: goroutine 1 [running]:
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.unsafeByteSlice(...)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/unsafe.go:27
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*leafPageElement).key(...)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/page.go:126
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).nsearch.func2(0x1a49, 0x1)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:328 +0xb8
Feb 18 14:17:11 raspberrypi lnd[10381]: sort.Search(0xd24c, 0x4000491a00, 0x60)
Feb 18 14:17:11 raspberrypi lnd[10381]:         sort/search.go:66 +0x4c
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).nsearch(0x4000491e68, 0x7f22ce0680, 0x8, 0x8)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:327 +0xb8
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).search(0x4000491e68, 0x7f22ce0680, 0x8, 0x8, 0x1c9bc)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:257 +0x148
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).searchPage(0x4000491e68, 0x7f22ce0680, 0x8, 0x8, 0x7f396e4000)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:308 +0x10c
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).search(0x4000491e68, 0x7f22ce0680, 0x8, 0x8, 0x3257d)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:265 +0x11c
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).searchPage(0x4000491e68, 0x7f22ce0680, 0x8, 0x8, 0x7f0d25c000)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:308 +0x10c
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).search(0x4000491e68, 0x7f22ce0680, 0x8, 0x8, 0x60f5)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:265 +0x11c
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Cursor).seek(0x4000491e68, 0x7f22ce0680, 0x8, 0x8, 0x0, 0x8, 0x4000491e38, 0x5b786c, 0x4000491ed8, 0x5b71e4, ...)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:159 +0x68
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Bucket).Get(0x400082da40, 0x7f22ce0680, 0x8, 0x8, 0x28, 0xe16bc0, 0x4000491ed8)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:262 +0x84
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/btcsuite/btcwallet/walletdb/bdb.(*bucket).Get(0x400082da40, 0x7f22ce0680, 0x8, 0x8, 0x0, 0x0, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/btcsuite/btcwallet/walletdb@v1.3.6-0.20210803004036-eebed51155ec/bdb/db.go:211 +0x40
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/channeldb.fetchChanEdgeInfo(0x7f87185de0, 0x400082da40, 0x7f22ce0680, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/channeldb/graph.go:3864 +0x6c
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).ChannelView.func1.1(0x7f22ce065c, 0x24, 0x24, 0x7f22ce0680, 0x8, 0x8, 0x8, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/channeldb/graph.go:3364 +0x118
Feb 18 14:17:11 raspberrypi lnd[10381]: go.etcd.io/bbolt.(*Bucket).ForEach(0x400082da00, 0x40000f5a40, 0x40004928a8, 0x5dda68)
Feb 18 14:17:11 raspberrypi lnd[10381]:         go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:390 +0xe0
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/btcsuite/btcwallet/walletdb/bdb.(*bucket).ForEach(0x400082da00, 0x40000f5a40, 0xa, 0xa)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/btcsuite/btcwallet/walletdb@v1.3.6-0.20210803004036-eebed51155ec/bdb/db.go:190 +0x30
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).ChannelView.func1(0x1263508, 0x400000e4d0, 0x400000e4d0, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/channeldb/graph.go:3355 +0x188
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/btcsuite/btcwallet/walletdb/bdb.(*db).View(0x40004f6200, 0x4000bb37c0, 0x4000bb37d0, 0x0, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/btcsuite/btcwallet/walletdb@v1.3.6-0.20210803004036-eebed51155ec/bdb/db.go:392 +0xa0
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/channeldb.(*DB).View(0x40000b4980, 0x4000bb37c0, 0x4000bb37d0, 0x7f87185f18, 0x40000b4980)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/channeldb/db.go:240 +0x6c
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/kvdb.View(0x1276290, 0x40000b4980, 0x4000bb37c0, 0x4000bb37d0, 0x0, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/kvdb@v1.0.0/interface.go:34 +0x68
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).ChannelView(0x4000450000, 0x40000eade0, 0x0, 0x0, 0x0, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/channeldb/graph.go:3335 +0xd0
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd/routing.(*ChannelRouter).Start(0x4000642140, 0x4000bb3160, 0x0)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/routing/router.go:548 +0x5e0
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd.(*server).Start.func1()
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/server.go:1637 +0xd04
Feb 18 14:17:11 raspberrypi lnd[10381]: sync.(*Once).doSlow(0x40001c2288, 0x4000492f88)
Feb 18 14:17:11 raspberrypi lnd[10381]:         sync/once.go:68 +0x11c
Feb 18 14:17:11 raspberrypi lnd[10381]: sync.(*Once).Do(...)
Feb 18 14:17:11 raspberrypi lnd[10381]:         sync/once.go:59
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd.(*server).Start(0x40001c2280, 0xfa39e1, 0x2e)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/server.go:1502 +0xc0
Feb 18 14:17:11 raspberrypi lnd[10381]: github.com/lightningnetwork/lnd.Main(0x40002fca00, 0x0, 0x0, 0x0, 0x0, 0x40001a2840, 0x4000182480, 0x40001824e0, 0x40001826c0, 0x0, ...)
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/lnd.go:992 +0x1d08
Feb 18 14:17:11 raspberrypi lnd[10381]: main.main()
Feb 18 14:17:11 raspberrypi lnd[10381]:         github.com/lightningnetwork/lnd/cmd/lnd/main.go:36 +0x198
Feb 18 14:17:11 raspberrypi systemd[1]: lnd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 18 14:17:11 raspberrypi systemd[1]: lnd.service: Failed with result 'exit-code'.

Your environment

Expected behaviour

Expected normal sync. I have 2 channels open for my node. channel.db has about 1.2gb. I've tried multiple times to fix it with running different versions of lnd (0.12, 0.13, 0.14.0), did multiple restores of LND on raspiblitz and still nothing

PeerZetZzZzZ commented 2 years ago

Currently closing - the problem is still present however the description I put can be misleading - I get also some other wired errors ("page" errors). I must try few more times to fix it by myself.