search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.67k stars 2.07k forks source link

Error after start lnd #5105

Closed zagota closed 3 years ago

zagota commented 3 years ago

Background

get the following error when starting lnd. What can I do?

Your environment

Steps to reproduce

Starting lnd

/usr/local/bin/lnd
2021-03-15 18:19:13.866 [INF] LTND: Version: 0.12.1-beta commit=v0.12.1-beta, build=production, logging=default, debuglevel=info
2021-03-15 18:19:13.866 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2021-03-15 18:19:13.866 [INF] LTND: Opening the main database, this might take a few minutes...
2021-03-15 18:19:13.866 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
unexpected fault address 0x7feb9fffe000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7feb9fffe000 pc=0x7c8262]

goroutine 1 [running]:
runtime.throw(0x143bcb3, 0x5)
    runtime/panic.go:1116 +0x72 fp=0xc00002c590 sp=0xc00002c560 pc=0x437692
runtime.sigpanic()
    runtime/signal_unix.go:749 +0x405 fp=0xc00002c5c0 sp=0xc00002c590 pc=0x44dde5
go.etcd.io/bbolt.(*Tx).checkBucket.func1(0x7feb9fffe000, 0x3)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:466 +0x62 fp=0xc00002c688 sp=0xc00002c5c0 pc=0x7c8262
go.etcd.io/bbolt.(*Tx).forEachPage(0xc00003c1c0, 0xb83ed, 0x3, 0xc00002c7f8)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:626 +0x89 fp=0xc00002c6d0 sp=0xc00002c688 pc=0x7c6b29
go.etcd.io/bbolt.(*Tx).forEachPage(0xc00003c1c0, 0xd16a, 0x2, 0xc00002c7f8)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:632 +0xe5 fp=0xc00002c718 sp=0xc00002c6d0 pc=0x7c6b85
go.etcd.io/bbolt.(*Tx).forEachPage(0xc00003c1c0, 0x15c1a, 0x1, 0xc00002c7f8)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:632 +0xe5 fp=0xc00002c760 sp=0xc00002c718 pc=0x7c6b85
go.etcd.io/bbolt.(*Tx).forEachPage(0xc00003c1c0, 0x7fb2, 0x0, 0xc00002c7f8)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:632 +0xe5 fp=0xc00002c7a8 sp=0xc00002c760 pc=0x7c6b85
go.etcd.io/bbolt.(*Tx).checkBucket(0xc00003c1c0, 0xc0005940c0, 0xc00002cbf8, 0xc00002cbc8, 0xc0000940c0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:465 +0xd4 fp=0xc00002c838 sp=0xc00002c7a8 pc=0x7c6254
go.etcd.io/bbolt.(*Tx).checkBucket.func2(0x7feaf2edc78f, 0xa, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:490 +0xc5 fp=0xc00002c890 sp=0xc00002c838 pc=0x7c87a5
go.etcd.io/bbolt.(*Bucket).ForEach(0xc000594b00, 0xc00002c920, 0x0, 0xc00002c950)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:390 +0xfe fp=0xc00002c900 sp=0xc00002c890 pc=0x7b397e
go.etcd.io/bbolt.(*Tx).checkBucket(0xc00003c1c0, 0xc000594b00, 0xc00002cbf8, 0xc00002cbc8, 0xc0000940c0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:488 +0x146 fp=0xc00002c990 sp=0xc00002c900 pc=0x7c62c6
go.etcd.io/bbolt.(*Tx).checkBucket.func2(0x7feafa542460, 0xa, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:490 +0xc5 fp=0xc00002c9e8 sp=0xc00002c990 pc=0x7c87a5
go.etcd.io/bbolt.(*Bucket).ForEach(0xc00003c1d8, 0xc00002ca78, 0x0, 0xc00002caa8)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:390 +0xfe fp=0xc00002ca58 sp=0xc00002c9e8 pc=0x7b397e
go.etcd.io/bbolt.(*Tx).checkBucket(0xc00003c1c0, 0xc00003c1d8, 0xc00002cbf8, 0xc00002cbc8, 0xc0000940c0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:488 +0x146 fp=0xc00002cae8 sp=0xc00002ca58 pc=0x7c62c6
go.etcd.io/bbolt.(*DB).freepages(0xc0000c2200, 0x0, 0x0, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1006 +0x210 fp=0xc00002cce0 sp=0xc00002cae8 pc=0x7bb010
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:313 +0x111 fp=0xc00002cd18 sp=0xc00002cce0 pc=0x7c7971
sync.(*Once).doSlow(0xc0000c2368, 0xc00002cd78)
    sync/once.go:66 +0xec fp=0xc00002cd68 sp=0xc00002cd18 pc=0x486f4c
sync.(*Once).Do(...)
    sync/once.go:57
go.etcd.io/bbolt.(*DB).loadFreelist(0xc0000c2200)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:309 +0x6a fp=0xc00002cd98 sp=0xc00002cd68 pc=0x7b80ea
go.etcd.io/bbolt.Open(0xc0000c6900, 0x2d, 0x180, 0xc00002ced0, 0x0, 0x203000, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:286 +0x36f fp=0xc00002ce80 sp=0xc00002cd98 pc=0x7b7baf
github.com/btcsuite/btcwallet/walletdb/bdb.openDB(0xc0000c6900, 0x2d, 0xc000540001, 0xdf8475800, 0x3, 0xc0000c6900, 0x2d, 0x1)
    github.com/btcsuite/btcwallet/walletdb@v1.3.4/bdb/db.go:386 +0xbc fp=0xc00002cf30 sp=0xc00002ce80 pc=0x7ce35c
github.com/btcsuite/btcwallet/walletdb/bdb.openDBDriver(0xc00054e0c0, 0x3, 0x3, 0x3, 0xc00009a088, 0x1, 0xc000544080)
    github.com/btcsuite/btcwallet/walletdb@v1.3.4/bdb/driver.go:61 +0xbd fp=0xc00002cf98 sp=0xc00002cf30 pc=0x7cecfd
github.com/btcsuite/btcwallet/walletdb.Open(0x143a76c, 0x3, 0xc00054e0c0, 0x3, 0x3, 0xc0000c68d0, 0x22, 0x0, 0x0)
    github.com/btcsuite/btcwallet/walletdb@v1.3.4/interface.go:363 +0x88 fp=0xc00002cfe0 sp=0xc00002cf98 pc=0x7aecc8
github.com/lightningnetwork/lnd/channeldb/kvdb.GetBoltBackend(0xc00002d0d8, 0x1, 0xc0000c68d0, 0x22, 0xc00002d110)
    github.com/lightningnetwork/lnd/channeldb/kvdb/backend.go:100 +0x1c5 fp=0xc00002d078 sp=0xc00002cfe0 pc=0x7da1c5
github.com/lightningnetwork/lnd/lncfg.(*DB).GetBackends(0xc000264060, 0x17221a0, 0xc0000be000, 0xc0000c68d0, 0x22, 0x143e904, 0x7, 0x203000, 0x203000, 0x7feb07f11118)
    github.com/lightningnetwork/lnd/lncfg/db.go:101 +0x11f fp=0xc00002d120 sp=0xc00002d078 pc=0xc8be9f
github.com/lightningnetwork/lnd.initializeDatabases(0x17221a0, 0xc0000be000, 0xc0000f6000, 0xc0000be000, 0xc000546060, 0x2, 0x0, 0x0)
    github.com/lightningnetwork/lnd/lnd.go:1358 +0x139 fp=0xc00002d2a8 sp=0xc00002d120 pc=0x1085ed9
github.com/lightningnetwork/lnd.Main(0xc0000f6000, 0x0, 0x0, 0xc000094360, 0x0, 0x0)
    github.com/lightningnetwork/lnd/lnd.go:261 +0x465 fp=0xc00002df20 sp=0xc00002d2a8 pc=0x107de85
main.main()
    github.com/lightningnetwork/lnd/cmd/lnd/main.go:35 +0x16c fp=0xc00002df88 sp=0xc00002df20 pc=0x10e91cc
runtime.main()
    runtime/proc.go:204 +0x209 fp=0xc00002dfe0 sp=0xc00002df88 pc=0x439e89
runtime.goexit()
    runtime/asm_amd64.s:1374 +0x1 fp=0xc00002dfe8 sp=0xc00002dfe0 pc=0x46fa61

goroutine 9 [select]:
io.(*pipe).Read(0xc0000770e0, 0xc000542000, 0x1000, 0x1000, 0xc000346da8, 0x4d8a45, 0xc000077080)
    io/pipe.go:57 +0xe7
io.(*PipeReader).Read(0xc0000ba340, 0xc000542000, 0x1000, 0x1000, 0x64, 0xc000077098, 0x4dcf00)
    io/pipe.go:134 +0x4c
bufio.(*Reader).fill(0xc000346ef8)
    bufio/bufio.go:101 +0x105
bufio.(*Reader).ReadSlice(0xc000346ef8, 0xa, 0x1, 0x0, 0x0, 0x0, 0x0)
    bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc000346ef8, 0x1e1503e, 0x1, 0x1, 0x1, 0x0, 0x0)
    bufio/bufio.go:389 +0x34
github.com/jrick/logrotate/rotator.(*Rotator).Run(0xc0002c6b00, 0x1702180, 0xc0000ba340, 0x0, 0x0)
    github.com/jrick/logrotate@v1.0.0/rotator/rotator.go:100 +0x93
github.com/lightningnetwork/lnd/build.(*RotatingLogWriter).InitLogRotator.func1(0xc000264000, 0xc0000ba340)
    github.com/lightningnetwork/lnd/build/logrotator.go:83 +0x49
created by github.com/lightningnetwork/lnd/build.(*RotatingLogWriter).InitLogRotator
    github.com/lightningnetwork/lnd/build/logrotator.go:82 +0x325

goroutine 49 [syscall]:
os/signal.signal_recv(0x0)
    runtime/sigqueue.go:147 +0x9d
os/signal.loop()
    os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
    os/signal/signal.go:150 +0x45

goroutine 50 [select]:
github.com/lightningnetwork/lnd/signal.mainInterruptHandler()
    github.com/lightningnetwork/lnd/signal/signal.go:83 +0x11b
created by github.com/lightningnetwork/lnd/signal.Intercept
    github.com/lightningnetwork/lnd/signal/signal.go:49 +0xf2

goroutine 51 [chan receive]:
go.etcd.io/bbolt.(*DB).freepages.func2(0xc0000940c0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1002 +0x45
created by go.etcd.io/bbolt.(*DB).freepages
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1001 +0x1cd

Expected behaviour

Tell us what should happen

Actual behaviour

Tell us what happens instead

wpaulino commented 3 years ago

Looks like your database might've become corrupted. Could you try running with the db.bolt.nofreelistsync and db.bolt.auto-compact config options?

zagota commented 3 years ago

Thanks, but did not help

~$ /usr/local/bin/lnd --db.bolt.nofreelistsync --db.bolt.auto-compact
2021-03-15 19:22:38.067 [INF] LTND: Version: 0.12.1-beta commit=v0.12.1-beta, build=production, logging=default, debuglevel=info
2021-03-15 19:22:38.067 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2021-03-15 19:22:38.067 [INF] LTND: Opening the main database, this might take a few minutes...
2021-03-15 19:22:38.067 [INF] LTND: Opening bbolt database, sync_freelist=true, auto_compact=true
2021-03-15 19:22:38.067 [INF] CHDB: Compacting database file at /home/lnd1/.lnd/data/graph/mainnet/channel.db
unexpected fault address 0x7f31996a2008
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f31996a2008 pc=0x7b6497]

goroutine 1 [running]:
runtime.throw(0x143bcb3, 0x5)
    runtime/panic.go:1116 +0x72 fp=0xc0002b24b0 sp=0xc0002b2480 pc=0x437692
runtime.sigpanic()
    runtime/signal_unix.go:749 +0x405 fp=0xc0002b24e0 sp=0xc0002b24b0 pc=0x44dde5
go.etcd.io/bbolt.(*elemRef).isLeaf(...)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:387
go.etcd.io/bbolt.(*Cursor).first(0xc0002b2640)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:170 +0x1b7 fp=0xc0002b2540 sp=0xc0002b24e0 pc=0x7b6497
go.etcd.io/bbolt.(*Cursor).next(0xc0002b2640, 0x8, 0x7f30f3ad465c, 0x1f9, 0x1f9, 0x0, 0xc0002b2bb8, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:234 +0x8b fp=0xc0002b2590 sp=0xc0002b2540 pc=0x7b678b
go.etcd.io/bbolt.(*Cursor).Next(0xc0002b2640, 0x8, 0x8, 0x7f30f3ad465c, 0x1f9, 0x1f9, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/cursor.go:75 +0x45 fp=0xc0002b2600 sp=0xc0002b2590 pc=0x7b5ae5
go.etcd.io/bbolt.(*Bucket).ForEach(0xc00030e280, 0xc0002b26d0, 0x1, 0x1)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:389 +0xb4 fp=0xc0002b2670 sp=0xc0002b2600 pc=0x7b3934
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).walkBucket(0xc0002b2f88, 0xc00030e280, 0xc000376000, 0x1, 0x1, 0x7f30ec58078f, 0xa, 0xa, 0x0, 0x0, ...)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:244 +0x1ce fp=0xc0002b2718 sp=0xc0002b2670 pc=0x7dc74e
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).walkBucket.func1(0x7f30ec58078f, 0xa, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:247 +0x1c8 fp=0xc0002b27b8 sp=0xc0002b2718 pc=0x7dd6c8
go.etcd.io/bbolt.(*Bucket).ForEach(0xc000127e00, 0xc0002b2888, 0x0, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:390 +0xfe fp=0xc0002b2828 sp=0xc0002b27b8 pc=0x7b397e
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).walkBucket(0xc0002b2f88, 0xc000127e00, 0x0, 0x0, 0x0, 0x7f30f3be6460, 0xa, 0xa, 0x0, 0x0, ...)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:244 +0x1ce fp=0xc0002b28d0 sp=0xc0002b2828 pc=0x7dc74e
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).walk.func1.1(0x7f30f3be6460, 0xa, 0xa, 0xc000127e00, 0xc000127e00, 0x0)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:221 +0x173 fp=0xc0002b2970 sp=0xc0002b28d0 pc=0x7dd413
go.etcd.io/bbolt.(*Tx).ForEach.func1(0x7f30f3be6460, 0xa, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:129 +0x8c fp=0xc0002b29b8 sp=0xc0002b2970 pc=0x7c812c
go.etcd.io/bbolt.(*Bucket).ForEach(0xc00003c398, 0xc0002b2a48, 0x1, 0xc000304048)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/bucket.go:390 +0xfe fp=0xc0002b2a28 sp=0xc0002b29b8 pc=0x7b397e
go.etcd.io/bbolt.(*Tx).ForEach(0xc00003c380, 0xc0002b2a90, 0x7b8f6f, 0xc000614000)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/tx.go:128 +0x67 fp=0xc0002b2a70 sp=0xc0002b2a28 pc=0x7c3fe7
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).walk.func1(0xc00003c380, 0x0, 0xc00003c380)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:216 +0x65 fp=0xc0002b2ab8 sp=0xc0002b2a70 pc=0x7dd4c5
go.etcd.io/bbolt.(*DB).View(0xc000614000, 0xc0002b2b40, 0x0, 0x0)
    go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:725 +0x96 fp=0xc0002b2b20 sp=0xc0002b2ab8 pc=0x7b9c76
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).walk(0xc0002b2f88, 0xc000614000, 0xc0002b2bb8, 0x0, 0x0)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:215 +0x66 fp=0xc0002b2b68 sp=0xc0002b2b20 pc=0x7dc546
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).compact(0xc0002b2f88, 0xc000614200, 0xc000614000, 0x0, 0x0)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:143 +0x125 fp=0xc0002b2c08 sp=0xc0002b2b68 pc=0x7dc3e5
github.com/lightningnetwork/lnd/channeldb/kvdb.(*compacter).execute(0xc0002b2f88, 0x0, 0x0, 0x0, 0x0)
    github.com/lightningnetwork/lnd/channeldb/kvdb/bolt_compact.go:113 +0x6db fp=0xc0002b2e48 sp=0xc0002b2c08 pc=0x7dbebb
github.com/lightningnetwork/lnd/channeldb/kvdb.compactAndSwap(0xc0002b30d8, 0x0, 0x0)
    github.com/lightningnetwork/lnd/channeldb/kvdb/backend.go:173 +0x58d fp=0xc0002b2fe0 sp=0xc0002b2e48 pc=0x7daa2d
github.com/lightningnetwork/lnd/channeldb/kvdb.GetBoltBackend(0xc0002b30d8, 0x1, 0xc0000433b0, 0x22, 0xc0002b3110)
    github.com/lightningnetwork/lnd/channeldb/kvdb/backend.go:95 +0x245 fp=0xc0002b3078 sp=0xc0002b2fe0 pc=0x7da245
github.com/lightningnetwork/lnd/lncfg.(*DB).GetBackends(0xc0003004b0, 0x17221a0, 0xc00030f340, 0xc0000433b0, 0x22, 0x143e904, 0x7, 0x203000, 0x203000, 0x14c)
    github.com/lightningnetwork/lnd/lncfg/db.go:101 +0x11f fp=0xc0002b3120 sp=0xc0002b3078 pc=0xc8be9f
github.com/lightningnetwork/lnd.initializeDatabases(0x17221a0, 0xc00030f340, 0xc000545200, 0xc00030f340, 0xc0003be250, 0x2, 0x0, 0x0)
    github.com/lightningnetwork/lnd/lnd.go:1358 +0x139 fp=0xc0002b32a8 sp=0xc0002b3120 pc=0x1085ed9
github.com/lightningnetwork/lnd.Main(0xc000545200, 0x0, 0x0, 0xc00007c300, 0x0, 0x0)
    github.com/lightningnetwork/lnd/lnd.go:261 +0x465 fp=0xc0002b3f20 sp=0xc0002b32a8 pc=0x107de85
main.main()
    github.com/lightningnetwork/lnd/cmd/lnd/main.go:35 +0x16c fp=0xc0002b3f88 sp=0xc0002b3f20 pc=0x10e91cc
runtime.main()
    runtime/proc.go:204 +0x209 fp=0xc0002b3fe0 sp=0xc0002b3f88 pc=0x439e89
runtime.goexit()
    runtime/asm_amd64.s:1374 +0x1 fp=0xc0002b3fe8 sp=0xc0002b3fe0 pc=0x46fa61

goroutine 37 [select]:
io.(*pipe).Read(0xc0001fef60, 0xc0003c0000, 0x1000, 0x1000, 0xc000396da8, 0x4d8a45, 0xc0001fef00)
    io/pipe.go:57 +0xe7
io.(*PipeReader).Read(0xc00000e488, 0xc0003c0000, 0x1000, 0x1000, 0x6e, 0xc0001fef18, 0x4dcf00)
    io/pipe.go:134 +0x4c
bufio.(*Reader).fill(0xc000396ef8)
    bufio/bufio.go:101 +0x105
bufio.(*Reader).ReadSlice(0xc000396ef8, 0xa, 0x1, 0x0, 0x0, 0x0, 0x0)
    bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc000396ef8, 0x1e1503e, 0x1, 0x1, 0x1, 0x0, 0x0)
    bufio/bufio.go:389 +0x34
github.com/jrick/logrotate/rotator.(*Rotator).Run(0xc00030f280, 0x1702180, 0xc00000e488, 0x0, 0x0)
    github.com/jrick/logrotate@v1.0.0/rotator/rotator.go:100 +0x93
github.com/lightningnetwork/lnd/build.(*RotatingLogWriter).InitLogRotator.func1(0xc000300450, 0xc00000e488)
    github.com/lightningnetwork/lnd/build/logrotator.go:83 +0x49
created by github.com/lightningnetwork/lnd/build.(*RotatingLogWriter).InitLogRotator
    github.com/lightningnetwork/lnd/build/logrotator.go:82 +0x325

goroutine 40 [syscall]:
os/signal.signal_recv(0x0)
    runtime/sigqueue.go:147 +0x9d
os/signal.loop()
    os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
    os/signal/signal.go:150 +0x45

goroutine 41 [select]:
github.com/lightningnetwork/lnd/signal.mainInterruptHandler()
    github.com/lightningnetwork/lnd/signal/signal.go:83 +0x11b
created by github.com/lightningnetwork/lnd/signal.Intercept
    github.com/lightningnetwork/lnd/signal/signal.go:49 +0xf2
wpaulino commented 3 years ago

Did the node experience any unexpected shutdowns (e.g. power failure)? You'll want to become familiar with our recovery process if you had any funds on the node.

zagota commented 3 years ago

The node was running for 7 days and have stopped properly. Is there no way to repair the database? During a restore all my channels are closed.

wpaulino commented 3 years ago

Not that I'm aware of. Any ideas @guggero?

guggero commented 3 years ago

The SIGSEGV: segmentation violation code=0x1 addr=0x7f31996a2008 pc=0x7b6497 error is very typical for data corruption. Are you running the system on a SD card or some otherwise unreliable storage medium? You can try chantools dropchannelgraph on the DB to remove all graph related data (maybe the corruption is limited to that) and then try compaction again. If that doesn't work, I don't see any other way than using the recovery process and closing out all channels.

zagota commented 3 years ago

The node runs on a SSD RAID and ZFS file system.

Found these errors in lnd.log.

2021-03-15 12:21:17.555 [ERR] HSWC: insufficient bandwidth to route htlc
2021-03-15 12:21:17.555 [ERR] HSWC: Unhandled error while reforwarding htlc settle/fail over htlcswitch: insufficient bandwidth to route htlc

2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called
2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called
2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Unavailable desc = transport is closing
2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Unavailable desc = transport is closing
2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called
2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called
2021-03-15 10:38:48.472 [INF] INVC: Cancelling invoice subscription for client=21
2021-03-15 10:38:48.472 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: rpc error: code = Internal desc = transport: transport: the stream is done or WriteHeader was already called
2021-03-15 10:38:48.472 [INF] INVC: Cancelling invoice subscription for client=22

Result from chantools

chantools dropchannelgraph --channeldb ~/.lnd/data/graph/mainnet/channel.db
2021-03-16 12:59:26.954 [INF] CHAN: chantools version v0.8.2 commit 
unexpected fault address 0x7f8e49166000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f8e49166000 pc=0x751f82]

Start recovery and hope for the best.

guggero commented 3 years ago

The node runs on a SSD RAID and ZFS file system.

Then it should be very unlikely for data corruption to occur, unless you were very unlucky. What's the status of the ZFS tank? Does it indicate any read/write errors if you run zpool status?

Those errors you posted seem unrelated and are probably coming from an RPC client closing the connection improperly. What RPC clients are using your lnd node?

zagota commented 3 years ago

zpool status shows no errors, scrubbing run without errors.

lncli and Zap Deskotp Wallet access the node.

zagota commented 3 years ago

all funds have been restored, thank you @guggero @wpaulino