lnd won't start up (unable to open databases: unable to obtain database backends: cannot allocate memory)

hpbock commented 3 years ago

Background

LND unexpectedly crashed this afternoon and will no longer start up.

The last three line of the logfile before it suddenly ended:

2021-08-21 16:24:21.567 [WRN] DISC: ignoring remote ChannelAnnouncement for own channel
2021-08-21 16:24:23.871 [WRN] DISC: ignoring remote ChannelAnnouncement for own channel
2021-08-21 16:24:28.135 [WRN] DISC: ignoring remote ChannelAnnouncement for own channel

Your environment

0.13.1-beta
Linux odroid 4.14.180-178 # 1 SMP PREEMPT Wed Sep 2 12:39:45 -03 2020 armv7l armv7l armv7l GNU/Linux
bitcoind

Steps to reproduce

I just start lnd and it immediately shuts down.

Expected behaviour

I expect lnd to start up and wait for the passphrase to unlock the wallet.

Actual behaviour

$ lnd 
Attempting automatic RPC configuration to bitcoind
Automatically obtained bitcoind's RPC credentials
2021-08-21 21:51:08.489 [INF] LTND: Version: 0.13.1-beta commit=v0.13.1-beta, build=production, logging=default, debuglevel=debug
2021-08-21 21:51:08.490 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2021-08-21 21:51:08.495 [DBG] WLKT: WalletKit RPC server successfully registered with root gRPC server
2021-08-21 21:51:08.495 [DBG] RRPC: Router RPC server successfully register with root gRPC server
2021-08-21 21:51:08.495 [DBG] SGNR: Signer RPC server successfully register with root gRPC server
2021-08-21 21:51:08.495 [DBG] ARPC: Autopilot RPC server successfully register with root gRPC server
2021-08-21 21:51:08.495 [DBG] NTFR: ChainNotifier RPC server successfully register with root gRPC server
2021-08-21 21:51:08.495 [DBG] IRPC: Invoices RPC server successfully registered with root gRPC server
2021-08-21 21:51:08.496 [DBG] VRPC: Versioner RPC server successfully registered with root gRPC server
2021-08-21 21:51:08.496 [INF] RPCS: RPC server listening on 127.0.0.1:10007
2021-08-21 21:51:08.500 [DBG] WLKT: WalletKit REST server successfully registered with root REST server
2021-08-21 21:51:08.504 [DBG] RRPC: Router REST server successfully registered with root REST server
2021-08-21 21:51:08.505 [DBG] SGNR: Signer REST server successfully registered with root REST server
2021-08-21 21:51:08.506 [DBG] ARPC: Autopilot REST server successfully registered with root REST server
2021-08-21 21:51:08.506 [DBG] NTFR: ChainNotifier REST server successfully registered with root REST server
2021-08-21 21:51:08.506 [DBG] IRPC: Invoices REST server successfully registered with root REST server
2021-08-21 21:51:08.506 [DBG] VRPC: Versioner REST server successfully registered with root REST server
2021-08-21 21:51:08.507 [INF] RPCS: gRPC proxy started at 0.0.0.0:8087
2021-08-21 21:51:08.507 [INF] LTND: Opening the main database, this might take a few minutes...
2021-08-21 21:51:08.507 [INF] LTND: Opening bbolt database, sync_freelist=true, auto_compact=false
2021-08-21 21:51:08.529 [INF] LTND: Shutdown complete

unable to open databases: unable to obtain database backends: cannot allocate memory

Roasbeef commented 3 years ago

How large is your database? You may need to try compacting it if you're on a 32-bit system, and don't have enough memory for the initial memory map.

hpbock commented 3 years ago

$ ls -la .lnd/data/chain/bitcoin/mainnet
total 71916
-rw------- 1 lnd-mainnet lnd-mainnet      293 Mai 12 13:24 admin.macaroon
-rw-r--r-- 1 lnd-mainnet lnd-mainnet       83 Mai 12 13:29 chainnotifier.macaroon
-rw-rw-r-- 1 lnd-mainnet lnd-mainnet    31050 Aug 21 16:07 channel.backup
-rw-r--r-- 1 lnd-mainnet lnd-mainnet      132 Mai 12 13:24 invoice.macaroon
-rw-r--r-- 1 lnd-mainnet lnd-mainnet       91 Mai 12 13:29 invoices.macaroon
-rw------- 1 lnd-mainnet lnd-mainnet    32768 Aug  9 10:42 macaroons.db
-rw-r--r-- 1 lnd-mainnet lnd-mainnet      217 Mai 12 13:24 readonly.macaroon
-rw-r--r-- 1 lnd-mainnet lnd-mainnet       91 Mai 12 13:29 router.macaroon
-rw-r--r-- 1 lnd-mainnet lnd-mainnet       92 Mai 12 13:29 signer.macaroon
-rw------- 1 lnd-mainnet lnd-mainnet 81629184 Aug 21 16:22 wallet.db
-rw-r--r-- 1 lnd-mainnet lnd-mainnet      114 Mai 12 13:29 walletkit.macaroon
$ ls -la .lnd/data/graph/mainnet/
total 1104596
-rw------- 1 lnd-mainnet lnd-mainnet 1075343360 Aug 21 16:24 channel.db
-rw------- 1 lnd-mainnet lnd-mainnet   57761792 Aug 21 16:22 sphinxreplay.db
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           1,9G        1,0G         45M        2,6M        905M        922M
Swap:          8,0G         74M        7,9G

@Roasbeef i guess you mean the channel.db?

Roasbeef commented 3 years ago

See this comment: https://github.com/lightningnetwork/lnd/issues/4811#issuecomment-813553011

Yes, I mean channel.db.

hpbock commented 3 years ago

@Roasbeef thank you for your quick response on this sunday! :smiley: I'll check it right now!

hpbock commented 3 years ago

After compacting the database using mv channel.db old-channel.db && bbolt compact -o channel.db old-channel.db by 2.28 using bbolt, my lnd node now starts fine! @Roasbeef thank you a lot for your help!

lightningnetwork / lnd