Open lorenzolfm opened 4 months ago
Thanks for raising this issue. I'd recommend to keep lnd shutdown for the moment to not publish any outdated state.
Could you try the compaction with chantools which is described a bit here and see if that succeeds? https://github.com/lightningnetwork/lnd/issues/8532#issuecomment-1985577491
It seem to have compacted the db with no issue.
chantools compactdb channel.db --sourcedb ./channel.db --destdb ./compacted.db
2024-04-10 13:17:13.335 [INF] CHAN: chantools version v0.13.0 commit
~/tmp $ ls
channel.db compacted.db results
Ok that's great, could you check that the .last-compacted
file in your live-directory is empty? If so then move it out to a different temporary place, then turn the autocompaction of your node back on and restart lnd.
Seems like the .last-compacted
file was corrupted? Are you maybe running from an SD card?
Ok that's great, could you check that the .last-compacted file in your live-directory is empty? If so then move it out to a different temporary place, then turn the autocompaction of your node back on and restart lnd.
Moving the channel.db.last-compacted
still results is the same issue. The output of cat
file is:
^WÄõÈy^[à<80>
Seems like the .last-compacted file was corrupted? Are you maybe running from an SD card?
No I'm using an SSD.
I'm not discarding corruption, but since moving the file and starting LND from scratch didn't work it doesn't seem that corruption is involved.
func lastCompactionDate(dbFile string) (time.Time, error) {
zeroTime := time.Unix(0, 0)
tsFile := fmt.Sprintf("%s%s", dbFile, LastCompactionFileNameSuffix)
if !fileExists(tsFile) {
return zeroTime, nil
}
tsBytes, err := ioutil.ReadFile(tsFile)
if err != nil {
return zeroTime, err
}
tsNano := byteOrder.Uint64(tsBytes)
return time.Unix(0, int64(tsNano)), nil
}
If I understand correctly, if the .last-compacted file does not exists, the function returns early. Which is odd, since the file channel.db.last-compacted
is created before this function even runs. Reading from an empty file could explain the crash on byteOrder.Uint64(tsBytes), tho.
Background
LND panics when calling
lastCompactionDate
:lnd[5921]: panic: runtime error: index out of range [7] with length 0
. This happened after a power outage that rebooted my machine. I was able to avoid the issue by settingdb.bolt.auto-compact=false
.Your environment
lnd
: lnd version 0.17.4-beta commit=v0.17.4-betauname -a
on *Nix): Debian GNU/Linux 11 (bullseye) aarch64btcd
,bitcoind
, or other backend: v26.1.0Expected behaviour
LND should be able to startup with
db.bolt.auto-compact=true
, or at least fail gracefully with better error handling.Actual behaviour
LND panics, see stacktrace: