search

raspiblitz / raspiblitz

Get your own Bitcoin & Lightning Node running - on a RaspberryPi with a nice LCD
MIT License
2.45k stars 520 forks source link

crash, lnd unlock cycle, fs error? #2573

Closed klnit closed 3 years ago

klnit commented 3 years ago

Hi,

I have my node running for almost one year. Hardware from Fulmo. No big problems. Updated in April. The last couple of weeks I didn't to much with it so i didn't realized it crashed sometime in the last 6 weeks. I don't remember exactly the error on the screen. I think it was filesystem related. I did a cold reboot and had several times the lnd unlock cycle where it asks for Pw C repeatedly. Sometimes it seemed to work eventually. Bitcoin was synced but lnd does not seem to start or connect channels. Under SSH I got this error one or two times: Message from syslogd@raspberrypi at Sep 23 14:47:20 ...───────────────────────────────────┘ kernel:[ 709.294725] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss! (inode 16252933, error -30)

The first couple of minutes after reboot, everything looks normal. Infoscreen good except lnd does not connect channels. Then it goes into error mode, bincoind not running and showing periodically: /home/admin/config.scrripts/blitz.statusscan.sh: line 130: /hdd/temp/.lnd.eror: Read-only file system

I have attached the logs before and after it goes in error mode. I am a little bit lost. What can I do? Burn new SD card? Migrate to new SSD? justAfterReboot.txt after19Min.txt

openoms commented 3 years ago

Most likely a hardware error of the disk or disk adapter. Use the recommended or tested parts.

I would get a new SSD and start fresh on that. Create a migration file from the REPAIR menu if you have funds on the node.

Maybe return the old disk as faulty (mind the privacy implications of the unencrypted data)

klnit commented 3 years ago

I have flashed a new SD Card and bought a new SSD and new adapter. None of the above mentioned errors. I used the migration but after bitcoin syncing lnd still wont start. After multiple attempts is says just "exit" "status 2". I also tried "reset lnd" with rescue file. Same. Tried 1.7.1.RC2. Same. LND wont start. Tried chantools and a few other things but channel.db seemed to be corrupt. So I went with Seed+SCB. It took ages but got my coins back.

klnit commented 3 years ago

Is it possible to run two blitz in a kind of failover cluster, or something similar to minimize downtime and protect against crashes? Do you have a hint?

openoms commented 3 years ago

the upcoming postgres integration of LND might make thsi possibel, but it is far outside of the scope of raspiblitz itself