search

raspiblitz / raspiblitz

Get your own Bitcoin & Lightning Node running - on a RaspberryPi with a nice LCD
MIT License
2.44k stars 520 forks source link

Power Outage - reloading blockchain? #845

Closed thephizzz closed 4 years ago

thephizzz commented 4 years ago

We had a power outage that knocked the Raspiblitz off line (~24 hours). After restarting, it looks like it is re-loading the entire blockchain (sync 1.9%) and states the LND service is not running. Is this normal? And is there a way around it? Thanks.

BLOCKCHAIN SYSTEMD STATUS ● bitcoind.service - Bitcoin daemon Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-02 12:52:31 GMT; 7min ago Process: 2321 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log blockchain STARTED (code=exited, status=0/SUCCESS) Process: 2328 ExecStart=/usr/local/bin/bitcoind -daemon -conf=/home/bitcoin/.bitcoin/bitcoin.conf -pid=/home/bitcoin/.bitcoin/bitcoind.pid (code=exited, status=0/SUCCESS) Main PID: 2380 (bitcoind) Tasks: 20 (limit: 4915) Memory: 592.5M CGroup: /system.slice/bitcoind.service └─2380 /usr/local/bin/bitcoind -daemon -conf=/home/bitcoin/.bitcoin/bitcoin.conf -pid=/home/bitcoin/.bitcoin…

Nov 02 12:52:30 lndnode systemd[1]: Starting Bitcoin daemon... Nov 02 12:52:31 lndnode systemd[1]: Started Bitcoin daemon.

LAST BLOCKCHAIN ERROR LOGS sudo journalctl -u bitcoind -b --no-pager -n8 -- Logs begin at Thu 2019-02-14 10:11:59 GMT, end at Sat 2019-11-02 13:00:19 GMT. -- Nov 02 12:52:30 lndnode systemd[1]: Starting Bitcoin daemon... Nov 02 12:52:31 lndnode systemd[1]: Started Bitcoin daemon.

LAST BLOCKCHAIN 20 INFO LOGS sudo tail -n 20 /mnt/hdd/bitcoin/debug.log 2019-11-02T13:00:14Z UpdateTip: new best=000000000000001572b3c20fb9756eb370c2838f4b6e2209274beb9adac75c4e height=208601 version=0x00000001 log2_work=69.044104 tx=9030896 date='2012-11-19T11:53:18Z' progress=0.019379 cache=48.8MiB(441132txo) 2019-11-02T13:00:15Z UpdateTip: new best=000000000000014aca46b0472ac5846212412a2a34d14e1345177555da789308 height=208602 version=0x00000001 log2_work=69.044138 tx=9031556 date='2012-11-19T11:54:34Z' progress=0.019380 cache=48.9MiB(441817txo) 2019-11-02T13:00:15Z UpdateTip: new best=00000000000001e2ed2cd49b212123551c816f2c4427b696051277dfa08ae1ba height=208603 version=0x00000001 log2_work=69.044173 tx=9031809 date='2012-11-19T12:08:17Z' progress=0.019381 cache=48.9MiB(442006txo) 2019-11-02T13:00:16Z Pre-allocating up to position 0x900000 in rev00029.dat 2019-11-02T13:00:16Z UpdateTip: new best=000000000000037ddc8e76c44b2ab1ab315c7d12143d0d3ebe0c977b820d31ed height=208604 version=0x00000002 log2_work=69.044207 tx=9032658 date='2012-11-19T12:45:32Z' progress=0.019383 cache=48.9MiB(442133txo) 2019-11-02T13:00:16Z UpdateTip: new best=000000000000007e2c4663969380fb0f4af6058cd137000d021bcf4022e640d2 height=208605 version=0x00000002 log2_work=69.044241 tx=9032939 date='2012-11-19T12:51:48Z' progress=0.019383 cache=48.9MiB(442169txo) 2019-11-02T13:00:16Z UpdateTip: new best=00000000000002e94d024e70579fc59ba239e63e6ff6bc94c887fb6c18b3f537 height=208606 version=0x00000001 log2_work=69.044275 tx=9033000 date='2012-11-19T12:57:55Z' progress=0.019384 cache=49.0MiB(442182txo) 2019-11-02T13:00:16Z UpdateTip: new best=00000000000001321aad7921ebb42333fa2c6d46b75a46be10bfcc6e0ab11ebf height=208607 version=0x00000001 log2_work=69.04431 tx=9033010 date='2012-11-19T13:09:05Z' progress=0.019384 cache=49.0MiB(442185txo) 2019-11-02T13:00:16Z UpdateTip: new best=000000000000014e85187adcfee10893629c27e874bc20e34ace070a712bb39e height=208608 version=0x00000002 log2_work=69.044344 tx=9033663 date='2012-11-19T13:22:29Z' progress=0.019385 cache=49.0MiB(442865txo) 2019-11-02T13:00:16Z UpdateTip: new best=00000000000004ad0c50e9858aa47f94ae7574d0a26366ce4f3274ce15d8cad3 height=208609 version=0x00000001 log2_work=69.044378 tx=9033800 date='2012-11-19T13:21:07Z' progress=0.019385 cache=49.0MiB(442912txo) 2019-11-02T13:00:16Z UpdateTip: new best=00000000000001275c69730dd41ddb28354d51e4c0e0bf63544ee2df6feb89e1 height=208610 version=0x00000001 log2_work=69.044413 tx=9033947 date='2012-11-19T13:47:07Z' progress=0.019386 cache=49.0MiB(443019txo) 2019-11-02T13:00:17Z UpdateTip: new best=000000000000011ee0d0b45abf37ba749497dd08d427a5bf28ba4033bd1aa62e height=208611 version=0x00000001 log2_work=69.044447 tx=9034442 date='2012-11-19T13:51:28Z' progress=0.019387 cache=49.1MiB(443195txo) 2019-11-02T13:00:17Z UpdateTip: new best=00000000000001f46ea736faa84fd02cce3b7f942ac3f1b9f70d3857f5b4d48f height=208612 version=0x00000001 log2_work=69.044481 tx=9034539 date='2012-11-19T13:56:30Z' progress=0.019387 cache=49.1MiB(443305txo) 2019-11-02T13:00:17Z UpdateTip: new best=0000000000000238e4eb2e3d3ca49aaf4d8ba5a24be70ba6cd93ddb9cea33b47 height=208613 version=0x00000001 log2_work=69.044515 tx=9034716 date='2012-11-19T13:59:27Z' progress=0.019387 cache=49.1MiB(443277txo) 2019-11-02T13:00:17Z UpdateTip: new best=00000000000004cdd4eb2ec88021a4ca01b05841ac0eb757f8ce17c42691f1e7 height=208614 version=0x00000001 log2_work=69.04455 tx=9034822 date='2012-11-19T14:03:00Z' progress=0.019387 cache=49.1MiB(443385txo) 2019-11-02T13:00:17Z UpdateTip: new best=00000000000000190c96933fa6abf93f7c1fb9ae30c3a7b723e5d703b785294f height=208615 version=0x00000002 log2_work=69.044584 tx=9034976 date='2012-11-19T14:13:55Z' progress=0.019388 cache=49.1MiB(443479txo) 2019-11-02T13:00:18Z UpdateTip: new best=00000000000001ca8c695f6cd7b77e277cd1feb0f6c0e261398bcd55267e25d8 height=208616 version=0x00000001 log2_work=69.044618 tx=9035736 date='2012-11-19T14:39:30Z' progress=0.019389 cache=49.1MiB(443793txo) 2019-11-02T13:00:18Z UpdateTip: new best=000000000000028d84cff2b28d91087713ab3fd65bf5c8322b909089a742d8f3 height=208617 version=0x00000002 log2_work=69.044653 tx=9036266 date='2012-11-19T14:54:42Z' progress=0.019391 cache=49.1MiB(443928txo) 2019-11-02T13:00:18Z UpdateTip: new best=000000000000001f9c155bb2d069cbb899276fca87351dcaeb50d80134dc7d17 height=208618 version=0x00000002 log2_work=69.044687 tx=9036692 date='2012-11-19T15:10:15Z' progress=0.019392 cache=49.2MiB(444310txo) 2019-11-02T13:00:19Z UpdateTip: new best=00000000000002c582ffc8bf6cec674f0418b75e2dee7eca05ab07c2628a4846 height=208619 version=0x00000002 log2_work=69.044721 tx=9037263 date='2012-11-19T15:28:00Z' progress=0.019393 cache=51.0MiB(444735txo)

LND SYSTEMD STATUS ● lnd.service - LND Lightning Daemon Loaded: loaded (/etc/systemd/system/lnd.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Sat 2019-11-02 12:59:34 GMT; 45s ago Process: 14814 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log lightning STARTED (code=exited, status=0/SUCCESS) Process: 14816 ExecStart=/usr/local/bin/lnd --externalip=${publicIP}:${lndPort} (code=exited, status=1/FAILURE) Main PID: 14816 (code=exited, status=1/FAILURE)

Nov 02 12:59:34 lndnode systemd[1]: lnd.service: Main process exited, code=exited, status=1/FAILURE Nov 02 12:59:34 lndnode systemd[1]: lnd.service: Failed with result 'exit-code'.

LAST LND ERROR LOGS sudo journalctl -u lnd -b --no-pager -n12 -- Logs begin at Thu 2019-02-14 10:11:59 GMT, end at Sat 2019-11-02 13:00:19 GMT. -- Nov 02 12:58:33 lndnode systemd[1]: Started LND Lightning Daemon. Nov 02 12:58:34 lndnode lnd[12167]: /home/bitcoin/.lnd/lnd.conf:22: strconv.ParseUint: parsing "0.00001": invalid syntax Nov 02 12:58:34 lndnode systemd[1]: lnd.service: Main process exited, code=exited, status=1/FAILURE Nov 02 12:58:34 lndnode systemd[1]: lnd.service: Failed with result 'exit-code'. Nov 02 12:59:34 lndnode systemd[1]: lnd.service: Service RestartSec=1min expired, scheduling restart. Nov 02 12:59:34 lndnode systemd[1]: lnd.service: Scheduled restart job, restart counter is at 7. Nov 02 12:59:34 lndnode systemd[1]: Stopped LND Lightning Daemon. Nov 02 12:59:34 lndnode systemd[1]: Starting LND Lightning Daemon... Nov 02 12:59:34 lndnode systemd[1]: Started LND Lightning Daemon. Nov 02 12:59:34 lndnode lnd[14816]: /home/bitcoin/.lnd/lnd.conf:22: strconv.ParseUint: parsing "0.00001": invalid syntax Nov 02 12:59:34 lndnode systemd[1]: lnd.service: Main process exited, code=exited, status=1/FAILURE Nov 02 12:59:34 lndnode systemd[1]: lnd.service: Failed with result 'exit-code'.

LAST 30 LND INFO LOGS sudo tail -n 30 /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log tail: cannot open '/mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log' for reading: No such file or directory

LAST 20 RTL LOGS -- Logs begin at Thu 2019-02-14 10:11:59 GMT, end at Sat 2019-11-02 13:00:20 GMT. -- Nov 02 12:52:31 lndnode systemd[1]: Started RTL daemon.

HARDWARE TEST RESULTS UndervoltageReports in Logs: 0 raspberryPi=4 powerFAIL=0 powerWARN=0 powerMIN='854200 microVolt' tempFAIL=2 tempWARN=11 tempMAX='7800 centiGrad' IMPORTANT: There are some hardware issues with your setup. 'Run Hardwaretest' in main menu or: sudo /home/admin/05hardwareTest.sh

SYSTEM STATUS (can take some seconds to gather) localIP='192.168.1.237' tempCelsius='74.9' uptime=658 upsStatus='OFF' startcountBlockchain=1 bitcoinActive=1

thephizzz commented 4 years ago

BTW. The node had been up and running for a week or two before the outage with a completely synced blockchain.

thephizzz commented 4 years ago

Actually - now it looks like it may just be re-indexing.

BLOCKCHAIN SYSTEMD STATUS ● bitcoind.service - Bitcoin daemon Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-02 12:52:31 GMT; 45min ago Process: 2321 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log blockchain STARTED (code=exited, status=0/SUCCESS) Process: 2328 ExecStart=/usr/local/bin/bitcoind -daemon -conf=/home/bitcoin/.bitcoin/bitcoin.conf -pid=/home/bitcoin/.bitcoin/bitcoind.pid (code=exited, status=0/SUCCESS) Main PID: 2380 (bitcoind) Tasks: 20 (limit: 4915) Memory: 3.5G CGroup: /system.slice/bitcoind.service └─2380 /usr/local/bin/bitcoind -daemon -conf=/home/bitcoin/.bitcoin/bitcoin.conf -pid=/home/bitcoin/.bitcoin/bitcoind.pid

Nov 02 12:52:30 lndnode systemd[1]: Starting Bitcoin daemon... Nov 02 12:52:31 lndnode systemd[1]: Started Bitcoin daemon.

LAST BLOCKCHAIN ERROR LOGS sudo journalctl -u bitcoind -b --no-pager -n8 -- Logs begin at Thu 2019-02-14 10:11:59 GMT, end at Sat 2019-11-02 13:37:35 GMT. -- Nov 02 12:52:30 lndnode systemd[1]: Starting Bitcoin daemon... Nov 02 12:52:31 lndnode systemd[1]: Started Bitcoin daemon.

LAST BLOCKCHAIN 20 INFO LOGS sudo tail -n 20 /mnt/hdd/bitcoin/debug.log 2019-11-02T13:35:49Z Loaded 572 blocks from external file in 11094ms 2019-11-02T13:35:49Z Reindexing block file blk00147.dat... 2019-11-02T13:35:59Z Loaded 567 blocks from external file in 10609ms 2019-11-02T13:35:59Z Reindexing block file blk00148.dat... 2019-11-02T13:36:09Z Loaded 646 blocks from external file in 10333ms 2019-11-02T13:36:09Z Reindexing block file blk00149.dat... 2019-11-02T13:36:20Z Loaded 624 blocks from external file in 10675ms 2019-11-02T13:36:20Z Reindexing block file blk00150.dat... 2019-11-02T13:36:31Z Loaded 727 blocks from external file in 10711ms 2019-11-02T13:36:31Z Reindexing block file blk00151.dat... 2019-11-02T13:36:42Z Loaded 652 blocks from external file in 10919ms 2019-11-02T13:36:42Z Reindexing block file blk00152.dat... 2019-11-02T13:36:52Z Loaded 658 blocks from external file in 10596ms 2019-11-02T13:36:52Z Reindexing block file blk00153.dat... 2019-11-02T13:37:03Z Loaded 441 blocks from external file in 11078ms 2019-11-02T13:37:03Z Reindexing block file blk00154.dat... 2019-11-02T13:37:15Z Loaded 626 blocks from external file in 11025ms 2019-11-02T13:37:15Z Reindexing block file blk00155.dat... 2019-11-02T13:37:25Z Loaded 520 blocks from external file in 10700ms 2019-11-02T13:37:25Z Reindexing block file blk00156.dat...

LND SYSTEMD STATUS ● lnd.service - LND Lightning Daemon Loaded: loaded (/etc/systemd/system/lnd.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Sat 2019-11-02 13:36:55 GMT; 40s ago Process: 30833 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log lightning STARTED (code=exited, status=0/SUCCESS) Process: 30835 ExecStart=/usr/local/bin/lnd --externalip=${publicIP}:${lndPort} (code=exited, status=1/FAILURE) Main PID: 30835 (code=exited, status=1/FAILURE)

LAST LND ERROR LOGS sudo journalctl -u lnd -b --no-pager -n12 -- Logs begin at Thu 2019-02-14 10:11:59 GMT, end at Sat 2019-11-02 13:37:36 GMT. -- Nov 02 13:35:54 lndnode systemd[1]: Started LND Lightning Daemon. Nov 02 13:35:54 lndnode lnd[29250]: /home/bitcoin/.lnd/lnd.conf:22: strconv.ParseUint: parsing "0.00001": invalid syntax Nov 02 13:35:54 lndnode systemd[1]: lnd.service: Main process exited, code=exited, status=1/FAILURE Nov 02 13:35:54 lndnode systemd[1]: lnd.service: Failed with result 'exit-code'. Nov 02 13:36:55 lndnode systemd[1]: lnd.service: Service RestartSec=1min expired, scheduling restart. Nov 02 13:36:55 lndnode systemd[1]: lnd.service: Scheduled restart job, restart counter is at 44. Nov 02 13:36:55 lndnode systemd[1]: Stopped LND Lightning Daemon. Nov 02 13:36:55 lndnode systemd[1]: Starting LND Lightning Daemon... Nov 02 13:36:55 lndnode systemd[1]: Started LND Lightning Daemon. Nov 02 13:36:55 lndnode lnd[30835]: /home/bitcoin/.lnd/lnd.conf:22: strconv.ParseUint: parsing "0.00001": invalid syntax Nov 02 13:36:55 lndnode systemd[1]: lnd.service: Main process exited, code=exited, status=1/FAILURE Nov 02 13:36:55 lndnode systemd[1]: lnd.service: Failed with result 'exit-code'.

LAST 30 LND INFO LOGS sudo tail -n 30 /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log tail: cannot open '/mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log' for reading: No such file or directory

LAST 20 RTL LOGS -- Logs begin at Thu 2019-02-14 10:11:59 GMT, end at Sat 2019-11-02 13:37:36 GMT. -- Nov 02 12:52:31 lndnode systemd[1]: Started RTL daemon.

HARDWARE TEST RESULTS UndervoltageReports in Logs: 0 raspberryPi=4 powerFAIL=0 powerWARN=0 powerMIN='854200 microVolt' tempFAIL=2 tempWARN=11 tempMAX='7800 centiGrad' IMPORTANT: There are some hardware issues with your setup. 'Run Hardwaretest' in main menu or: sudo /home/admin/05hardwareTest.sh

SYSTEM STATUS (can take some seconds to gather) localIP='192.168.1.237' tempCelsius='51.6' uptime=2894 upsStatus='OFF' startcountBlockchain=1 bitcoinActive=1 blockchainHeight=212890 initialSync=1 syncProgress=2.17 startcountLightning=45 lndActive=0 scriptRuntime=0

OPTION: SHARE THIS DEBUG OUTPUT An easy way to share this debug output on GitHub or on a support chat use the following command and share the resulting link: /home/admin/XXdebugLogs.sh | nc termbin.com 9999

rootzoll commented 4 years ago

Yeah - a power outage is most often creates a data corruption in the blockchain and after that you need to repair it or let the bitcoind do the reindexing. Thats normal.

To protect against power outages you can check the UPS support of the RaspiBlitz since v1.3: https://github.com/rootzoll/raspiblitz/issues/707

I am closing the issue - if you have any further reports, feel free to post here anyway.