search

Start9Labs / bitcoind-startos

wrapper for building bitcoind.s9pk
Other
14 stars 20 forks source link

[bug]: bitcoind sometimes hard restarts, losing sync progress, even after synced 100% #42

Open chrisguida opened 2 years ago

chrisguida commented 2 years ago

Sometimes bitcoind just dies. This results in lost sync progress. I recently had this happen and it started resyncing 200 blocks back.

It is not being OOM killed, since there is no record of this in dmesg.

No idea what else it could be


Jan 28 21:42:22 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:37:38Z Socks5() connect to yhjkylctf577degc7ufffvg3gsdxgnx64pdhzfujxjgfhbulmtc7yaad.onion:8333 failed: InterruptibleRecv() timeout or other failure
Jan 28 21:53:46 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:53:46Z UpdateTip: new best=0000000000000000000847dc41fd2a0f88636d8c34cca72e4987a1571740e68e height=720802 version=0x20800000 log2_work=93.314192 tx=706379614 date='2022-01-28T21:53:18Z' progress=1.000000 cache=141.1MiB(1055756txo)
Jan 28 21:56:14 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:56:14Z UpdateTip: new best=0000000000000000000a71cc573d186eb8a76496f41c4f6f72a108b89c717f33 height=720803 version=0x20c00000 log2_work=93.314206 tx=706382750 date='2022-01-28T21:55:48Z' progress=1.000000 cache=141.6MiB(1060232txo)
Jan 28 21:58:40 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:58:40Z UpdateTip: new best=00000000000000000005f8cb80c40350bdbd8652dcf0665036f3ed3909529fcd height=720804 version=0x20e00000 log2_work=93.314219 tx=706385939 date='2022-01-28T21:57:56Z' progress=1.000000 cache=142.0MiB(1063005txo)
Jan 28 21:59:04 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:59:04Z Pre-allocating up to position 0x1200000 in rev02900.dat
Jan 28 21:59:04 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:59:04Z UpdateTip: new best=00000000000000000006c1da699f758005c0a4e52e36a0ce88c931a5b5f96da4 height=720805 version=0x3fff0004 log2_work=93.314233 tx=706389088 date='2022-01-28T21:59:02Z' progress=1.000000 cache=142.0MiB(1063560txo)
Jan 28 21:59:49 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T21:59:49Z UpdateTip: new best=0000000000000000000129a548cfc246b918eea9645a5b6e78e32d8cd8a62444 height=720806 version=0x2fffe004 log2_work=93.314246 tx=706391921 date='2022-01-28T21:59:44Z' progress=1.000000 cache=142.0MiB(1062993txo)
Jan 28 22:01:59 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:01:59Z UpdateTip: new best=000000000000000000034084f00db2b7695ea68944c633bfbc794c7fc666bf19 height=720807 version=0x20600000 log2_work=93.314259 tx=706392841 date='2022-01-28T22:01:50Z' progress=1.000000 cache=142.1MiB(1063765txo)
Jan 28 22:06:37 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:06:37Z UpdateTip: new best=00000000000000000009a9ac7a914d78101bfa6362e7fdd5ad37465bf5eb62d9 height=720808 version=0x20800000 log2_work=93.314273 tx=706393961 date='2022-01-28T22:05:51Z' progress=1.000000 cache=142.0MiB(1063481txo)
Jan 28 22:08:26 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:08:26Z Socks5() connect to rryaasbegshf3ghxi47pl35g7clhy26ea4knvbwibhpfww2voygzyyid.onion:8333 failed: general failure
Jan 28 22:15:50 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:15:50Z Leaving block file 2900: CBlockFileInfo(blocks=103, size=132765408, heights=720706...720808, time=2022-01-28...2022-01-28)
Jan 28 22:15:50 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:15:50Z Pre-allocating up to position 0x1000000 in blk02901.dat
Jan 28 22:15:50 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:15:50Z Pre-allocating up to position 0x100000 in rev02901.dat
Jan 28 22:15:50 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:15:50Z UpdateTip: new best=00000000000000000008aebac97838756feed8adb428b56fe57a563480824c63 height=720809 version=0x20004000 log2_work=93.314286 tx=706396215 date='2022-01-28T22:15:46Z' progress=1.000000 cache=142.5MiB(1067300txo)
Jan 28 22:22:44 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:22:44Z UpdateTip: new best=000000000000000000052254db4f32b004f73ceb60e7ae0d47a8fb6d01f633e5 height=720810 version=0x20800004 log2_work=93.314300 tx=706397666 date='2022-01-28T22:21:35Z' progress=1.000000 cache=142.5MiB(1067397txo)
Jan 28 22:28:44 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:28:44Z UpdateTip: new best=00000000000000000009266b9524f676687fefb5ec0577ea77e9f68a3ae20c32 height=720811 version=0x2000a004 log2_work=93.314313 tx=706399389 date='2022-01-28T22:28:16Z' progress=1.000000 cache=142.8MiB(1069118txo)
Jan 28 22:33:20 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:33:20Z UpdateTip: new best=0000000000000000000362091d029aef26dd91819a7737ab475eff312a71ad88 height=720812 version=0x20000000 log2_work=93.314327 tx=706400417 date='2022-01-28T22:33:11Z' progress=1.000000 cache=143.3MiB(1073249txo)
Jan 28 22:47:39 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:47:39Z UpdateTip: new best=00000000000000000000a2cdf80983a8aa0b63ea8b5835d60309c9f34f965ccf height=720813 version=0x3fffe004 log2_work=93.314340 tx=706402839 date='2022-01-28T22:47:32Z' progress=1.000000 cache=144.2MiB(1081056txo)
Jan 28 22:48:44 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:48:44Z Pre-allocating up to position 0x200000 in rev02901.dat
Jan 28 22:48:44 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T22:48:44Z UpdateTip: new best=0000000000000000000637a189398b0d83c94e38699ccacb77e225d1ba53d698 height=720814 version=0x20400000 log2_work=93.314353 tx=706403374 date='2022-01-28T22:48:30Z' progress=1.000000 cache=144.2MiB(1081273txo)
Jan 28 23:07:18 embassy-2ef08502 f86020a38247[2579]: 2022-01-28T23:05:38Z UpdateTip: new best=000000000000000000060bae026f06f24293ca46bf7c7f89fa08f815d608d220 height=720815 version=0x20e00004 log2_work=93.314367 tx=706405033 date='2022-01-28T23:05:19Z' progress=1.000000 cache=145.6MiB(1092039txo)
Jan 28 23:13:24 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error: Could not connect to the server 127.0.0.1:8332
Jan 28 23:13:24 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:24 embassy-2ef08502 c97087f5efad[2579]: Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
Jan 28 23:13:24 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Bitcoin Core version v0.21.1.0-g194b9b8792d9b0798fdb570b79fa51f1d1f5ebaf (release build)
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z InitParameterInteraction: parameter interaction: -externalip set -> setting -discover=0
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Assuming ancestors of block 0000000000000000000b9d2ec5a352ecba0592946514a92f14319dc2b367fc72 have valid signatures.
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Setting nMinimumChainWork=00000000000000000000000000000000000000001533efd8d716a517fe2c5008
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using the 'standard' SHA256 implementation
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Default data directory /root/.bitcoin
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using data directory /root/.bitcoin
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file: /root/.bitcoin/bitcoin.conf
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: avoidpartialspends="1"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: bind="0.0.0.0:8333"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: disablewallet="0"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: discardfee="0.0001"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: listen="1"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: maxmempool="300"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: mempoolexpiry="336"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: persistmempool="1"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcallowip="0.0.0.0/0"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcbind=****
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcpassword=****
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcserialversion="1"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcservertimeout="30"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcthreads="1"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcuser=****
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: rpcworkqueue="16"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: whitelist="172.18.0.0/16"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: zmqpubrawblock="tcp://0.0.0.0:28332"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config file arg: zmqpubrawtx="tcp://0.0.0.0:28333"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Command-line arg: conf="/root/.bitcoin/bitcoin.conf"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Command-line arg: datadir="/root/.bitcoin"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Command-line arg: externalip="wlw7o2xf6sf4yy2wgvs7adc2k3i5qlakqqk4ples7dsx43x7ndf6ciad.onion"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Command-line arg: onion="172.18.0.1:9050"
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using at most 125 automatic connections (1048576 file descriptors available)
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Script verification uses 3 additional threads
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z scheduler thread start
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z WARNING: the RPC server is not safe to expose to untrusted networks such as the public internet
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z HTTP: creating work queue of depth 16
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z HTTP: starting 1 worker threads
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using wallet directory /root/.bitcoin
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z init message: Verifying wallet(s)...
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z Using wallet /root/.bitcoin/wallet.dat
Jan 28 23:13:25 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:25Z BerkeleyEnvironment::Open: LogDir=/root/.bitcoin/database ErrorFile=/root/.bitcoin/db.log
Jan 28 23:13:26 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:26Z init message: Loading banlist...
Jan 28 23:13:26 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:26Z SetNetworkActive: true
Jan 28 23:13:26 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:26Z AddLocal(wlw7o2xf6sf4yy2wgvs7adc2k3i5qlakqqk4ples7dsx43x7ndf6ciad.onion:8333,4)
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:26Z Using /16 prefix for IP bucketing
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z Cache configuration:
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z * Using 2.0 MiB for block index database
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z * Using 8.0 MiB for chain state database
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z init message: Loading block index...
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z Opening LevelDB in /root/.bitcoin/blocks/index
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z Opened LevelDB successfully
Jan 28 23:13:27 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:27Z Using obfuscation key for /root/.bitcoin/blocks/index: 0000000000000000
Jan 28 23:13:29 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:13:29 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:13:29 embassy-2ef08502 c97087f5efad[2579]: Loading block index...
Jan 28 23:13:29 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:35 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:13:35 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:13:35 embassy-2ef08502 c97087f5efad[2579]: Loading block index...
Jan 28 23:13:35 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:40 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:13:40 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:13:40 embassy-2ef08502 c97087f5efad[2579]: Loading block index...
Jan 28 23:13:40 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:45 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:13:45 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:13:45 embassy-2ef08502 c97087f5efad[2579]: Loading block index...
Jan 28 23:13:45 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:50 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:13:50 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:13:50 embassy-2ef08502 c97087f5efad[2579]: Loading block index...
Jan 28 23:13:50 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:55Z LoadBlockIndexDB: last block file = 2901
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: Loading block index...
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:55Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=3, size=4218518, heights=720809...720811, time=2022-01-28...2022-01-28)
Jan 28 23:13:55 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:55Z Checking all blk files are present...
Jan 28 23:13:56 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:56Z Opening LevelDB in /root/.bitcoin/chainstate
Jan 28 23:13:57 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:57Z Opened LevelDB successfully
Jan 28 23:13:57 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:57Z Using obfuscation key for /root/.bitcoin/chainstate: 29512183616839f7
Jan 28 23:13:58 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z Loaded best chain: hashBestChain=0000000000000000000958676181a13f0cdb71d9d8ccfacd0569050c3494416d height=720618 date=2022-01-27T15:54:16Z progress=0.999440
Jan 28 23:13:58 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z init message: Rewinding blocks...
Jan 28 23:13:58 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
Jan 28 23:13:58 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
Jan 28 23:13:58 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z init message: Verifying blocks...
Jan 28 23:13:58 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z Verifying last 6 blocks at level 3
Jan 28 23:14:00 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:14:00 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:14:00 embassy-2ef08502 c97087f5efad[2579]: Verifying blocks...
Jan 28 23:14:00 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:14:05 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:14:05 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:14:05 embassy-2ef08502 c97087f5efad[2579]: Verifying blocks...
Jan 28 23:14:05 embassy-2ef08502 c97087f5efad[2579]: 
-------------
Jan 28 23:15:31 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:15:36 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:15:36 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:15:36 embassy-2ef08502 c97087f5efad[2579]: Verifying blocks...
Jan 28 23:15:36 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:15:40 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:13:58Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
Jan 28 23:15:40 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:40Z No coin database inconsistencies in last 6 blocks (7936 transactions)
Jan 28 23:15:40 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:40Z  block index          133678ms
Jan 28 23:15:40 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:40Z init message: Loading wallet...
Jan 28 23:15:40 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:40Z BerkeleyEnvironment::Open: LogDir=/root/.bitcoin/database ErrorFile=/root/.bitcoin/db.log
Jan 28 23:15:41 embassy-2ef08502 c97087f5efad[2579]: Error updating blockchain info: error code: -28
Jan 28 23:15:41 embassy-2ef08502 c97087f5efad[2579]: error message:
Jan 28 23:15:41 embassy-2ef08502 c97087f5efad[2579]: Loading wallet...
Jan 28 23:15:41 embassy-2ef08502 c97087f5efad[2579]: 
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z [default wallet] Wallet File Version = 169900
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z [default wallet] Keys: 2001 plaintext, 0 encrypted, 6109 w/ metadata, 2001 total. Unknown wallet records: 0
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z [default wallet] Wallet completed loading in            1490ms
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z [default wallet] setKeyPool.size() = 2000
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z [default wallet] mapWallet.size() = 731
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z [default wallet] m_address_book.size() = 4108
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z loadblk thread start
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z block tree size = 720813
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z nBestHeight = 720618
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z torcontrol thread start
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z Bound to 0.0.0.0:8333
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z Bound to 127.0.0.1:8334
Jan 28 23:15:42 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:42Z init message: Loading P2P addresses...
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z Loaded 67209 addresses from peers.dat  3086ms
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z ERROR: DeserializeFileDB: Failed to open file /root/.bitcoin/anchors.dat
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z 0 block-relay-only anchors will be tried for connections.
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z init message: Starting network threads...
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z init message: Done loading
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z dnsseed thread start
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z Waiting 300 seconds before querying DNS seeds.
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z addcon thread start
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z opencon thread start
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z msghand thread start
Jan 28 23:15:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:15:45Z net thread start
Jan 28 23:16:30 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:16:29Z UpdateTip: new best=000000000000000000013f49e5d9c8b2851cf0274513bf291654a0d3eb2ead2f height=720619 version=0x2000e004 log2_work=93.311737 tx=706052029 date='2022-01-27T16:05:48Z' progress=0.999442 cache=1.8MiB(13481txo)
Jan 28 23:17:16 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:17:16Z UpdateTip: new best=000000000000000000022774f52c8c07126bd96058b72e23bb267a163960e610 height=720620 version=0x20c00004 log2_work=93.311750 tx=706053427 date='2022-01-27T16:06:53Z' progress=0.999443 cache=3.1MiB(22967txo)
Jan 28 23:17:56 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:17:56Z UpdateTip: new best=000000000000000000064dfd8b80cad761422ff773039eff9dbc6e5fb2a904a8 height=720621 version=0x20004004 log2_work=93.311764 tx=706055528 date='2022-01-27T16:15:12Z' progress=0.999445 cache=4.5MiB(33629txo)
Jan 28 23:18:36 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:18:36Z UpdateTip: new best=000000000000000000032d121e70080e8039b61cf6419084daa480e067cf9b68 height=720622 version=0x20800004 log2_work=93.311777 tx=706058478 date='2022-01-27T16:33:06Z' progress=0.999450 cache=6.4MiB(47029txo)
Jan 28 23:19:19 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:19:19Z UpdateTip: new best=000000000000000000009ceca1b251c807644b3b3868b90fd449a54a00959fb9 height=720623 version=0x20400000 log2_work=93.311790 tx=706061230 date='2022-01-27T16:39:13Z' progress=0.999452 cache=7.8MiB(57685txo)
Jan 28 23:20:02 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:20:02Z UpdateTip: new best=00000000000000000004500fec82e3eac7591b9751e90bc51a954a9ba57fb97f height=720624 version=0x20000004 log2_work=93.311804 tx=706062587 date='2022-01-27T16:44:07Z' progress=0.999453 cache=8.7MiB(65396txo)
Jan 28 23:20:39 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:20:39Z UpdateTip: new best=00000000000000000004dcb829b164c7847ef8845b6e52bcbfe0c50679c1a621 height=720625 version=0x20800004 log2_work=93.311817 tx=706065700 date='2022-01-27T17:12:27Z' progress=0.999461 cache=10.2MiB(77489txo)
Jan 28 23:20:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:20:45Z Loading addresses from DNS seed seed.bitcoin.sipa.be
Jan 28 23:20:45 embassy-2ef08502 c97087f5efad[2579]: 2022-01-28T23:20:45Z Loading addresses from DNS seed seed.bitcoinstats.com
dr-bonez commented 2 years ago

docker OOM kills aren't necessarily shown in dmesg

ProofOfKeags commented 2 years ago

Are we able to find that in docker logs?

chrisguida commented 2 years ago

Blocked by https://github.com/Start9Labs/embassy-os/issues/1155 and https://github.com/Start9Labs/embassy-os/issues/1138

ProofOfKeags commented 2 years ago

why does this depend on Start9Labs/embassy-os#1138?

chrisguida commented 2 years ago

It might not. I thought that was the "get resources under control" ticket. If it's not relevant we can remove it.

ProofOfKeags commented 2 years ago

I'd rather not designate it as BLOCKED on 1138. Having the relation link is fine.

chrisguida commented 2 years ago

I had blocked on there because Aiden suggested that we wait until the system-level resource control is finished before revisiting this.

ProofOfKeags commented 2 years ago

I believe this is the result of a race condition with the system clock. If this happens, it will cause a message to be spit into the logs that triggers the autoreindex logic.

chrisguida commented 2 years ago

The case above is not an autoreindex. Notice that only a couple hundred blocks were dropped. This is more likely to do with an ungraceful shutdown. The question is why the ungraceful shutdown occurred. The autoreindex/ntp bug is a separate issue

ProofOfKeags commented 2 years ago

Does that other bug have a ticket or do I need to track it?

chrisguida commented 2 years ago

Just created one now: https://github.com/Start9Labs/embassy-os/issues/1193