BAD BLOCK errors - Githubissues

System information

Geth version: 1.9.10-unstable-f51cf573-20191212 OS & Version: Ubuntu 18.04.3

Expected behaviour

Block is imported successfully

Actual behaviour

BAD BLOCK

Number: 4385565
Hash: 0xeaba95c4970a9a0aaed6432a15ef86906db7b3ec882e1e9834c5a56a9a67c820
...
Error: invalid merkle root (remote: 57bb8a49fdbecb83bc43d1f8a2c5218052775cadf27aec4a91e80870899648de local: 540273554e5ed35860ea6cb1738581957d550d28296d3f53229059d5e76bf564)
...
ERROR[12-20|15:07:39.893] Import error                             err="invalid block 4387500: invalid merkle root (remote: 57bb8a49fdbecb83bc43d1f8a2c5218052775cadf27aec4a91e80870899648de local: 540273554e5ed35860ea6cb1738581957d550d28296d3f53229059d5e76bf564)"

Steps to reproduce the behaviour

From a fully sync'd node (Fast Sync'd > 3 months ago) geth --datadir /data/ethereum --datadir.ancient /archive/ancient export <file>

Wiped chaindata, wiped ancient folder

geth --datadir /data/ethereum --datadir.ancient /archive/ancient --syncmode full import file

Crash ( #20477 ). Restart

geth --datadir /data/ethereum --datadir.ancient /archive/ancient --syncmode full import file

This error. Restart (same command). Same error.

Backtrace

error.log

That repro isn't accurate is it? You also had a non-empty ancients but otherwise wiped chaindata right?

Oh, sorry, you're correct. I updated.

The attached log file is from my re-try to import.

Upon initial import my chaindata and ancients was empty. While importing, ancients data was filling up. There was a crash ( #20477 ) which seemed to recover just fine...hours later, this happened. I tried to restart it again, same error. Can't proceed with import any further.

I received the same error on a different block.

This time it was from the network, not DB import. This is like the 4th time I've been unable to perform a full sync.

1.9.9-stable-01744997

INFO [01-02|09:25:01.777] Imported new chain segment               blocks=52   txs=6101  mgas=366.683 elapsed=8.142s    mgasps=45.035  number=5642714 hash=cf14d6…a808b2 age=1y7mo3w  dirty=1.29GiB
INFO [01-02|09:25:10.015] Imported new chain segment               blocks=53   txs=6103  mgas=366.276 elapsed=8.237s    mgasps=44.466  number=5642767 hash=9166a8…a2e245 age=1y7mo3w  dirty=1.29GiB
INFO [01-02|09:25:18.096] Imported new chain segment               blocks=48   txs=5734  mgas=355.059 elapsed=8.080s    mgasps=43.938  number=5642815 hash=57d3d4…add936 age=1y7mo3w  dirty=1.29GiB
INFO [01-02|09:25:24.994] Deep froze chain segment                 blocks=370  elapsed=4.231s    number=5552829 hash=5f9dfe…c4fc57
INFO [01-02|09:25:26.102] Imported new chain segment               blocks=41   txs=4742  mgas=282.153 elapsed=8.006s    mgasps=35.242  number=5642856 hash=93a873…f260ff age=1y7mo3w  dirty=1.29GiB
INFO [01-02|09:25:27.684] message loop                             peer=d10e9e308d6d5cfa err=EOF
INFO [01-02|09:25:34.199] Imported new chain segment               blocks=46   txs=6151  mgas=305.808 elapsed=8.096s    mgasps=37.768  number=5642902 hash=7c47b9…5a8d52 age=1y7mo3w  dirty=1.30GiB
INFO [01-02|09:25:42.273] Imported new chain segment               blocks=43   txs=5305  mgas=315.946 elapsed=8.074s    mgasps=39.130  number=5642945 hash=216203…9d22f5 age=1y7mo3w  dirty=1.29GiB
ERROR[01-02|09:25:43.742] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 5642963
Hash: 0x3a73459dc39604c4e1152c3f6fbc88137675a7c5663349536d6d875dc2f86800
     0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6dd299de608293de3448eafa0eea82155abf9382b7e9119422863c91e7f65103 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     1: cumulative: 73848 gas: 52848 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x88c5fd75d6d6546505eb8fb683c82cfc7751b575e3e135de35eac6b11ab83e4b logs: [0xc020e76a50] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000010000000000000800000000000000000000000000000000000000000000000000000000000004100000000000000000000000000000000000000000000008000000000000000002002000000000000000000000000000000000000000000000000000000000000000000000000000100000000200000000000000000000000100000000000 state: 
     2: cumulative: 126895 gas: 53047 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x844c0763ae86d6f07a8cbb2ba28bcf3f1e125e3bc4ce2bf23ad9bc935600e8f1 logs: [0xc020e76b00] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000008000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000010000000000040000000400000000000040000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000004000000200000000006000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     3: cumulative: 147895 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xa4c15f866bebb1a449d29a18226282210ae51194122a43d4c4fa102365b2f0c5 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     4: cumulative: 200425 gas: 52530 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd2a0e6073305a2043c6e05d88c3b8f24152333af8b93221e5377113da24047ce logs: [0xc020e76bb0] bloom: 00000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000008000000000002000000000000000000000000000000000000000000090000000000000000000000000000000000000010000000000000000040000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000020000000000000000 state: 
     5: cumulative: 256128 gas: 55703 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xe09e169afc0e263eff60c0b7dca743889a711220adf61fde7f230e2c66ce3373 logs: [0xc020e76c60] bloom: 00000000000000010100000802000000000000000000000000000000000000000000000000000000000000000000040000002000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000004000000000000000000000000000008000000000000000000000000000000000000000800000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000000000000000000040000000000000000800000 state: 
     6: cumulative: 339910 gas: 83782 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xe88248168355e1b1154f0c87773ee1eac10020b53c75925f721ea56dbf455820 logs: [0xc020e76d10 0xc020e76dc0] bloom: 00400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000800000000000008000000000000000000000000000000000000080000000000020000000000000000000800000000000000000000000010100000000000000100000000000000000000000000008000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000020000000010000000000000000000000000000000000000000000000000000000000 state: 
     7: cumulative: 423736 gas: 83826 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x703caffd4ba14780ae123a33a18dcf6a20a9a7d34987c42d3ab2094b32999626 logs: [0xc020e76e70 0xc020e76f20] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000008000000000000000000000000000000000000000000000000020000000000000000000800000000000000000000000010100000000000000000000000100000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000002000000000000000000200000000000200000000000000000000020000000010000000000000000000000800000000000000080000000000000000000 state: 
     8: cumulative: 507518 gas: 83782 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x558523a606eccd55af2a06ef59c567e14e01747ab757a7d3cdacb31f8c206f9b logs: [0xc020e76fd0 0xc020e77080] bloom: 80000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000020000000000000000000800000000000080000800000010100000000000000000000000000000000000000000000000000000000000000000000000080000000004000000400000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000020000000010000000000000000000000000000000000000000000000000000000000 state: 
     9: cumulative: 741239 gas: 233721 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf2a600611f6394bc1f7e6b3ba7d9091d56b25169c2edcfecc3fe301dbfae16cd logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     10: cumulative: 819815 gas: 78576 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x09ca133c64e3239e7480f2f841623053c23016fbc0a6298e863ee32abc85117c logs: [0xc020e77130 0xc020e771e0] bloom: 00000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000004000000000000000000000000000000000008000000000000000000000000000000000000000000000000020000000000000000000800000000000000000000000010100000000000000000800000000000000000000000000000000000080000000080000000080000000000000000000000000000000000000000000000000000000000000000000002000000000000000001000000000000000000000000000000000020000000010000000000000000000000000000000000000000000000000000000000 state: 
     11: cumulative: 842699 gas: 22884 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x33b9bf44d36fea09aa5eaf65a79a7cd335f415d0745fc032651e81969ea3e2aa logs: [0xc020e77290] bloom: 00000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000002000000010000000000000000000000018000000000000000200000000000000000000000000000000000000000000000000000000080000000000000000000080000000000000000000000000000000000000000800000002000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     12: cumulative: 865583 gas: 22884 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xaf7f89a4f394d4a87080a1f9ed1e7509b6bc9ebab1cf728e579b20849bb11df2 logs: [0xc020e77340] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001008000000000000000000000000000000000000000000000000000000000000002000000010000000000000000000000018000000000000000200000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000002000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000010000000000 state: 
     13: cumulative: 888467 gas: 22884 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xfb9f4b08ce870f3109bfbff36189dd82aa6fd76b05ba80c020b07024ba61f454 logs: [0xc020e773f0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000008000000000000000000000000000000000000000000000000000000000000002000000010000000000000000000000018000000000000000200000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000002000000000000000000000000000000000000004000000000000000000000000000000000000000000000000001000000000000000000000000000000 state: 
     14: cumulative: 911351 gas: 22884 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x488315adfa85977f5f63492f490a937fd7c6afba32c326b247f9af955983d8bd logs: [0xc020e774a0] bloom: 00000000000000000000000000000000000000000000000400000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000002000000010000000000000000000000018000000000000000200000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000002000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     15: cumulative: 934235 gas: 22884 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xbf31d6155ac378f8db4a8dd0c444281845dc0f9d27ccff0dfd47dcae43b5bfe8 logs: [0xc020e77550] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000002000008010000000000000000000000018000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000002000000000000000000000000010000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     16: cumulative: 986731 gas: 52496 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x87b794da4bb5e9350a6d95fa602867984983b76c2f1405f032046abfa8707505 logs: [0xc020e77600] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000400000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000010000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000002100000000000000000000000000000000000000000000000000400200000000020000000000200000000000000000000000000000000000000000000 state: 
     17: cumulative: 1021456 gas: 34725 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x232d5099b8b4f3179ee4361a542aa6d345f8969253ac8a4125699122df1bbe82 logs: [0xc020e776b0] bloom: 00000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000800000000400000000000000000000000000000000 state: 
     18: cumulative: 1179548 gas: 158092 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x5a905702ddcc992570a92c6f0f90f09bf55ae2a1e23e20e46c5f24f9fddf534d logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     19: cumulative: 1257701 gas: 78153 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x23a79fe68854f26387630c7118ad2574a198c74f84e03ee4a713575048f82375 logs: [0xc020e77760 0xc020e77810] bloom: 00000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000008000000000000000000000000000000000000000000000000020000000000200000000800000000000000000000000010100000000000000000000000000000000000000000000000040000000000000000000000080000000000000000000000000000400000000000000000000000000000000000000002000000000000000000000000000000000000001000000000000020000000010000000000000000000000000000000000000000000000000000000000 state: 
     20: cumulative: 1287194 gas: 29493 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc9a1cd24d8932a56ead64c9ad9701d9505450f1b397e4ffd672637c593ffef44 logs: [0xc020e778c0] bloom: 00000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000040008000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     21: cumulative: 1537194 gas: 250000 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0xc61663a131b2b42ecb349c835664af6610a65fe739e8e1e6dc57fdf76194a02e logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     22: cumulative: 1713195 gas: 176001 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x860393837d9326fe37ede4ce9d67c1f9991ee7686a522871dad12ab944fb268f logs: [0xc020e77970 0xc020e77a20 0xc020e77ad0 0xc020e77b80 0xc020e77c30 0xc020e77ce0 0xc020e77d90] bloom: 00000000020000000000000000000000000000000000000000000400000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000008000000400000000000000000020000000000014000000800000000000000000880000110000000000000400000000000000020000000000000000000000000000000000000000000004000000000000000000008000040000000000000000400000000000000000000000002000000000000002000000000000800000000000001000000000020000000000000000000000000000000000401000000000000000000000000000000 state: 
     23: cumulative: 2269535 gas: 556340 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xcecde21a58f2b93e79c43207dfcb4c15067e653b23a7b7aa220233d558c046de logs: [0xc020e77e40 0xc020e77ef0 0xc022340000 0xc0223400b0 0xc022340160 0xc022340210 0xc0223402c0 0xc022340370 0xc022340420 0xc0223404d0 0xc022340580 0xc022340630 0xc0223406e0 0xc022340790 0xc022340840 0xc0223408f0 0xc0223409a0 0xc022340a50 0xc022340b00 0xc022340bb0] bloom: 00800008000000010000000000000000000080000000000000000000000000000002000000000010000000000000000000000040100004002040000000200000000800000000000000041008000000000a000010800004000000000022000000000080000800000001000000000000000000000004400000000000100000000000000000000000004000000c0000000000000040048000080801040000000000020000000000000080000000000000000000000080002000000100800000000000000002010000000000002040000400000080000102080000000000000000000010000000020000800000000000000000000000000000000200002020004000 state: 
     24: cumulative: 2291928 gas: 22393 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc649333bb53f8c0319dd8b12a0e022e176ae50dbf87c66b9bd2e479b5886fa6b logs: [0xc022340c60] bloom: 00008000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000400000000000000000000000000000000008000000000000000000000000000000004000002000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000004000000000000000400000000000000000000000000000000000000000000000800000000000000000000040 state: 
     25: cumulative: 2314037 gas: 22109 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc783a7cea3131ab6821b62a226fcd7a2265ddb687d462517892b2b1a3148c595 logs: [0xc022340d10] bloom: 00000000000000000000000000000000000800000000000000000000000000000080000000000000040000000000000000000000000000000000000000040000000000000000000000000008800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000080000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000080000000000 state: 
     26: cumulative: 2336430 gas: 22393 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x9d8fb35cad9bf3ceedc5183f824b5274ec118e612bb206bcf7ce63821b464d0e logs: [0xc022340dc0] bloom: 00008000000000000000000000000000000000000400000000000000000000000000000000000000000000000200000000000000000000000000400000000000000000000000000000000008000000000000000000000000000000004000000000000000000000000000000000000004000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000002000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000040 state: 
     27: cumulative: 2358588 gas: 22158 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xe6652ce2ffe6a62153e7dd8495cc7ce773d0271ce1c88e19a0a0dbf3954a43d3 logs: [0xc022340e70] bloom: 00000000000000100000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000020000000000400000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000004000000100400000000 state: 
     28: cumulative: 2379588 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x22936747575ad4a725665ddd98bf9845b87a8dd385e731cb8c8a466e3a2a420f logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     29: cumulative: 2416061 gas: 36473 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0c0f1eeff0cd4f132a5f5be0870f3d838ffcdb9e0e26a007c110e5fdc3d9c19c logs: [0xc022340f20] bloom: 00000000000000000000100000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008c00000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000200000000000000000002000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010 state: 
     30: cumulative: 2449421 gas: 33360 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x238f56be77dba1faefbac4dcd585b2c1c795d020fd082f6a2d1d887215a1bd5a logs: [0xc022340fd0 0xc022341080] bloom: 00000000000000000000000000000000000001000000000020000000000000000000000000000000200000000000000000000000000000000000000008000000000000000000000000000008000000000000000000000400000000000000000000000000000000800000000000000000800000000800000000000010000000002000000000000000000000000000000000000000000000000000000004000000004000000000000000000000000000000000000100000000000000000000000000000002000004000008000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000 state: 
     31: cumulative: 2627194 gas: 177773 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0cda5cc0cccbb0ba65afbc072baff6bfeec2aa1ec761d21d2f4d3a2b1dc8a4f0 logs: [0xc022341130 0xc0223411e0] bloom: 00000000000080000000000000000000000200000000000000020000000000000004100000000000000000020000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000040400000000000008000000000000040000000000000000000000000000000000000008000020000000000000000008000000000000000000008200000040000000000000000000000000000000000000008000000000000000000000080000000000000000 state: 
     32: cumulative: 2676683 gas: 49489 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x24214e267cae4cc51098306f9d5cb721ce13ccc2d4ce3ef83b70b3c4a2f72ef3 logs: [0xc022341290 0xc022341340] bloom: 00000000000000000000000000000000000001000000000000000000000000040000000000000000000000000400000000000000800000000008000000200000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000010000000800000000000000000000002000000000000000000000000000000004000000000000000000000000000000020000000000000000000000000000000000000100000010000000000000000000000008000000000008000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000 state: 
     33: cumulative: 2726172 gas: 49489 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x9e0f49d451a51cd3ce53df12df119e0b72fd4b3358a118454f7fb365a815d417 logs: [0xc0223413f0 0xc0223414a0] bloom: 00000000000000000000000000000000000001000000000000010000000000040000000000000000000000000400000000000000800000000008000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000800000000000000000000002000000000000000000000000000000000000000000000800000000000000000020000000000000000000000000000000000000100000000000000000000000000000008000000000008000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000 state: 
     34: cumulative: 2775661 gas: 49489 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x43166b35a0d76b03a5098c2541fdd4100e88cd4b8bc6ebdaf86a928b4321a31e logs: [0xc022341550 0xc022341600] bloom: 00000000000000000000000000000000000001000000000000000000000000040000000000000000000000000400400000000000820000000008000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000800000000000000000000002000000000000000000000000000000000000000000000000000000000000000020000000000000000040000000000000000000100000000000000000000000000000008000000000008000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000 state: 
     35: cumulative: 3403377 gas: 627716 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6f5e8c3f53eb21e6e97fef9abdfee38fbcf10e991eab01bb864fe719be76f8a6 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     36: cumulative: 4531714 gas: 1128337 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x87aa5e421f0386954cff74ab9f3f1f5bfc746c7266b814aa4ca81e2a7294ebba logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     37: cumulative: 5660051 gas: 1128337 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xfd91ed264ca65b6b4e26d4bf222577c3f795accde2bfb8c630bae5d02838478e logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     38: cumulative: 5697241 gas: 37190 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x24d5b48f8b03963248c5c00a600d6f991216fdf5aa8353fd2bd592f31c240e6b logs: [0xc0223416b0] bloom: 00000000040000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000008000020000000000080040000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000010000000000002000040000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     39: cumulative: 5740150 gas: 42909 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x60c5d9161ace05fd9594ab3e02b8f5176e7370ba8700f1d7f1d1b20355e59c17 logs: [0xc022341760 0xc022341810] bloom: 00000000000000000000000000040000000000000000000000000000000000000000000800000000000000000200000000000000000000000000000000000000000000000000000000000000000000400000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000001000000800000000000000001000000010000000000000200000000000000000000000000000000000000000000000000000000000000000000000000002000000002080000000000000000000000000002080000000002000000000000000000000000000000004000000000000000000000000000008000000000000000000 state: 
     40: cumulative: 5784447 gas: 44297 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf9f36322433ae6a72021744fe8c1e169c0ab377640b5b8d60e30f874c7c92d9e logs: [0xc0223418c0] bloom: 00000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000004000000000000000000000000000000000000004000000000000000000000000000000000000000000000000100000000020000000000000 state: 
     41: cumulative: 6049446 gas: 264999 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x251523f85d56741b228246043af58257b3fe5eccdca31f3d2684ddab8e6a4b62 logs: [0xc022341970 0xc022341a20 0xc022341ad0 0xc022341b80 0xc022341c30 0xc022341ce0 0xc022341d90 0xc022341e40 0xc022341ef0] bloom: 0010000000000000000400100000000000000000002000000000010000000000400000100010800000000000020400000000000000800000000000000000010202080000001000000008000a200800020000000000200000000000000000000000000000000000020080000000000000000000000800004000200010000000000020000000080000000204000400400002000000000000200000000000000000000000000020000082000000000400000000000000000000000000000000000000000002000000000000000000000020000080000000000000000004000000000008000000000800000000040800000000000200500000100100000000000400 state: 
     42: cumulative: 7335837 gas: 1286391 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc8381080f91b8f5af741dfc1a13ab23584689e4d915ae3288070d596bf0a3056 logs: [0xc0228ca000 0xc0228ca0b0 0xc0228ca160 0xc0228ca210] bloom: 04000000000000000000000000000000000000002002000410080000000000080800010000000000000000000000000040000000000000000000000000000000000000000000022000000000000000400000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000100000000000000000000400000000000000000000000000100000001000000000000000000000000080000000000000000000004000000000000000000000004000000000000000000080000004000000000 state: 
     43: cumulative: 7385196 gas: 49359 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xef3f3b6b5acf501e7b610edb2399fda04cd4f7894c2f40bef25b42b3ad925595 logs: [0xc0228ca2c0 0xc0228ca370 0xc0228ca420] bloom: 00000000000080000001000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000020000000000000000400000000000000000400020000000000000000000000001000000020000000000000000400000010000200000000000000000000000000000000000000000000000000000000000000000000000000000000000020000010000000000000000000000000080000000000000000000000000000010000000000000000000400000040000000002000000000020000000000000 state: 
     44: cumulative: 7434555 gas: 49359 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xdfe0c3aba09bfcec14c3e2ce5a250ad396daccbb40cdfcb9df8d8db46b656742 logs: [0xc0228ca4d0 0xc0228ca580 0xc0228ca630] bloom: 00000000000080000000000400000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000001000000020000000020000000000000000000000000000000000000000000000400000000000000000400020000004000000000000000000040000020000000000020000400000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000020000010000000000000000000020000080000000000000020000000000000000000000000000000000000000000000000000000000000020000000000000 state: 
     45: cumulative: 7455555 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x214a5a20e21df83960937a4dfeaad543662f6a52a16be246becbb89ac7f4e318 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     46: cumulative: 7504914 gas: 49359 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7e7e8fa9ed56864aca0fbb8e93572dfa1d8463499e4e674f5f6d1f6e42631652 logs: [0xc0228ca6e0 0xc0228ca790 0xc0228ca840] bloom: 00000000000080000000000000000000000000000000000000400000000000000000000000000000000000000200000000000000000000000000000400000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000400000200000000000400020000000000000000000000000000000020000000000000000400000000000200000000000000080000000000000000000020000000000000000000000000000001000000000000000024000010000000000000000000020000080000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000 state: 
     47: cumulative: 7529938 gas: 25024 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0xb971c0d131c3e359d6f03c4a7e36ca305b6822cdf8217fe9fc6feb3a85a6f4b7 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     48: cumulative: 7613738 gas: 83800 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xfec8304f78b19a20aade895b9a99d7fc73796ced7702111ab623fcb0400be3ac logs: [0xc0228ca8f0 0xc0228ca9a0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000400000000000000000000000000000000008000040000000000000000000000000000000000000000000020000000000000000000800000000000000000010000010100000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000004004000000000000000000020000000010000000000000000000000000000000000000000000000000000000000 state: 
     49: cumulative: 7693738 gas: 80000 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x3852a3c30f97260dc10321b7e9c1f4449d116abb46587b3aed5614e94e66694f logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
     50: cumulative: 7777538 gas: 83800 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xa5ab19109d0c16f078e5ad5d834937f84457d0dd9295f6c3c39e44e02cb3ec86 logs: [0xc0228cabb0 0xc0228cac60] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000400000000000000000000000000000000008000000000000000000000000000000000000000000000000020000000000000000000800000000800000000000000010100000000000000000000000000000000000000000000000000000000000000000100000080000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000004000000000000000000020000000010000000000000000000000000000000000000000000000000000800000 state: 
     51: cumulative: 7861338 gas: 83800 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x3b81eea85fea7508b7afc5128c31d0b078d58c8e20c1ca2d5d02745d1ad73e57 logs: [0xc0228cad10 0xc0228cadc0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000400000000000000000000000000000000008000000100000000000000000000000000000000000000000020000000000000000000800000000001000000000000010100000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000001000000000000000000000000000000000000000000000002000000000000000000000000000000004000000000000000000020000000010000000000000000000000000000000000000000000000000000000000 state: 
     52: cumulative: 7898909 gas: 37571 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x4fb951243885f3206b9d515422c00f6c6eed494eb495bcca139e610614b46d63 logs: [0xc0228cae70] bloom: 00000000000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000100000000000100000000000000000000000000000000000000000000000000000000000000000000000010008004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000000000000022000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000 state: 
     53: cumulative: 7936668 gas: 37759 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xccb26798f6b56002a5d0db4fb3e8020e18bde0a05f766c9270e4217b8211d5bc logs: [0xc0228caf20] bloom: 00000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000010000000000000000000000000000010000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000400000000000000000000000000200000000000000000000000000000000000000 state: 
     54: cumulative: 7957668 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x1410efe0dedb319c30384024bdf1a1024d52440889460bcff76fe09095bfec3e logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 

Error: invalid merkle root (remote: ba1a2ea6c8fed949facf7b843141e1d93f0ab66202508d96a2efcba216062a70 local: 02be8ade558a4698a8d77d8608feb3cd52a126a72b976c9ede37b94097ea027d)
##############################

WARN [01-02|09:25:43.742] Synchronisation failed, dropping peer    peer=75f9b28001f63a0c err=timeout

Again. Block number: 3793414

USBHDD_BADBLOCK.txt

The difference between this run and the other runs is that I moved the freezer folder to a locally attached USB drive. Didn't help.

I'm going to buy a new SSD and try again after it comes...

I suspect your disk is toast...

Alright, I tried to perform a full sync on a 100% different workstation. Windows this time.

INFO [01-04|21:14:16.153] Imported new chain segment               blocks=66    txs=14378 mgas=518.846  elapsed=8.015s    mgasps=64.733  number=4761891 hash=a31073…7a530b age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:14:24.159] Imported new chain segment               blocks=46    txs=9830  mgas=367.454  elapsed=8.006s    mgasps=45.897  number=4761937 hash=13174c…ce35dc age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:14:26.600] Deep froze chain segment                 blocks=473   elapsed=447.380ms number=4671950 hash=8eb710…683dee
INFO [01-04|21:14:32.209] Imported new chain segment               blocks=41    txs=7751  mgas=314.650  elapsed=8.049s    mgasps=39.088  number=4761978 hash=81bf2f…3450a9 age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:14:40.277] Imported new chain segment               blocks=63    txs=12891 mgas=495.303  elapsed=8.067s    mgasps=61.392  number=4762041 hash=770cbc…613358 age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:14:48.286] Imported new chain segment               blocks=87    txs=15893 mgas=662.162  elapsed=8.009s    mgasps=82.677  number=4762128 hash=dd87b7…90ad63 age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:14:54.264] Imported new chain segment               blocks=61    txs=11342 mgas=474.084  elapsed=5.977s    mgasps=79.313  number=4762189 hash=c51177…ccf70b age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:15:02.553] Imported new chain segment               blocks=41    txs=7747  mgas=320.175  elapsed=8.260s    mgasps=38.760  number=4762230 hash=a8ef01…50ca95 age=2y3w5d    dirty=2.00GiB
INFO [01-04|21:15:10.574] Imported new chain segment               blocks=83    txs=14275 mgas=658.683  elapsed=8.021s    mgasps=82.119  number=4762313 hash=03fec1…2abd66 age=2y3w5d    dirty=1.99GiB
ERROR[01-04|21:15:11.061] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 4762319
Hash: 0x41e7e12dd0e1cd2a0d8d6a993448dd12111c0a9dde81c36c97d5dda90c4749e1

Error: nonce too high
##############################

WARN [01-04|21:15:11.061] Synchronisation failed, dropping peer    peer=51d8940abf23af67 err="retrieved hash chain is invalid"
WARN [01-04|21:15:12.342] Synchronisation failed, dropping peer    peer=e12e3d7e68fb81cc err="action from bad peer ignored"

Error is different...?

When exiting I received this:

INFO [01-04|23:38:46.470] Writing cached state to disk             block=4762318 hash=cc961c…e1830f root=b783a9…77cbec
ERROR[01-04|23:38:46.473] Failed to commit recent state trie       err="leveldb/table: corruption on data-block (pos=683158): checksum mismatch, want=0xa7c23af got=0x99f2513d [file=088662.ldb]"
INFO [01-04|23:38:46.474] Writing cached state to disk             block=4762317 hash=e03c0d…505d7a root=72090f…34732f
ERROR[01-04|23:38:46.475] Failed to commit recent state trie       err="leveldb/table: corruption on data-block (pos=683158): checksum mismatch, want=0xa7c23af got=0x99f2513d [file=088662.ldb]"
INFO [01-04|23:38:46.475] Writing cached state to disk             block=4762191 hash=e6c31a…bd59af root=919289…85476a
ERROR[01-04|23:38:46.476] Failed to commit recent state trie       err="leveldb/table: corruption on data-block (pos=683158): checksum mismatch, want=0xa7c23af got=0x99f2513d [file=088662.ldb]"
WARN [01-04|23:38:59.681] Header broke chain ancestry              peer=aa447ecec6052875 number=4777491 hash=718a20…62a64b
WARN [01-04|23:39:01.123] Header broke chain ancestry              peer=aa447ecec6052875 number=4778284 hash=786bf4…b567a9
WARN [01-04|23:39:03.677] Header broke chain ancestry              peer=aa447ecec6052875 number=4780953 hash=21d68b…a5dc38
WARN [01-04|23:39:04.489] Header broke chain ancestry              peer=aa447ecec6052875 number=4781910 hash=7f8079…fe58ed
WARN [01-04|23:39:06.988] Header broke chain ancestry              peer=aa447ecec6052875 number=4784595 hash=708542…401206
ERROR[01-04|23:39:10.287] Dangling trie nodes after full cleanup
INFO [01-04|23:39:10.287] Blockchain manager stopped

After restarting the node I got this:

INFO [01-04|23:45:51.575] Opened ancient database                  database=D:\\Geth\\Freezer
CRIT [01-04|23:45:51.583] Failed to store chain config             err="leveldb/table: corruption on data-block (pos=683158): checksum mismatch, want=0xa7c23af got=0x99f2513d [file=088662.ldb]"

I purchased brand new SSDs yesterday, re-installed Ubuntu 18.04.3 LTS fresh.

I then ran a full sync and it failed, again.

1.9.9-stable-01744997

INFO [01-12|05:10:54.323] Deep froze chain segment                 blocks=9531  elapsed=5.333s    number=2859097 hash=93eafd…57fc3d
INFO [01-12|05:10:56.805] Imported new chain segment               blocks=1062  txs=7743  mgas=230.886  elapsed=8.003s    mgasps=28.849  number=2950152 hash=e26718…2d03c8 age=3y2w6d    dirty=298.83MiB
INFO [01-12|05:11:01.009] Imported new chain segment               blocks=986   txs=6882  mgas=223.298  elapsed=4.203s    mgasps=53.118  number=2951138 hash=ee7e89…2119ef age=3y2w5d    dirty=298.85MiB
INFO [01-12|05:11:07.719] Storing bloom trie                       section=89 head=76d4cdfdab94e1caea622afbd11154a4c3351e507576907a8ce3e6e7d51c3982 root=d061b34a42294ae97ea6d3cd58c053f981d726de5e892830b554f5561790494e compression=0.053
INFO [01-12|05:11:08.447] Storing CHT                              section=89 head=76d4cdfdab94e1caea622afbd11154a4c3351e507576907a8ce3e6e7d51c3982 root=a4883b10d5d99a985925ccc256754cc583b82241e1a29a071c5ec1e48bb800f3
INFO [01-12|05:11:09.052] Imported new chain segment               blocks=558   txs=3265  mgas=112.902  elapsed=8.012s    mgasps=14.091  number=2951696 hash=a59579…684075 age=3y2w5d    dirty=299.96MiB
ERROR[01-12|05:11:11.615] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 2952165
Hash: 0xe3821eeacc776d2f5c5ce6509c2dfd6f2f1711973f758365d5b044d1a5be6c7e
     0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2766a974e0f0c46322eadd24f20f75847457a25b9b295cc0a541ead837ba50f2 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 1409e5827cf5b2ecf99e447ac5c58e89a655053bc897471c4e2c1833e48e0fff
     1: cumulative: 86864 gas: 65864 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x874411e7c0ed99b451f2248556cd038ae4ec9cba62b766ac6a7b8e90afd26088 logs: [0xc0525cc630] bloom: 00000000000000000000000000000000000000000000000000000000000020000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000120000000000200000000000000000000000000000000000000000000000000000000000000000 state: ac6438485489effbdee6150e3f0d7c3094373d27e1326c0e90674fa1776dc3bc
     2: cumulative: 107864 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x60ad0e3ce7a0bbeb03afe94bd642934bf245878a65df742a5b4a354923f31293 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 75b368b78eaff54414e983620bfc43fb0be4360f7f3b5c7c96016a781a269422
     3: cumulative: 128864 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x90a60a0996ff5eb47e943861d332325fdf28ceb5e248c3758b24667086b12aa2 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 704dfb1295e0dd2ec372d8a91ef701eceb42ac78cfe74c46a757c28f53e99413
     4: cumulative: 149864 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xbf5ab50cadc0054c5b7720911b5746c74defd5617f2e133fd28d80eb9eecbb27 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 83c846cd617267b510a0d5130ead4d72818cbd6546ac03179c15b951b07e8dd3
     5: cumulative: 170864 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc90560d72e6055bef162efe7c75631ed639080539a1d0ba21fdb23da24c6bda8 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 5aca7fa2a54ceccf78a1981ad2400f342440debdc28918eeddf39a0162bff746
     6: cumulative: 191864 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd16a73540ae98c6ed6ca41f9f5f3bbe40c2054cf161c436daedb974af4806d9c logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 49d7098d71ef7167b9330c9612b36de9115ec499918cf857457fbcfcfa93dc1a
     7: cumulative: 212864 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd213ea5d1b760fb8a082a13bef3dc2dd4939d89adb38dad65b07779be43bbb0a logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 1aaa89d4b7352f6f07ad922b4014d88adbd04372c83bb32ec0ff92a94241a7d8
     8: cumulative: 365354 gas: 152490 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf7884ab6f7afa9ce16e6587566256fecefd0b3ff57eb20ec8d36875f6a630092 logs: [0xc0525cc6e0 0xc0525cc790] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000100000000800000000000000004000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000040100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000040000000000000000000008000000000000000000000080000000000000000 state: 45a5dde5f8a7635b99a3db595e525bdd5c1fea6a467e3ca4fd0222cfe2069332
     9: cumulative: 386354 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x004b72e6345f847942c481807e68a328936bf2d97959fb8fdd627e120a55b526 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 43039e130d08b7a621d6afa65e868488143f05e1d87e4f93854693b51dc59678
     10: cumulative: 416770 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x05180be3ccaba7f8b5ce6739c8208ac4cbd0103376f7d59d2adfa5f511950ac8 logs: [0xc0525cc840] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 06f368c0eb68d0bc57c659761c1713bffbeee849e23939a4e9879383fe0f9e29
     11: cumulative: 447186 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x07e0f18b3ed4d1e1775c1c543a1543b36cecd5bac667235258716dd18621b727 logs: [0xc0525cc8f0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 46facf42cb811da9bb657197462fbecbb6d9f40fb8bef5e414a89eb72c987ff0
     12: cumulative: 477602 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0f28fab5eb609cfe1e77156da759188578f7d45c0e3a08feb54f25e2060d4184 logs: [0xc0525cc9a0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 8b9fa0cacb847956f6304dc881e975338f8a5b41d77197b73001ae7bcf6fe89c
     13: cumulative: 508018 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x28f279a03d609a0dbf1da99a2f99ae647f6058083cb148e47095fa4ad01508f2 logs: [0xc0525cca50] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 7d8878fdb133f808fe95bf4c1e070c413c6013762bf7ef529b8feae7108aded9
     14: cumulative: 529018 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2c9f8fcd708248c0085537da6933975190bd388d6294d9141caf412d3dd093f1 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 9d59d2b0c8b2796bca5a66d710d4418053942045151810fa83f58c83f78e3e6d
     15: cumulative: 550018 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2cc2bacc1790900b129f8eb62f9cd629f7c4877d7bc7798a69e40c3ba7f3a874 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 8307ff3ebb61c86c4824c8bdb3361d57fc2eb85fd5e9723821ce30c1b4446fea
     16: cumulative: 580434 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2e2d63e3e4fa88d3d564da171dc69ed630f3365f9738ae53b42db3fc506eea2d logs: [0xc0525ccb00] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 4f3bc2c9117b3f7436e7a1969a56651523cded57b0a68c80b9c49edf4759fafd
     17: cumulative: 601434 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2eb0a1a61cd32aece92bf9b16c82cfcbbae8480d2b852f3139bcb9a00f0b3fde logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: a3e33287e9e73404153ed87e313f3d44c646c76562c95dc460afd71fe0eee042
     18: cumulative: 631850 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x3b163efa024972364394e1e2eca34a0fa66a1ee94405cf6d8d17fa96984ddf60 logs: [0xc0525ccbb0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: e6b8b5a55b5e7ba6df0894ee8b5f64ee424c2bbb245c2521077f85bcb0f7077d
     19: cumulative: 662266 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x3f3c52b860ceab2c777c3a018af55c16f8a5b4e9e5571c0b4174edac4b54c83b logs: [0xc0525ccc60] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 4f72126ebbde0aa2788306842a8b57bb4ec96899f312dea432e3b4de33164fe6
     20: cumulative: 696736 gas: 34470 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x4284407bcc87f6e5ca65d73cd334a6acdddf8ed2cadf9e0c8bb18c82470e6e89 logs: [0xc0525ccd10] bloom: 00000000040000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 7cbc20d9fd2f6aad2cfdab46bde76c9ac4eeb8c7921c7dc4814262082ca122da
     21: cumulative: 727152 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x4360b19a7b2a9eb35ba06a7bcef6c0bee080a2eecf689438c6f87553ee6518ac logs: [0xc0525ccdc0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 7b5cfab9d2ee5ad951886c6ac7709992075a4ac4eb5e3470851b103a19f5c958
     22: cumulative: 748152 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x4c7ff012801c9b94657e939291d3be394aee3f7ad613ccd2fbf4be746fbea451 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: a26547cd0c84ebc3e05dcfad5e79db9499b3bf3fc7d6b91328cc003f397667bd
     23: cumulative: 769152 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x546eff5d67f64f5085472365220afec5d623e78e981d1fae06c67b292c7521cc logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: da11b701765f35e88851a9c1df950bf02b73840c86faef5b6a7e05b2470c41cf
     24: cumulative: 799568 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x589f48526b34c4bd8e375870dbfe54c34fd6aa69ef63327798dea8261a624e8c logs: [0xc0525cce70] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 931d49231c5957ead72830c5a2e1473ec90ba50654174004c361f5705802f7e4
     25: cumulative: 829984 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x5ac7a209995677b2bfe08ece6171852fbe14fd57523ad64ff9b8c73662770a1d logs: [0xc0525ccf20] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 52f045fd30d45e2fb1eead1ced781327cfa1944a454b20c2d58f59f33f32d28e
     26: cumulative: 860400 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x5bd2219c752bae7f14e46d9561b235ef4f6d7b418e224b9d46876444c41c28c3 logs: [0xc0525ccfd0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: ee8422e51fbed1db966544aea1ba92be63f6854dcbcdc7d642daa3783d28ba05
     27: cumulative: 890816 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x65879f37534fc8a916e7f0bb268f74b7f1c16e4a8c63bfa8c484e9c16fc7c48e logs: [0xc0525cd080] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 4caa488d2bdd2ceca08e8336b9aec8265cf70cf796ecb1a74b71e70e085e117f
     28: cumulative: 921232 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6b6b941cc70fdd5e350fb2e6b79edcf5075a50b7a8f1e13d66f18a4b5cd52e77 logs: [0xc0525cd130] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 49aa78ecefb38989dc04499ab7d6bc697ab137847009ebb351361213f0554087
     29: cumulative: 951648 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6d9fb004de8e774352fb8e2a3d3fd007446bf5795d5afe101a075ab7ea82bcc3 logs: [0xc0525cd1e0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 578c05bf559451e5204c41aee4f02e4c60c25fc524f7e36cf199624e8235c3eb
     30: cumulative: 982064 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6e77df2b37791d1509f422fbaf8b5ea053d3f14afdbc0db0f2047cc1caab459f logs: [0xc0525cd290] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: f0dd33d769ba3fef616e53d0428692e0969806d93bd40b012cd1a9206de6ab0d
     31: cumulative: 1012480 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x71f993c8db92d4c00e22dbe18bb5dc0235fcce4339612ff3fdbf973f3f10d000 logs: [0xc0525cd340] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 9ad8945cda0bf3bbd4baf08d49a76b1ff2ada61dcf7d826dfd776896dc59d99c
     32: cumulative: 1033480 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x75338922aec17dfb68a8a096c9dde3281964097b7e4fdda08a2a876275926208 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 8fa65c71ec510fbf239bc0890caa0c7aa48cb34b905295b3d8dade4cf1764324
     33: cumulative: 1063896 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x75efd63b54077156ddf6d31b3543dca3de89d1dc8d452e5ae8aec25c5530b73c logs: [0xc0525cd3f0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: ddd96c4f787f44149b58bf247b20678130b1e8095d3d1d411ac346ce568e73f8
     34: cumulative: 1094312 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x764ba98bee8bd81e1ff45985d60e9e20e4ec9c8fe8956408dad560677360fbc4 logs: [0xc0525cd4a0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: c5612433b17f90915d5e402a256643bb2d00d1a96ee3931f91928c2ccdc714eb
     35: cumulative: 1115312 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7c2c3a498573506f1a6c2ba136e7fce754b2a6e5851a816de40dffa737db931d logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 6fba194e2ce5f83889de05eaa3d2bcee15086301232fbd3651c944a7fa79a39d
     36: cumulative: 1145728 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x8740821e4ff3907efaf0dc57967c5caaa1bae64b28d477d64c5e496e6f135d69 logs: [0xc0525cd550] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 24b6c26672b2d60f3a9b7154e1683eee8c82fefeae1dcc486b8e7c8161cc3cf4
     37: cumulative: 1176144 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x91bbc946b58c545866ce27afa78eaa0e477361401d9ac11029ffef71702c281d logs: [0xc0525cd600] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 1e443b73ae1620e14327183ff0b325f9fde3ba6b3c07671c6a08d58e921068c1
     38: cumulative: 1197144 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x96803c37de9f44b7d5c9077e601785c3dbc745a2d4f52c5d4f01234b5c0e1da2 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 9f3cec44c4a96af0bf6b60cfb55c843e33be8431367095f5fc46037972f0e53b
     39: cumulative: 1227560 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xa030c4c4e7201c9b9064fa487c1d93822bd2acde38741c6ae4d2a6bed2bf2408 logs: [0xc0525cd6b0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 99b5384698371cb8095fb6b3966cd73bf5fa4ee08ab54218c7796de09bc68ef5
     40: cumulative: 1257976 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xa1ba79cc0f663c056c11e35ad548f4d7f95ba772f6b74d184aebd145c77d2671 logs: [0xc0525cd760] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 76bee72816e7f3b8103fa0cf97c266af35331e4b53d288e2d9ac8e7028ac2043
     41: cumulative: 1288392 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xa86932f948dc515fba7838bf9061e04affbb86294ce5c4ceb38544e75e9ff7c2 logs: [0xc0525cd810] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 933a915878d278c4f8f9c02c53a6f61189dca27af2bb3a1233baa1ad7d7ea0ca
     42: cumulative: 1318808 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc6192f7808a900c808561e93c14b912b6349f2f088fcfccab774d33952a167dc logs: [0xc0525cd8c0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 09a2393c6d3a2195b1c89ca6d4ca45101e7ef6a4790228600c236c750299d34d
     43: cumulative: 1349224 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd6c81388d4b136f25a0e9a6db444e6f4f90e2d9c5a232232eec99f0ba77c2dff logs: [0xc0525cd970] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 3957cb30992b94e72da39d889e4a5dc519628a5333e32e90dd4a6233038835fc
     44: cumulative: 1379640 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd99dca9041354db018128f3725e3d3146277b674bf60fb4ada4f680a68520712 logs: [0xc0525cda20] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: bca2bff53de30b83f1923210aba704a5262502336450c59abacee59ce705eda0
     45: cumulative: 1410056 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xda59f56091b10fc4e87d15d2eb61ac52c65401ebe6fbd760db6b2101e893be7c logs: [0xc0525cdad0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 2e576ba048b4b47bf19c3aaa145d5b3ea88a8cf1ac56365e82e6eedcb68d6b2f
     46: cumulative: 1444526 gas: 34470 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xe2447607f60fd836303959c12545cd0fde02da5fed8a8f49d0f539bf3cd1dcd5 logs: [0xc0525cdb80] bloom: 00000000040000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: ef9b06c18aeb7973bd753d0535ba588cb46aabec5fb93b7a0b9bbbc5708bd00b
     47: cumulative: 1474942 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xe5c1bd1842315b085b1e9888514ee8c0452a8e98f87d6364231c6130b86d899c logs: [0xc0525cdc30] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 2043595e6f5366b92c0decaf57335f83fb8feacdb4497d64205c3f5876cb90dd
     48: cumulative: 1505358 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xe985af519d549d77672ae90fd65782368ad3893cea0603b14cfb57d429f3c168 logs: [0xc0525cdce0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 87608e11a0d5b2acf3ef95b8ecf5fbb7eae6180b626f27f8eace2fa27699e55c
     49: cumulative: 1535774 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf502ace00d411ecbbadd77d1e8814fad4827210afe6425ed6a4a77180cd54b75 logs: [0xc0525cdd90] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: d7088cf90494d3b6e7ae153aa8fdaf8b4f54aa59d95a6db4887448e8ea124b8d
     50: cumulative: 1566190 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xf552b73406dd68e4594e23243001965fa5f3b5e586eb0ea4c802af1f7faaa5fc logs: [0xc0525cde40] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: fbd6dbb3f0e06da8cc0823be9065a93f2d96e7550def586faec4da624a6cf5d4
     51: cumulative: 1596606 gas: 30416 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xfe8a5ff3df5b7161bd719fdefa42654fa38dc89f98b59f14cce44739ea9449ce logs: [0xc0525cdef0] bloom: 00000000000000000000000000000002000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 96c772ae824974ac7a6f3cea8b201e06f81e4bbea0ad6e4109edfbb3827ac08c
     52: cumulative: 1617606 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xffb1df7f66cbc116ba5fedfe3f0d6854d9bdd0c3f3bfe135595563d3e83cc3f1 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 939f68e942915b82e599066ee87a89de0f925df182fae0182852555aa46a7c10
     53: cumulative: 1638606 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xd95b8c2a41b156bf8471b798350e7fca119916dd455f845a7ac7f1d5927b9a23 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 5bae5d0a42f13b3db6d63d4b1772cc9820c9c7e382838e9ca5ff721e79586b35
     54: cumulative: 1659606 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x1508c9046f42c7f22e3c4afd6d4cdf11a172098ebf941dabdde138397b3aa535 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 5de40df573df81583e5b63fb6d9b17e38321b5cd361163eb534f5a968f0d08b1
     55: cumulative: 1680606 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x1b95892c8eab91c55f74fa1f39a4f89b7db9e79f93bfc1cefec3b14757b4582a logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: e418f61106bd89812b3f6ba5cfe8c572a215e265f43774f287546afb739e3ed5
     56: cumulative: 1701606 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xa58e9b2bf62105b206c3d0ba50a37a515af33083eedd91b142e50dc59276157a logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 5e7e43bf736ac31d33a76f647d8fa2233160fdfeae94cef3224aee073a55429e

Error: invalid receipt root hash (remote: 10e976294b1eda359c0a0fca9c6bccd9eef4c8649a6a1b924356632ec5af5aa3 local: ec49e2ad4e89122c7c97c4c88913400272086a77ab8d987d3fe3d041a6477819)
##############################

WARN [01-12|05:11:11.616] Synchronisation failed, dropping peer    peer=7ee49e2a929b84d5 err="retrieved hash chain is invalid"

Wow, that is strange indeed! I've done four-five full syncs the last couple of weeks, never hit any problems like these.. makes me wonder... Gripping at straws here, could you do a bios memtest?

20200112_151920

MemTest86 reports no memory errors

Theoretically, could a single 'static-peer' provide bad block(s) resulting in the behavior? I have to assume that even 1 connected peer providing good block(s) would be enough, but I'm curious if that scenario would be enough to cause this behavior? Would receiving invalid or corrupt block(s) cause this behavior? I would assume it would NOT given that the node is processing each block, but I'm also grasping at straws 😄

Hm, I'm a bit confused, I thought this ticket and https://github.com/ethereum/go-ethereum/issues/20485 were the same, but they're not. So, is everything in this ticket about import of existing rlp dump?

Theoretically, could a single 'static-peer' provide bad block(s) resulting in the behavior?

No, block 2952165 does indeed have hash 0xe3821eeacc776d2f5c5ce6509c2dfd6f2f1711973f758365d5b044d1a5be6c7e, so it's a canon block all right. If a peer screwed with it, it would choke earlier, because the block body would not match up to the hash specified in the header, and if the header was modified the header hash would be different (and the PoW would fail)

The receipts you posted seem to match http://mon04.ethdevops.io/block/2952165 , at least concerning the accumulated gas, though I suspect that there's some mismatch in one of the receipts, will need to investigate it further.

All of this (both tickets) seem to come from a data corruption, and things like this typically happens when there's a disk corruption which causes leveldb to be unable to provide e.g. correct state (which makes geth miss a state entry, which could make it think a nonce is 0, but it can also lead to different hash root -- basically a whole plethora of possible faults). So it's really odd that you're seeing this on a brand new SSD. I really can't figure out what could possibly cause this.

Did you compile the code yourself? And if so, what go compiler version did you use?

Sorry for the confusion. I didn't want to open a new error every time.

Basically I only tried to import the DB twice, then you suggested that the export was corrupt. So I switched to just syncing from the network.

I tried that a few times but it failed so I switched to the PPA release on Ubuntu (stable). That also failed multiple times.

So I switched to the binary release for Windows on a different workstation, that failed too.

I bought a new SSD, tried again on Ubuntu stable from PPA after fresh OS install.

The only thing that is equal between all runs (2 different workstations, 2 different OS) is that they're all on the same local network. Which is where my last question came from, I thought it might be possible that I have a neighbor peer node with corrupt data poisoning the sync process somehow.

To fuel my theory, I had 2 Geth nodes:

Geth01 - The one I was trying to do a full sync with many times
Geth02 - A previously Fast Sync node that I never messed with, but always pointed to via 'static-peers.json'

Yesterday I thought screw it I'll try to do a Full Sync on Geth02 also. So I wiped BOTH nodes and started them at the same time. Geth01 is now on block 4,731,858 and still going. Granted, it could still fail, it's only 1/2 way to the tip. But so far it's going better than previous runs.

I thought it might be possible that I have a neighbor peer node with corrupt data poisoning the sync process somehow.

Nope (unless there's some deep flaw being exploited by some really really advanced attacker). A full sync fetches blocks from the network. It verifies the block content against the header, and executes every transaction. There's no way a peer can influence that, really.

A fast sync is a different matter, where in theory at least there are more avenues for a peer to do something bad. However, all state downloads are verified against the trie, so in practice I don't think a peer can do stuff which actually causes corrupt data on the victim.

On your windows-machine, this message indicates that you got a data corruption on leveldb:

ERROR[01-04|23:38:46.473] Failed to commit recent state trie       err="leveldb/table: corruption on data-block (pos=683158): checksum mismatch, want=0xa7c23af got=0x99f2513d [file=088662.ldb]"

specifically: leveldb/table: corruption on data-block (pos=683158): checksum mismatch, want=0xa7c23af got=0x99f2513d [file=088662.ldb].

So is Get01 the windows-machine that you replaced SSD on?

No, Geth01 is the Ubuntu machine that all of these errors came from except for the single run that I did on windows. Geth01 is the system that I replaced the SSD.

FWIW the windows machine is also brand new...

Well Geth01 failed again. I got all the way to 5,013,983 this time.

INFO [01-15|05:26:04.798] Imported new chain segment               blocks=13    txs=2303  mgas=94.746   elapsed=2.759s    mgasps=34.340  number=5013911 hash=d4d1c8…bf9128 age=1y11mo3w  dirty=1.29GiB
INFO [01-15|05:26:07.564] message loop                             peer=40f97c9725dff51e err=EOF
INFO [01-15|05:26:14.506] Imported new chain segment               blocks=6     txs=794   mgas=29.704   elapsed=9.665s    mgasps=3.073   number=5013917 hash=166223…9ef573 age=1y11mo3w  dirty=1.29GiB
INFO [01-15|05:26:19.048] Storing bloom trie                       section=152 head=2d64a929ec585ec947a2f05dcaf789ad75e8b8987354abaf47fd482de512274e root=9d5d778dd8601b95bc1b82067a63de0922286ea3b149599a7819ba32ab8f37f0 compression=0.743
INFO [01-15|05:26:21.015] Storing CHT                              section=152 head=2d64a929ec585ec947a2f05dcaf789ad75e8b8987354abaf47fd482de512274e root=7c0b8c9a34a09e7997ac0f807c00027e8d814ce945eeb63d7cb656d651abb259
INFO [01-15|05:26:23.113] Imported new chain segment               blocks=20    txs=2421  mgas=134.005  elapsed=8.607s    mgasps=15.569  number=5013937 hash=11090a…dc9b2c age=1y11mo3w  dirty=1.29GiB
INFO [01-15|05:26:31.343] Imported new chain segment               blocks=24    txs=4524  mgas=174.264  elapsed=8.230s    mgasps=21.173  number=5013961 hash=80e081…b63406 age=1y11mo3w  dirty=1.29GiB
WARN [01-15|05:26:37.707] Synchronisation failed, dropping peer    peer=9cdcd6a6f9dceae1 err=timeout
INFO [01-15|05:26:42.663] message loop                             peer=40f97c9725dff51e err=EOF
INFO [01-15|05:26:43.289] message loop                             peer=73316a0d92f6e2c4 err=EOF
WARN [01-15|05:26:43.740] Header broke chain ancestry              peer=a4b5b35d87a74de7 number=5026482 hash=199e58…10f6ea
ERROR[01-15|05:26:43.937]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 5013983
Hash: 0x2bb121b6832005d7519fcd5c1731567fc8404a019b9057ef2c420123c8d3b1f8

Error: invalid mix digest
##############################

WARN [01-15|05:26:44.016] Synchronisation failed, dropping peer    peer=08f823268b21a54f err="retrieved hash chain is invalid"
INFO [01-15|05:26:45.854] Deep froze chain segment                 blocks=187   elapsed=198.533ms number=4923981 hash=321ca4…a80b3f
WARN [01-15|05:26:50.826] Checkpoint challenge timed out, dropping id=6121da935ccfec4c conn=inbound addr=100.24.116.235:39380  type=Geth/v1.9.9-stable/linux-amd64/go1.13.4
WARN [01-15|05:26:51.665] Header broke chain ancestry              peer=a4b5b35d87a74de7 number=5020775 hash=fb7dd0…04726b
INFO [01-15|05:26:52.605] message loop                             peer=cff8e1c3029cdcfb err=EOF

Someone else on Twitter says they experienced similar issues:

https://twitter.com/MrBobGilbert/status/1217513656323321858?s=20

Unfortunately they didn't open any issue about it. Replacing the CPU/Mobo seems extreme. I mean, it's not like my systems exhibit any other issues / crashes / blue screens / etc. My systems have been stable otherwise.

I have to assume that all of the Geth benchmark nodes run on virtual machines and/or dockers, which could be abstracting some underlying driver/hardware issue or bug. All of my machines are bare metal...

Hm, Error: invalid mix digest -- this is very interesting. The previous errors you reported indicated that there was some corruption in the state. Now, such corruptions can have quite a lot of sources,

when state is copied,
the in-memory state pruner,
leveldb corruption

The invalid mix digest, however, is different. since it's not dependent upon state, but is fully contained within the ethash seal verification. I'm still not sure what to make of this, but that was very interesting.

If you restart this one (shut it down cleanly and restart it with same command line) does it then hit the same error? If so, we can do a custom build with some more detailed printout to find out what the expected versus actual values are.

I suspect that this error is ephemeral, but if we're really really lucky and it's reproducable, this pr: https://github.com/ethereum/go-ethereum/pull/20567 would provide more details about the internals.

Unfortunately I did not expect you to come back with a potential debug and I already wiped it in an attempt to run it inside a docker to see if it was still reproducible. Terribly sorry. I'll try to reproduce. I have 3 runs going right now. I'll keep at it.

So, I ran Geth in a docker this time...and I still got the error...

Number: 5901771
Hash: 0xa6718ce1ec8490a9155177c1a3a70360237040b2f578426b219ae7ba97ddd9ba
Error: invalid mix digest

I mounted drives to the docker so that it would be easy to restart geth outside of docker with the same chaindata. After Geth crashed in the docker with that, I stopped it. I then re-ran geth outside of the docker and reproduced.

INFO [01-20|13:20:09.612] Stats daemon started
INFO [01-20|13:20:09.615] IPC endpoint opened                      url=/data/0/ethereum/geth.ipc
INFO [01-20|13:20:09.616] HTTP endpoint opened                     url=http://0.0.0.0:8545       cors=* vhosts=*
INFO [01-20|13:20:59.611] Block synchronisation started
ERROR[01-20|13:21:01.329]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 5901771
Hash: 0xa6718ce1ec8490a9155177c1a3a70360237040b2f578426b219ae7ba97ddd9ba

Error: invalid mix digest
##############################

WARN [01-20|13:21:01.334] Synchronisation failed, dropping peer    peer=b5e858d3910cf838 err="retrieved hash chain is invalid"

If you can give me some basic instructions for how to test your patch I can get you the output.

Still on 1.9.9

Interesting! So you'll need to build from source, and specifically check out that patch. I've never done this on windows, but the linux steps are basically

go get -d github.com/ethereum/go-ethereum
cd ~/go/src/github.com/ethereum/go-ethereum
git remote add holiman https://github.com/ethereum/go-ethereum
git fetch holiman
git checkout  verbose_verification_err
make

That would make the geth land in the ~/go/src/github.com/ethereum/go-ethereum/build/bin/ folder.

No worries, it's Ubuntu

I didn't have Go since I had reinstalled Ubuntu, so I grabbed that: https://github.com/ethereum/go-ethereum/wiki/Installation-Instructions-for-Ubuntu

sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt-get update
sudo apt-get install golang-go

But it says it can't find your branch?

chase@geth01:~/go/src/github.com/ethereum/go-ethereum$ git remote add holiman https://github.com/ethereum/go-ethereum
chase@geth01:~/go/src/github.com/ethereum/go-ethereum$ git fetch holiman
From https://github.com/ethereum/go-ethereum
 * [new branch]          gh-pages       -> holiman/gh-pages
 * [new branch]          master         -> holiman/master
 * [new branch]          poc8           -> holiman/poc8
 * [new branch]          release/0.9.36 -> holiman/release/0.9.36
 * [new branch]          release/1.0.0  -> holiman/release/1.0.0
 * [new branch]          release/1.0.1  -> holiman/release/1.0.1
 * [new branch]          release/1.1.0  -> holiman/release/1.1.0
 * [new branch]          release/1.2.1  -> holiman/release/1.2.1
 * [new branch]          release/1.3.0  -> holiman/release/1.3.0
 * [new branch]          release/1.3.2  -> holiman/release/1.3.2
 * [new branch]          release/1.3.3  -> holiman/release/1.3.3
 * [new branch]          release/1.3.4  -> holiman/release/1.3.4
 * [new branch]          release/1.3.5  -> holiman/release/1.3.5
 * [new branch]          release/1.3.6  -> holiman/release/1.3.6
 * [new branch]          release/1.4    -> holiman/release/1.4
 * [new branch]          release/1.5    -> holiman/release/1.5
 * [new branch]          release/1.6    -> holiman/release/1.6
 * [new branch]          release/1.7    -> holiman/release/1.7
 * [new branch]          release/1.8    -> holiman/release/1.8
 * [new branch]          release/1.9    -> holiman/release/1.9
chase@geth01:~/go/src/github.com/ethereum/go-ethereum$ git checkout  verbose_verification_err
error: pathspec 'verbose_verification_err' did not match any file(s) known to git.

Nevermind, I figured it out. Should be: git remote add holiman https://github.com/holiman/go-ethereum

Boom:

INFO [01-20|14:19:41.141] Block synchronisation started
ERROR[01-20|14:19:41.708] Invalid mix digest                       number=5901768 datasetSize=2717907328 hdr.nonce=8774555798626709531 sealHash="[69 201 230 234 206 75 24 9 247 242 180 231 191 184 43 24 242 195 86 93 114 177 250 81 31 11 16 38 7 157 51 241]" result="[103 158 62 246 38 246 6 82 71 33 210 238 177 37 178 13 191 241 61 186 140 226 136 148 83 111 108 166 252 245 249 55]" digest="[243 53 19 163 179 107 241 143 214 217 69 240 230 214 186 30 147 91 37 133 63 119 50 135 143 2 34 91 56 44 57 133]" hdr.digest="[7 230 161 170 212 130 52 40 152 14 130 41 51 80 133 198 228 85 153 243 214 89 81 206 124 121 106 216 112 129 235 114]"
ERROR[01-20|14:19:41.708]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 5901771
Hash: 0xa6718ce1ec8490a9155177c1a3a70360237040b2f578426b219ae7ba97ddd9ba

Error: invalid mix digest
##############################

WARN [01-20|14:19:41.708] Synchronisation failed, dropping peer    peer=b5e858d3910cf838 err="retrieved hash chain is invalid"

It's great that it's reproducable!

Now, please do a git pull (from my repo), followed by go test ./consensus/ethash/ -run TestEthashVerification -v (no need to rebuild). On my machine, I get the following output:

=== RUN   TestEthashVerification
datasetSize: 2717907328
nonce: 8774555798626709531
sealHash: 45c9e6eace4b1809f7f2b4e7bfb82b18f2c3565d72b1fa511f0b1026079d33f1
result: 00000000000010513b8b8750cc444765323bd0cc00c6562fe5faf0b92106c317

digest: 07e6a1aad4823428980e8229335085c6e45599f3d65951ce7c796ad87081eb72
xor sum of cache contents: 7a0b747e

And the resulting digest that I get on my machine matches what was in the header that your machine rejected. I suspect that you will get a different result, and it will be interesting to see if it's because of a corrupt verification cache or something else.

Btw, cc @karalabe .. ^ is using https://launchpad.net/~longsleep/+archive/ubuntu/golang-backports . Are there any known issues with that version of golang?

@MysticRyuujin the installation instructions on the wiki are not the most recent, here is the more updated: https://geth.ethereum.org/docs/install-and-build/installing-geth#most-linux-systems-and-macos . Tldr; install from golang official.

However, it's still interesting to follow this bug a bit and see where it leads. What does go version tell you?

Here's the results:

chase@geth01:~$ go version
go version go1.13.4 linux/amd64

chase@geth01:~/go/src/github.com/ethereum/go-ethereum$ git pull
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 11 (delta 5), reused 11 (delta 5), pack-reused 0
Unpacking objects: 100% (11/11), done.
From https://github.com/holiman/go-ethereum
   bf15e31b7..b1756c431  verbose_verification_err -> holiman/verbose_verification_err
Updating bf15e31b7..b1756c431
Fast-forward
 consensus/ethash/consensus.go   |  8 ++++----
 consensus/ethash/ethash_test.go | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 4 deletions(-)
chase@geth01:~/go/src/github.com/ethereum/go-ethereum$ go test ./consensus/ethash/ -run TestEthashVerification -v
=== RUN   TestEthashVerification
datasetSize: 2717907328
nonce: 8774555798626709531
sealHash: 45c9e6eace4b1809f7f2b4e7bfb82b18f2c3565d72b1fa511f0b1026079d33f1
result: 00000000000010513b8b8750cc444765323bd0cc00c6562fe5faf0b92106c317

digest: 07e6a1aad4823428980e8229335085c6e45599f3d65951ce7c796ad87081eb72
xor sum of cache contents: 7a0b747e
--- PASS: TestEthashVerification (1.51s)
PASS
ok      github.com/ethereum/go-ethereum/consensus/ethash        1.533s

Do you want me to go back and reinstall Go using the other instructions and try that go test command again?

This longsleep repo is not an official PPA btw. The Go project does not provide a PPA and someone just dumped their own packages on the Go wiki. We are uploading and compiling the entire Go when building our own PPA packages specifically not to have to rely on this weird distribution.

But Go does provide pre-built binaries for linux, you can get it from them https://golang.org/dl/.

@MysticRyuujin This error is very very strange because apparently you are the only one getting them :D. Curious what's so unique about your system. Do you have any specialized hardware? Storage/memory? Some strange permissions or file systems?

This node was a Fast Sync node for years, off and on, but it ran stable for months and months at a time, and whenever new versions of Geth came out I'd usually just wipe the chaindata and start a fresh Fast Sync. This is the "first time" I've tried doing Full Sync. Not that any of that should matter...

There's nothing special about my hardware that I can tell...

https://www.intel.com/content/www/us/en/products/boards-kits/nuc/kits/nuc5i7ryh.html

The memory is Kingston and passing MemTest86 as seen above. The storage is just SATA SSDs, I've tried multiple consumer SSDs.

The steps I took to produce this most recent error after buying new SSDs:

Download Ubuntu 18.04.03 LTS, stick it on a USB stick. Install it through the wizard, configure static IP, perform apt update && apt -y dist-upgrade, mount storage, install geth from official PPA. Run it via Supervisor. Then I tried running via the official Docker, same thing. Here we are.

I don't know how to answer your question about strange permissions or file systems, as I didn't do anything with them after installing Ubuntu beyond formatting and mounting the 2.5" SSD that is serving as the freezer. /data/1/ethereum and I usually run everything as root 🤷‍♂ (right now)

I generally install Geth from the PPA, so I generally don't install Go anyway. The docker doesn't require that I install Go either... I only installed Go to perform the requested tests. I'll install literally any version of Go you guys want me to install if it helps...

cat /etc/fstab
UUID=2bbf21cb-bbb8-4c4a-a1d6-9f30caeeed3d / ext4 defaults 0 0
UUID=D916-0485 /boot/efi vfat defaults 0 0
/swap.img       none    swap    sw      0       0
UUID=613350dd-cdb5-4daa-9f17-bf875fd8e0ce /data/1 ext4 defaults 0 0

The only thing on my entire network that could be of interest is my pfSense firewall running Snort. I don't believe Snort would actually modify any data in transit, but it could drop packets or block hosts.

$ /usr/local/go/bin/go version
go version go1.13.6 linux/amd64

$ /usr/local/go/bin/go test ./consensus/ethash/ -run TestEthashVerification -v
=== RUN   TestEthashVerification
datasetSize: 2717907328
nonce: 8774555798626709531
sealHash: 45c9e6eace4b1809f7f2b4e7bfb82b18f2c3565d72b1fa511f0b1026079d33f1
result: 00000000000010513b8b8750cc444765323bd0cc00c6562fe5faf0b92106c317

digest: 07e6a1aad4823428980e8229335085c6e45599f3d65951ce7c796ad87081eb72
xor sum of cache contents: 7a0b747e
--- PASS: TestEthashVerification (1.52s)
PASS
ok      github.com/ethereum/go-ethereum/consensus/ethash        1.535s

Same

Not sure if it helps but I remove go via apt remote, did a make with go1.13.6

INFO [01-20|22:36:11.998] Block synchronisation started
ERROR[01-20|22:36:12.759] Invalid mix digest                       number=5901768 datasetSize=2717907328 hdr.nonce=8774555798626709531 sealHash=45c9e6eace4b1809f7f2b4e7bfb82b18f2c3565d72b1fa511f0b1026079d33f1 result=679e3ef626f606524721d2eeb125b20dbff13dba8ce28894536f6ca6fcf5f937 digest=f33513a3b36bf18fd6d945f0e6d6ba1e935b25853f7732878f02225b382c3985 hdr.digest=07e6a1aad4823428980e8229335085c6e45599f3d65951ce7c796ad87081eb72
ERROR[01-20|22:36:12.759]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 5901771
Hash: 0xa6718ce1ec8490a9155177c1a3a70360237040b2f578426b219ae7ba97ddd9ba

Error: invalid mix digest
##############################

WARN [01-20|22:36:12.759] Synchronisation failed, dropping peer    peer=b5e858d3910cf838 err="retrieved hash chain is invalid"

Appears to be the same.

Note for ourselves: The error is a bit misleading. The header with invalid mix digest is in fact the uncle at height 5901768 , but the BAD BLOCK message makes it look like header 5901771 has invalid digest. Both have the same epoch (196), though, so I don't think the cause is related to epoch-mixups.

@MysticRyuujin Interesting! So it appears that when run in isolation (the testcase), it behaves correctly, but it still fails when run within geth. I've now pushed a change which does the same xor-check of the ethash cache contents. My hunch is that the generated dag-data is corrupt, which would have happened when the cache was generated (and in that case, it doesn't matter if you switched compiler now, since it was generated a while ago).

If I am correct, then we can also adapt the testcase to use the same ethash directories as your geth installation does.

Anyway, please git pull, make, and restart geth, which will give us some more info. And huge "Thank You!" for taking the time to help us out with this!

Here you go.

INFO [01-21|08:40:42.849] Block synchronisation started
WARN [01-21|08:41:42.849] Synchronisation failed, dropping peer    peer=40f97c9725dff51e err=timeout
WARN [01-21|08:42:41.586] Checkpoint challenge timed out, dropping id=0f7926e3f0907c93 conn=dyndial addr=138.68.98.205:30301 type=Geth/v1.9.9-stable/linux-amd64/go1.13.4
WARN [01-21|08:43:02.849] Synchronisation failed, dropping peer    peer=d9f9252151cde7bc err=timeout
WARN [01-21|08:43:12.870] Synchronisation failed, dropping peer    peer=2615086a9dedb6f3 err="action from bad peer ignored"
ERROR[01-21|08:43:33.419] Invalid mix digest                       number=5901768 datasetSize=2717907328 hdr.nonce=8774555798626709531 sealHash=45c9e6eace4b1809f7f2b4e7bfb82b18f2c3565d72b1fa511f0b1026079d33f1 result=679e3ef626f606524721d2eeb125b20dbff13dba8ce28894536f6ca6fcf5f937 epoch=196 cachedir=/data/0/ethereum/geth/ethash datasetdir=/root/.ethash cachexor=7a0b743e digest=f33513a3b36bf18fd6d945f0e6d6ba1e935b25853f7732878f02225b382c3985 hdr.digest=07e6a1aad4823428980e8229335085c6e45599f3d65951ce7c796ad87081eb72
ERROR[01-21|08:43:33.419]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}

Number: 5901771
Hash: 0xa6718ce1ec8490a9155177c1a3a70360237040b2f578426b219ae7ba97ddd9ba

Error: invalid mix digest
##############################

WARN [01-21|08:43:33.419] Synchronisation failed, dropping peer    peer=f56453fd00f1234f err="retrieved hash chain is invalid"

I noticed you're dumping directory paths, in case you want the full Geth command I'm running:

/home/chase/go/src/github.com/ethereum/go-ethereum/build/bin/geth --datadir /data/0/ethereum --datadir.ancient /data/1/ethereum --nousb --cache 8192 --ethstats=MysticRyuujin-Geth01:<redacted>@ethstats.net --identity MysticRyuujin-Geth01 --light.serve 100 --port 30301 --maxpeers 200 --nat extip:<redacted> --rpc --rpcapi db,eth,net,web3,admin --rpcaddr "0.0.0.0" --rpccorsdomain "*" --rpcvhosts "*" --syncmode full

I notice that it says datasetdir=/root/.ethash but that's not a real directory...

root@geth01:~# ls /root/.ethash
ls: cannot access '/root/.ethash': No such file or directory
root@geth01:~# ls /data/0/ethereum/geth/ethash/
cache-R23-0ba3f65a466961c3  cache-R23-52341cf7371faf9a  cache-R23-9d3fef9bc522aee6

Wow, that's really curious! So what we can see, is:

The ethash.cache is correct: 7a0b743e (same as testcase). Not what I had expected!

This means that the call that produces an error is this one:digest, result = hashimotoLight(size, cache.cache, sealHash, nonce)

We have verified that size, cache.cache, sealHash and nonce is identical in the testcase versus the live geth node, but for some reason the live node fails whereas the testcase works.

I pushed another commit on my branch. The commit produces a detailed printout of the internals in hashimotoLight. Would be great if you could give it a spin (on a rebuilt geth) and paste the output here (should be quite verbose)

Perfect. This is the 'correct' execution:

...
lookup(22079407)=[c0e2ff5 440bc35 4de06699 f05b4789 3a9955c8 2c1a496b 51ebd86e 4fc4b92a 1cbcd613 bf1b551 ad9d5083 73c2761a 4dbb0b98 35fddc04 e970c8be e0ec090a]
lookup(11780086)=[f9d22e11 c86fd76c 86def046 a8bd9550 6c5f6470 f155ba50 2ce0faaa 193176d4 cf6a7cc4 ef97d37d bdfa6936 70f23a98 dcf1ea1f e9e42d1a 279912fe 8d801210]

lookup(11780087)=[c30ef0e5 3e8dfbd3 252d71f4 128949c4 74e0101d fd935dad 69c8d4dc ff68a148 49df6969 c363f2e7 dc6bfd04 33e2ae48 d5e4607e 4e16a37 a8a2c214 6d5398a1]
lookup(3886498)=[1e576d9a 3ed23c02 d9cce025 489684ca 2d37117a 9f35924b 9a3866a1 c2c5e4b8 27cf293 536ef5f 2362a0c0 75143f63 47219f33 30a83a0b a38002c1 9c9d22f7]
lookup(3886499)=[a6927e4 45773d33 27406892 2de5d270 277551a2 72d46969 864c7ab6 755b6f8 e2f668d cb7b9284 6741a65d e8c98267 70700153 aaed9364 63e64c25 3b6b49c]
...

This is yours, which has wrong values on lookup index 11780086:

lookup(22079407)=[c0e2ff5 440bc35 4de06699 f05b4789 3a9955c8 2c1a496b 51ebd86e 4fc4b92a 1cbcd613 bf1b551 ad9d5083 73c2761a 4dbb0b98 35fddc04 e970c8be e0ec090a]
lookup(11780086)=[1a6ce817 f431d445 54e9b63 c6a5af7d 322d1733 f7dd8009 bcd4596c 73bcf142 b4128b6f b8a6fd4b d2d59b31 4a988cde 205e24d7 ea86d13d 91f4d37d 2ed45567]

lookup(11780087)=[c30ef0e5 3e8dfbd3 252d71f4 128949c4 74e0101d fd935dad 69c8d4dc ff68a148 49df6969 c363f2e7 dc6bfd04 33e2ae48 d5e4607e 4e16a37 a8a2c214 6d5398a1]
lookup(7504002)=[5614ab51 35446e6d 8e784864 5cf7232a 662f83de 79caebb5 381f50ed 427f9d5f 822c658a ad4ec807 1b78b530 31e2c11a 156ce5e3 dceff46f 13433e65 4fee18c7]
lookup(7504003)=[3e1d854 b4cb15a5 8bcd5283 fdd4c0ba 69ab386 fdaaf353 ff9a8857 3ddf98a8 cea5b25d a4e4bdb1 d1ef02bc f0561197 cee4d906 93154e9f 787b98e2 b2ef1c7b]

Those invalid lookup values later causes the wrong indices to be used.

Do I recall correctly that you offered ssh access -- and if so, does that still stand? I think that would be easier for both of us :)

Yeah, I'm at work right now, but when I get home I'll set it up and send you credentials via Twitter DM or discord or whatever.

There appears to be two ethash directories:

root@geth01:~/ethereum/geth# ls -l /root/ethereum/geth/ethash/
total 113276
-rw-r--r-- 1 root root 38535112 Jan 15 01:58 cache-R23-3aa8f28cac16bdd8
-rw-r--r-- 1 root root 38795592 Jan 15 05:10 cache-R23-a8784097a4d03c2d
-rw-r--r-- 1 root root 38660680 Jan 15 03:34 cache-R23-b683d2a971567602

These caches correspond to

seedHash 4980000 (epoch 166): 3aa8f28cac16bdd8
seedHash 5010000 (epoch 167): b683d2a971567602
seedHash 5040000 (epoch 168): a8784097a4d03c2d

and

root@geth01:~/ethereum/geth# ls -l /data/0/ethereum/geth/ethash/
total 124416
-rw-r--r-- 1 root root 42597832 Jan 20 12:12 cache-R23-0ba3f65a466961c3
-rw-r--r-- 1 root root 42334920 Jan 20 08:57 cache-R23-52341cf7371faf9a
-rw-r--r-- 1 root root 42467016 Jan 20 10:29 cache-R23-9d3fef9bc522aee6

These correspond to

seedHash 5910000 (epoch 197): 0ba3f65a466961c3
seedHash 5850000 (epoch 195): 52341cf7371faf9a
seedHash 5880000 (epoch 196): 9d3fef9bc522aee6

So these latter ones are the ones that are interesting for us. These are the shasums for them:

# shasum /data/0/ethereum/geth/ethash/cache-R23-0ba3f65a466961c3 
d5efa7bfe895e961c2e76b404f0e7ffc600e0b96  /data/0/ethereum/geth/ethash/cache-R23-0ba3f65a466961c3
# shasum /data/0/ethereum/geth/ethash/cache-R23-52341cf7371faf9a 
caafb6737cefb7a3c7b82a613b2de75e4fb81ac0  /data/0/ethereum/geth/ethash/cache-R23-52341cf7371faf9a
# shasum /data/0/ethereum/geth/ethash/cache-R23-9d3fef9bc522aee6 
260a721fce0c9c2e17ae40c320a2e08e57d36d89  /data/0/ethereum/geth/ethash/cache-R23-9d3fef9bc522aee6

On my local machine, verifying the correctness:

build/bin/geth makecache 5910000 /tmp/ && ls -l /tmp/cache-R23-0ba3f65a466961c3 && shasum /tmp/cache-R23-0ba3f65a466961c3
-rw-rw-r-- 1 user user 42597832 Jan 22 09:24 /tmp/cache-R23-0ba3f65a466961c3
d5efa7bfe895e961c2e76b404f0e7ffc600e0b96  /tmp/cache-R23-0ba3f65a466961c3

Correct

[user@work go-ethereum]$ build/bin/geth makecache 5850000 /tmp/ && ls -l /tmp/cache-R23-52341cf7371faf9a && shasum /tmp/cache-R23-52341cf7371faf9a
-rw-rw-r-- 1 user user 42334920 Jan 22 09:27 /tmp/cache-R23-52341cf7371faf9a
caafb6737cefb7a3c7b82a613b2de75e4fb81ac0  /tmp/cache-R23-52341cf7371faf9a

Correct.

[user@work go-ethereum]$ build/bin/geth makecache 5880000 /tmp/ && ls -l /tmp/cache-R23-9d3fef9bc522aee6 && shasum /tmp/cache-R23-9d3fef9bc522aee6
-rw-rw-r-- 1 user user 42467016 Jan 22 09:26 /tmp/cache-R23-9d3fef9bc522aee6
371e76fa34e72ba47f837b5e7fa993e6ed054439  /tmp/cache-R23-9d3fef9bc522aee6

Mismatch. (371e76fa34e72ba47f837b5e7fa993e6ed054439 == correct, whereas 260a721fce0c9c2e17ae40c320a2e08e57d36d89 is incorrect)

So the ethash cache for epoch 196 is corrupt. Let's check generating the same thing on the geth01 server, with the stock geth

root@geth01:~# geth makecache 5880000 /tmp/ && shasum /tmp/cache-R23-9d3fef9bc522aee6
371e76fa34e72ba47f837b5e7fa993e6ed054439  /tmp/cache-R23-9d3fef9bc522aee6

This one generates the correct cache (1.9.9-stable-01744997).

Trying instead with the docker version:

root@geth01:~# docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
ethereum/client-go   stable              4c5ed2a7036c        6 weeks ago         41.3MB

rm /tmp/cache-R23-9d3fef9bc522aee6 && docker run -v /tmp:/foo ethereum/client-go makecache 5880000 /foo && shasum /tmp/cache-R23-9d3fef9bc522aee6
371e76fa34e72ba47f837b5e7fa993e6ed054439  /tmp/cache-R23-9d3fef9bc522aee6

This one is also correct. So both the stock geth and the 6 week old docker geth produces a correct ethash cache. Let's peek closer at the corrupted version.

The two ethash files are seemingly pretty similar, there's only one byte that differs:

root@geth01:~# cmp -l /tmp/cache-R23-9d3fef9bc522aee6 /data/0/ethereum/geth/ethash/cache-R23-9d3fef9bc522aee6 | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'
01AFFEB5 C9 89

root@geth01:~# xxd /tmp/cache-R23-9d3fef9bc522aee6 > /tmp/correct.hex
root@geth01:~# xxd /data/0/ethereum/geth/ethash/cache-R23-9d3fef9bc522aee6 > /tmp/incorrect.hex
root@geth01:~# diff /tmp/correct.hex /tmp/incorrect.hex
1769452c1769452
< 01affeb0: 60c7 0a4c c975 082c 6370 378f 1e51 19e5  `..L.u.,cp7..Q..
---
> 01affeb0: 60c7 0a4c 8975 082c 6370 378f 1e51 19e5  `..L.u.,cp7..Q..

Indeed, a 0xC9 got turned into a 0x89, a bitflip from 0b11001001 into 0b10001001. Diving in a bit deeper - the memory does not seem to be ECC (error correcting) (info):

root@geth01:~# dmidecode -t memory
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x0026, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: None
    Maximum Capacity: 16 GB
    Error Information Handle: Not Provided
    Number Of Devices: 2

Here's what I get from one of our aws machines:

# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x1000, DMI type 16, 19 bytes
Physical Memory Array
    Location: Other
    Use: System Memory
    Error Correction Type: Multi-bit ECC

So sources of corruption can either be the disk or the non-error-correcting RAM. From the info site above, it says

ECC (error-correcting code) RAM is essential in servers and many workstations as it dramatically improves the reliability of the system's memory by detecting common kinds of data corruption and fixing the errors automatically. Without the error correction of ECC memory, memory errors can cause data corruption or even blue screens.

I really have no idea how common that is, but at this point that is my primary suspect.

To follow-up, deleting the file /data/0/ethereum/geth/ethash/cache-R23-9d3fef9bc522aee6 should solve this problem and let the sync continue.

I also installed smartmontools, and ran a quick selftest.

root@geth01:~# smartctl -a /dev/sdb2
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-74-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
... 
Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 ec 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  c8 00 00 00 00 00 00 00      00:00:00.000  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       244         -

The short self-test completed without errors, but it does report some type of error. Perhaps would be worth it to run a longer self-test with smartctl -t long -a /dev/sdb2

Could you dump out the SMART stats of your SSD? Just wondering what the counters look like.

root@geth01:/home/holiman# smartctl -a /dev/sdb2
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-74-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     CT500MX500SSD4
Serial Number:    1951E2305A62
LU WWN Device Id: 5 00a075 1e2305a62
Firmware Version: M3CR023
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      M.2
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA >3.2 (0x1ff), 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jan 23 10:06:53 2020 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (  30) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x0031) SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   010    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       268
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       2
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
173 Unknown_Attribute       0x0032   099   099   000    Old_age   Always       -       22
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   000   000   000    Pre-fail  Always       -       45
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   040   029   000    Old_age   Always       -       60 (Min/Max 0/71)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
202 Unknown_SSD_Attribute   0x0030   099   099   001    Old_age   Offline      -       1
206 Unknown_SSD_Attribute   0x000e   100   100   000    Old_age   Always       -       0
210 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
246 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       5950116439
247 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       109711093
248 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       125703115

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1

ATA Error Count: 0
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 ec 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  c8 00 00 00 00 00 00 00      00:00:00.000  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       244         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

and for /dev/sda

root@geth01:/home/holiman# smartctl -a /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-74-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     SATA SSD
Serial Number:    19110148000277
LU WWN Device Id: 0 000000 000000000
Firmware Version: SBFMJ1.3
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   Unknown(0x0ff8) (minor revision not indicated)
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jan 23 10:07:59 2020 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (65535) seconds.
Offline data collection
capabilities:            (0x79) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (  30) minutes.
Conveyance self-test routine
recommended polling time:    (   6) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       277
 12 Power_Cycle_Count       0x0012   100   100   000    Old_age   Always       -       8
168 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       0
170 Unknown_Attribute       0x0003   078   078   010    Pre-fail  Always       -       201
173 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       1
192 Power-Off_Retract_Count 0x0012   100   100   000    Old_age   Always       -       5
194 Temperature_Celsius     0x0023   067   067   000    Pre-fail  Always       -       33 (Min/Max 33/33)
218 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
231 Temperature_Celsius     0x0013   100   100   000    Pre-fail  Always       -       100
241 Total_LBAs_Written      0x0012   100   100   000    Old_age   Always       -       91

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

ethereum / go-ethereum

BAD BLOCK errors #20478

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Backtrace