pistomat
commented
4 months ago Here are my logs reth.log EDIT: full log.1 (previously truncated) reth.log.1.zip
Closed pistomat closed 4 months ago
pistomat
commented
4 months ago Here are my logs reth.log EDIT: full log.1 (previously truncated) reth.log.1.zip
winksaville
commented
4 months ago @pistomat
The reth.log file starts with
2024-04-13T08:30:34.296405Z DEBUG storage::db::mdbx: Commit total_duration=24.489903ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 12, audit: 0, write: 0, sync: 1578, ending: 0, whole: 1591, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=falseThe reth.log.1.txt file ends with:
2024-04-10T18:20:13.744223Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=134
2024-04-10T18:20:13.886711Z DEBUG blockchain_tree::chain: Validated state root number=1320312 hash=0xc9610a7c6c9909648de9571e614bf87a0bee2e046e7b5fa63f2c1593f186e96d elapsed=144.087248ms
2024-04-10T18:20:13.886793Z DEBUGThe last line looks like it's truncated and the date is April 10, where as reth.log begins with a April 13 date. So it looks like reth.log.1.txt was truncated as I'd expect it to have a ending time stamp of 2024-04-13T08:30:33.xxx or 2024-04-13T08:30:34.yyyyyyZ, where yyyyyy is < 296405.
pistomat
commented
4 months ago Oh yeah, it got truncated on downloading from the node, it was way larger than I expected.
@winksaville nontruncated file: reth.log.1.zip
winksaville
commented
4 months ago Oh yeah, it got truncated on downloading from the node, it was way larger than I expected.
@winksaville nontruncated file: reth.log.1.zip
This reth.log.1 is similar to mine, there are many WARN Failed to insert block lines in the log that are identical except for the time stamp, using uniq it shows count, 262, and the first instance`:
wink@3900x 24-04-13T16:49:19.355Z:~/reth-lighthouse-issues/reth-issue-7619
$ rg 'Failed to insert block' reth.log.1 | uniq -cf 1
262 2024-04-13T06:56:51.378872Z WARN consensus::engine: Error while processing payload error=Failed to insert block (hash=0xe0276077b2294e2f5db876b8beff7c1c4d5c42c1710a95b743b6d708240d1329, number=1336932, parent_hash=0xb6031d3ce90a8dd0eddc0928020523803cf2b2bf4c427d239f5faf0e90c985b9): mismatched block state root: got 0xcc29631fbe7093c6c1901617130c44d96e3fd2226590cfa06923e1b6051e7784, expected 0xad7c30ac21e4a512b5e0a9ec0741865d84d6cb2e8ee300f2be1609932e5d2ca4
wink@3900x 24-04-13T16:49:30.780Z:~/reth-lighthouse-issues/reth-issue-7619My guess is that whatever lead up to this first Failed to insert block might be the root cause.
winksaville
commented
4 months ago Here is the uniq output from my holesky node:
kendall@ethel 24-04-13T17:54:52.608Z:~/eth2-data/reth/holesky
$ ls -lrt reth.log*
-rw-r--r-- 1 kendall kendall 209715241 Mar 25 08:57 reth.log.4
-rw-r--r-- 1 kendall kendall 209715331 Apr 3 10:07 reth.log.3
-rw-r--r-- 1 kendall kendall 209715655 Apr 8 10:26 reth.log.2
-rw-r--r-- 1 kendall kendall 209716221 Apr 12 12:59 reth.log.1
-rw-r--r-- 1 kendall kendall 36055006 Apr 13 10:55 reth.log
kendall@ethel 24-04-13T17:55:02.640Z:~/eth2-data/reth/holesky
$ rg 'Failed to insert block' reth.log.4 | uniq -cf 1
kendall@ethel 24-04-13T17:55:48.612Z:~/eth2-data/reth/holesky
$ rg 'Failed to insert block' reth.log.3 | uniq -cf 1
168 2024-03-26T17:10:51.389084Z WARN consensus::engine: Error while processing payload error=Failed to insert block (hash=0xd6bf93f17c15eef43e4fe34c675ce89d8c7d88ee2b5a341336ec587bfe066bbf, number=1221400, parent_hash=0xb211ed799d6b6fc2acfa9b71da8339127ec7fa397e3921868e0d906fcdbc4773): mismatched block state root: got 0xcdf1b990b63256e10ebd8e53f5e278047ae1b7f81c83f9b371a59e81d002ceb9, expected 0x871f69bf1cc2e24823e4a25f52e6e6af3134d97264bb16d617b4b69fc1114bf1
98 2024-04-02T05:16:50.721056Z WARN consensus::engine: Error while processing payload error=Failed to insert block (hash=0x168c92f7f36f0d968ee9f0353268af64c0e987e098f0b29d9d99483f1f6e4af2, number=1264353, parent_hash=0x4fe60b1e2aaccc7e25006671727879e2a2535fa087231efaa0424821bf9fa57e): mismatched block state root: got 0xa34bd0e7cebbf20708fbae4423721a00106e19e2fd075818f05d0582df314694, expected 0x4ed32b88c105a497378e0773cf741fcac17ab4474b074f515e0ad7f99dbb4af8
kendall@ethel 24-04-13T17:55:52.046Z:~/eth2-data/reth/holesky
$ rg 'Failed to insert block' reth.log.2 | uniq -cf 1
121 2024-04-04T00:45:02.611461Z WARN consensus::engine: Error while processing payload error=Failed to insert block (hash=0x20f3f909f84db85a78eb8b2b47716d188fe38471aa57f51c9e49959827f3e7ab, number=1276302, parent_hash=0x6e7d2560a8904fea597335628d2296185061316d821bd85ea818937316e07d36): mismatched block state root: got 0xd009436e581fa5d10f05a91189f8d4a244dc1c6f9eec91a6fcd8f8a4b1d51f97, expected 0xd03305890ff60b20ca96716ae8456f51918e37659a68e8bd2f2719527ecaac96
132 2024-04-08T10:14:16.306330Z WARN consensus::engine: Error while processing payload error=Failed to insert block (hash=0x75d33ca50ce5995c5829e318419a0457440a866982c717269d8ab59aa58d9378, number=1305364, parent_hash=0x5c44156fbe0b623395752c0881c6a52ebb7fd33f97307b6964da70fa0be94c65): mismatched block state root: got 0x1f4fa2a54d8d8d028b23247c85a769dbcc0c9b81fc61f5b1fc7a7ab07241d524, expected 0x617aa48146e8cec830faf41f4b8b36f1104e4c496c304ef5b37829bde06ac9df
kendall@ethel 24-04-13T17:55:55.046Z:~/eth2-data/reth/holesky
$ rg 'Failed to insert block' reth.log.1 | uniq -cf 1
kendall@ethel 24-04-13T17:55:59.022Z:~/eth2-data/reth/holesky
$ rg 'Failed to insert block' reth.log | uniq -cf 1
kendall@ethel 24-04-13T17:56:03.656Z:~/eth2-data/reth/holesky
gakonst
commented
4 months ago Fixed in #7753.
Describe the bug
Getting the state root mismatch error on Holesky full node after a short node downtime
I come from the state root mismatch warroom, checked out the latest main, built and run the
reth db stats --checksumSteps to reproduce
Run a holesky full node
Node logs
So it happened again, posting all the logs I get before crash
Platform(s)
Linux (x86)
What version/commit are you on?
What database version are you on?
What type of node are you running?
Full via --full flag
What prune config do you use, if any?
No response
If you've built Reth from source, provide the full command you used
make installCode of Conduct