search

NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.27k stars 437 forks source link

Energyweb archive sync stuck at block 18829722 #5776

Closed mclrch closed 1 year ago

mclrch commented 1 year ago

Description Energyweb archive sync stuck at block 18829722 when syncing from scratch.

Steps to Reproduce In order to replicate the behavior, please provide a detailed list of steps: Start a clean node and try to do an archive sync.

Actual behavior Please describe the outcome that occurred as a result of executing the above steps, and explain why it is considered an issue. Node is stuck.

Expected behavior Synced node.

Screenshots If applicable, please include screenshots to help illustrate the problem.

Desktop (please complete the following information): Please provide the following information regarding your setup:

Additional context Please provide any additional context about the problem.

Logs Please include any relevant logs that may help identify the issue.

Output log: nethermind_1 | 2023-06-05 12:22:05.7529|Processed block 18829722 (0x086a89...814712) is invalid: nethermind_1 | 2023-06-05 12:22:05.7529|- hash: expected 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b, got 0x086a898a1405546a3c0a06c5548b763166262d82f7abe04a5b1ad6acc0814712 nethermind_1 | 2023-06-05 12:22:05.7529|- gas used: expected 6000000, got 5818902 (diff: -181098) nethermind_1 | 2023-06-05 12:22:05.7529|- receipts root: expected 0x2c649a5d460f4d4a90df77002397f44178218d8f5641db1f53037695765b6113, got 0x08bb18651d8c18607189cddff21a05b8687faabef369eb7741452ceea1dd8f5a nethermind_1 | 2023-06-05 12:22:05.7529|- state root: expected 0xbee799d38eb3ad6e90a709accd42041bfed2731bee6fab4dca8754a2d5a4161c, got 0x964129cddaa6bd73a073db77d0ab483bcb687c4d1680b27020b25e8c939498ac nethermind_1 | 2023-06-05 12:22:05.7529|Processed block is not valid 18829722 (0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b) nethermind_1 | 2023-06-05 12:22:05.7529|Suggested block TD: 6407422370623267246122501039799329389803540696, Suggested block IsPostMerge False, Block TD: 6407422370623267246122501039799329389803540696, Block IsPostMerge False nethermind_1 | 2023-06-05 12:22:05.7598|Current contract validators (48): [0xd65b4c25a4ce1e024ff13425df1e0e574a1a0e9b, 0x83329c3fd90d7ee2efd546e0dc6453e9172a0643, 0xde15831ac319dab5eae5fd1fd9d52876c5e50f20, 0x86a5a44cff58638784c2028e7181cede57933321, 0xb5da81b0c1c808ef6d761dacfba725bfe7bdf0b8, 0xb1dd5415dfa3e82e63af394e48f0d4d8e686425d, 0x319736d0e5e921df081e50cb36c7b4ba3642a651, 0x9c562c804aedf1b506b61e45c83a789bf207b08b, 0xad6f905f533970f591df630670007e0c5315668d, 0xb920432447e50f21ebdeaa6e5431a8e91dd991d8, 0x045df4ea79b1f17e451b9eb5e153db8ae95fe79c, 0x6c87141be1105c37b8b27daa25975c1575c22fd5, 0xf02ba87b74453a54d5ebb5a048cbacb90046cdd9, 0x9430a215ffca453252137908caacd504c224e66d, 0xc56b672afd3fae873bf8a40bbe80f51075b187ca, 0x9e88dd25b6236e625709ca2830dee8d0c70bda7d, 0xfb7720716b3b2fa8940e3f46b7c53843718bf813, 0xbcc3f0c116cb7e98ad4fd981058dd047756574f2, 0x10b598dad5ddd32b85a1b8f4365615d6bdc42a78, 0x46495e58bc75cd44a8ec6ee6200a6035ef0332a0, 0x44bac1e2a2f6b44f71c5bfa8cfdcbeb73dac859b, 0xd28a4659df172152cf3812dec96eddaa99a38c42, 0xf37d30f8325771450c2ee1eb5b38bf4cc6e4338b, 0xc77b908353e648c3c08a9071e9129ea9fcda4e23, 0x25b02276431d29e481878140a0db68a8a1350212, 0x177e147e5552903629cce2e42361edd899ec7978, 0x5831d2fbbf925c5131c75a2197e73c416c966a59, 0x4c8fc2cad90bee5332e887d3c9184424e76b11e0, 0x292dada7a8cf9603e8ddfbd5bf2e580818a5ab76, 0xd7deccdc1cf5dfd34df8a03135162d7f294fedb0, 0x8afd93bf5bd9446c7a2070ba0b151f3f1cf1ff30, 0x46d8bcdb927e20a56b33bbd0da75e32b540af9f9, 0x63be1c775237f4c02f94396a7a7837d6c9b14424, 0x87449aa1412e34f0450cc5aa795200e05b050ae1, 0xa0827857fa91a9b4d720268b61b2dc96baa019fa, 0x9cd949e2ce9f1408dee663318f5281c5ed94e534, 0xc0c32dcf53c2ddd97087ee0042729705b2e6cc25, 0x54809ea74becdd734d2c4329729835ab39bb23f3, 0xa93c0d81ba7cd8a0c41016f09d02aa385586b23a, 0x904e3eab954eb80dabd5435db4bcc7413863c28a, 0x936fb33641b1e7dfe277d40c0c38621c8dcc4357, 0xa11214fdbce59e256b859ef0be2481acfd98a314, 0xf8eb4f3b9c179a5e0f76bc1a7a8b15a5b74996d9, 0xdad22897407d4f5e93e5d8b49b19abeba4b4bd89, 0x874008d366995b5aef76a5126d8a5f1ae934ab80, 0xabcae6fe36b82dfaeb8f4a21eab4ac67454e2a3d, 0xf7b159e913a3b137eac60256ac199e46c5cb13a6, 0x6e86b73cd9ec4cf5c18a618706c6d6b5a708a2aa]. nethermind_1 | 2023-06-05 12:22:05.7625|Processed block 18829722 (0x086a89...814712) is invalid: nethermind_1 | 2023-06-05 12:22:05.7625|- hash: expected 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b, got 0x086a898a1405546a3c0a06c5548b763166262d82f7abe04a5b1ad6acc0814712 nethermind_1 | 2023-06-05 12:22:05.7625|- gas used: expected 6000000, got 5818902 (diff: -181098) nethermind_1 | 2023-06-05 12:22:05.7625|- receipts root: expected 0x2c649a5d460f4d4a90df77002397f44178218d8f5641db1f53037695765b6113, got 0x08bb18651d8c18607189cddff21a05b8687faabef369eb7741452ceea1dd8f5a nethermind_1 | 2023-06-05 12:22:05.7625|- state root: expected 0xbee799d38eb3ad6e90a709accd42041bfed2731bee6fab4dca8754a2d5a4161c, got 0x964129cddaa6bd73a073db77d0ab483bcb687c4d1680b27020b25e8c939498ac nethermind_1 | 2023-06-05 12:22:05.7625|Processed block is not valid 18829722 (0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b) nethermind_1 | 2023-06-05 12:22:05.7625|Suggested block TD: 6407422370623267246122501039799329389803540696, Suggested block IsPostMerge False, Block TD: 6407422370623267246122501039799329389803540696, Block IsPostMerge False nethermind_1 | 2023-06-05 12:22:05.7764|Created a Receipts trace of block 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b in file /tmp/receipts_0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b.txt nethermind_1 | 2023-06-05 12:22:05.7780|Deleting an invalid block or its descendant 0x14c4751ac6e91c993d4dd38372f49d5fa222043285f399c068182770b17a553c nethermind_1 | 2023-06-05 12:22:05.7780|Deleting invalid block 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b at level 18829722 nethermind_1 | 2023-06-05 12:22:05.7780|Deleting an invalid block or its descendant 0x9b37d7c01bd8aa16818886858c876a2525d405cdab05e48e064a869f5315c9bd

/tmp/receipts_0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b.txt contents: [ { "transactionHash": "0x05d1416a0f8746b2ef8d205d77adee483465b37f476d52a7b8dbd1769c188401", "transactionIndex": "0", "blockHash": "0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b", "blockNumber": "18829722", "cumulativeGasUsed": "5818902", "gasUsed": "5818902", "effectiveGasPrice": "0x0", "from": "0xb5a911fbf5f5b6d46b19819d288de1ebbe1c7ddd", "to": null, "contractAddress": "0xd667c59d1e293324922da5a925399444f8e3d019", "logs": [], "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "status": "1", "type": "0x2" } ]

Result of eth_getBlockByHash 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b from an openethereum node. block.txt

Thank you

LukaszRozmej commented 1 year ago

Does the issue persist on restart? Can you post the chain spec used by OE node? So we can be sure they are compatible.

mclrch commented 1 year ago

Does the issue persist on restart? Can you post the chain spec used by OE node? So we can be sure they are compatible.

Yes, it persists after restart and I tried to delete the database too, always stop at same block. I'm using the chainspec below https://raw.githubusercontent.com/energywebfoundation/ewf-chainspec/master/EnergyWebChain.json

Demuirgos commented 1 year ago

also if possible, can you provide the vmTraces of the block from both Nethermind, and OpenEthereum ?

mclrch commented 1 year ago

also if possible, can you provide the vmTraces of the block from both Nethermind, and OpenEthereum ?

I don't know how to retrieve the vmtrace from nethermind since the block wasn't accepted From the openethereum I got it using the method trace_replayBlockTransactions I started Nethermind with trace log Hope it can help

energyweb.logs.txt vmtrace-openethereum.txt

LukaszRozmej commented 1 year ago

Can you try with option --Init.AutoDump=Parity ? Also maybe debug_resetHead to some older one would help.

mclrch commented 1 year ago

debug_resetHead

Hey, same result parityStyle_0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b.txt

2023-06-07 19:03:27.1020|INFO|24|Rerunning block after reorg or pruning: 18829718 (0xdbc4901c1fb881a56511fc4be68200218d2379d298f6b219f6d5c602d6400272) 2023-06-07 19:03:27.1020|INFO|24|Rerunning block after reorg or pruning: 18829719 (0xb1b442261c99732d5c69bb009d11ca712f23fb8533a72f86f5e4c3d2ac9e513d) 2023-06-07 19:03:27.1020|INFO|24|Rerunning block after reorg or pruning: 18829720 (0xeb0a62ba1b57e9acbce2ca041cefad91d9d28e4529497152e752c43ee4622d98) 2023-06-07 19:03:27.1020|INFO|24|Rerunning block after reorg or pruning: 18829721 (0x9b58ea53c060c36512a0421b0c1c58841853a5f4934cbb4d011aaa9b642dbb58) 2023-06-07 19:03:27.1109|ERROR|24|Processed block 18829722 (0x086a89...814712) is invalid: 2023-06-07 19:03:27.1109|ERROR|24|- hash: expected 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b, got 0x086a898a1405546a3c0a06c5548b763166262d82f7abe04a5b1ad6acc0814712 2023-06-07 19:03:27.1109|ERROR|24|- gas used: expected 6000000, got 5818902 (diff: -181098) 2023-06-07 19:03:27.1109|ERROR|24|- receipts root: expected 0x2c649a5d460f4d4a90df77002397f44178218d8f5641db1f53037695765b6113, got 0x08bb18651d8c18607189cddff21a05b8687faabef369eb7741452ceea1dd8f5a 2023-06-07 19:03:27.1109|ERROR|24|- state root: expected 0xbee799d38eb3ad6e90a709accd42041bfed2731bee6fab4dca8754a2d5a4161c, got 0x964129cddaa6bd73a073db77d0ab483bcb687c4d1680b27020b25e8c939498ac 2023-06-07 19:03:27.1109|ERROR|24|Processed block is not valid 18829722 (0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b) 2023-06-07 19:03:27.1109|ERROR|24|Suggested block TD: 6407422370623267246122501039799329389803540696, Suggested block IsPostMerge False, Block TD: 6407422370623267246122501039799329389803540696, Block IsPostMerge False 2023-06-07 19:03:27.1180|INFO|24|Current contract validators (48): [0xd65b4c25a4ce1e024ff13425df1e0e574a1a0e9b, 0x83329c3fd90d7ee2efd546e0dc6453e9172a0643, 0xde15831ac319dab5eae5fd1fd9d52876c5e50f20, 0x86a5a44cff58638784c2028e7181cede57933321, 0xb5da81b0c1c808ef6d761dacfba725bfe7bdf0b8, 0xb1dd5415dfa3e82e63af394e48f0d4d8e686425d, 0x319736d0e5e921df081e50cb36c7b4ba3642a651, 0x9c562c804aedf1b506b61e45c83a789bf207b08b, 0xad6f905f533970f591df630670007e0c5315668d, 0xb920432447e50f21ebdeaa6e5431a8e91dd991d8, 0x045df4ea79b1f17e451b9eb5e153db8ae95fe79c, 0x6c87141be1105c37b8b27daa25975c1575c22fd5, 0xf02ba87b74453a54d5ebb5a048cbacb90046cdd9, 0x9430a215ffca453252137908caacd504c224e66d, 0xc56b672afd3fae873bf8a40bbe80f51075b187ca, 0x9e88dd25b6236e625709ca2830dee8d0c70bda7d, 0xfb7720716b3b2fa8940e3f46b7c53843718bf813, 0xbcc3f0c116cb7e98ad4fd981058dd047756574f2, 0x10b598dad5ddd32b85a1b8f4365615d6bdc42a78, 0x46495e58bc75cd44a8ec6ee6200a6035ef0332a0, 0x44bac1e2a2f6b44f71c5bfa8cfdcbeb73dac859b, 0xd28a4659df172152cf3812dec96eddaa99a38c42, 0xf37d30f8325771450c2ee1eb5b38bf4cc6e4338b, 0xc77b908353e648c3c08a9071e9129ea9fcda4e23, 0x25b02276431d29e481878140a0db68a8a1350212, 0x177e147e5552903629cce2e42361edd899ec7978, 0x5831d2fbbf925c5131c75a2197e73c416c966a59, 0x4c8fc2cad90bee5332e887d3c9184424e76b11e0, 0x292dada7a8cf9603e8ddfbd5bf2e580818a5ab76, 0xd7deccdc1cf5dfd34df8a03135162d7f294fedb0, 0x8afd93bf5bd9446c7a2070ba0b151f3f1cf1ff30, 0x46d8bcdb927e20a56b33bbd0da75e32b540af9f9, 0x63be1c775237f4c02f94396a7a7837d6c9b14424, 0x87449aa1412e34f0450cc5aa795200e05b050ae1, 0xa0827857fa91a9b4d720268b61b2dc96baa019fa, 0x9cd949e2ce9f1408dee663318f5281c5ed94e534, 0xc0c32dcf53c2ddd97087ee0042729705b2e6cc25, 0x54809ea74becdd734d2c4329729835ab39bb23f3, 0xa93c0d81ba7cd8a0c41016f09d02aa385586b23a, 0x904e3eab954eb80dabd5435db4bcc7413863c28a, 0x936fb33641b1e7dfe277d40c0c38621c8dcc4357, 0xa11214fdbce59e256b859ef0be2481acfd98a314, 0xf8eb4f3b9c179a5e0f76bc1a7a8b15a5b74996d9, 0xdad22897407d4f5e93e5d8b49b19abeba4b4bd89, 0x874008d366995b5aef76a5126d8a5f1ae934ab80, 0xabcae6fe36b82dfaeb8f4a21eab4ac67454e2a3d, 0xf7b159e913a3b137eac60256ac199e46c5cb13a6, 0x6e86b73cd9ec4cf5c18a618706c6d6b5a708a2aa]. 2023-06-07 19:03:27.1251|ERROR|24|Processed block 18829722 (0x086a89...814712) is invalid: 2023-06-07 19:03:27.1251|ERROR|24|- hash: expected 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b, got 0x086a898a1405546a3c0a06c5548b763166262d82f7abe04a5b1ad6acc0814712 2023-06-07 19:03:27.1251|ERROR|24|- gas used: expected 6000000, got 5818902 (diff: -181098) 2023-06-07 19:03:27.1251|ERROR|24|- receipts root: expected 0x2c649a5d460f4d4a90df77002397f44178218d8f5641db1f53037695765b6113, got 0x08bb18651d8c18607189cddff21a05b8687faabef369eb7741452ceea1dd8f5a 2023-06-07 19:03:27.1251|ERROR|24|- state root: expected 0xbee799d38eb3ad6e90a709accd42041bfed2731bee6fab4dca8754a2d5a4161c, got 0x964129cddaa6bd73a073db77d0ab483bcb687c4d1680b27020b25e8c939498ac 2023-06-07 19:03:27.1251|ERROR|24|Processed block is not valid 18829722 (0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b) 2023-06-07 19:03:27.1251|ERROR|24|Suggested block TD: 6407422370623267246122501039799329389803540696, Suggested block IsPostMerge False, Block TD: 6407422370623267246122501039799329389803540696, Block IsPostMerge False 2023-06-07 19:03:27.1450|INFO|24|Created a Parity-style trace of block 0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b in file /tmp/parityStyle_0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b.txt

Demuirgos commented 1 year ago

also Can you try with option --Init.AutoDump=Geth? (I think it will result in a huge file but it would help a lot)

mclrch commented 1 year ago

also Can you try with option --Init.AutoDump=Geth? (I think it will result in a huge file but it would help a lot)

sure gethStyle_0x0b99558306b8e869f1121c2957b448daefcf4626011a92a12accd28cdd50d20b.txt

Demuirgos commented 1 year ago

hmmm, can you double check if you have any differences between your local chainspec and this https://raw.githubusercontent.com/energywebfoundation/ewf-chainspec/master/EnergyWebChain.json

mclrch commented 1 year ago

hmmm, can you double check if you have any differences between your local chainspec and this https://raw.githubusercontent.com/energywebfoundation/ewf-chainspec/master/EnergyWebChain.json

No differences: 

md5sum EnergyWebChain.json
dfbdead72a367aeba3fccea8ec444f4f  EnergyWebChain.json
Demuirgos commented 1 year ago

for now you can try this chainspec (from our codebase) instead, and tell us if it works as expected : https://[raw.githubusercontent.com/NethermindEth/nethermind/master/src/Nethermind/Chains/energyweb.json](https://raw.githubusercontent.com/NethermindEth/nethermind/master/src/Nethermind/Chains/energyweb.json)

mclrch commented 1 year ago

for now you can try this chainspec (from our codebase) instead, and tell us if it works as expected : https://[raw.githubusercontent.com/NethermindEth/nethermind/master/src/Nethermind/Chains/energyweb.json](https://raw.githubusercontent.com/NethermindEth/nethermind/master/src/Nethermind/Chains/energyweb.json)

That works! It's syncing...

mclrch commented 1 year ago

Node fully synced, thank you @Demuirgos @LukaszRozmej