search

paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.9k stars 1.14k forks source link

Fresh base op-reth sync crashing (critical error) #11955

Open CRossel87a opened 2 hours ago

CRossel87a commented 2 hours ago

Describe the bug

With --engine.experimental: 2024-10-22T05:53:42.266572Z INFO Block added to canonical chain number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 peers=4 txs=17 gas=3.75 Mgas gas_throughput=277.54 Mgas/second full=12.5% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=13.514576ms 2024-10-22T05:53:42.271360Z INFO Block added to canonical chain number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f peers=4 txs=7 gas=1.30 Mgas gas_throughput=273.92 Mgas/second full=4.3% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.761636ms 2024-10-22T05:53:42.271549Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=103.456µs 2024-10-22T05:53:52.216913Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9: add_branch key Nibbles("0c05") self.key Nibbles("0d0f")

Without: 2024-10-22T06:00:40.674219Z WARN Failed to insert downloaded block err=Failed to insert block (hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f, number=1985833, parent_hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370): block number is lower than the last finalized block number #1985833 2024-10-22T06:00:40.674275Z WARN Marking block as invalid invalid_number=1985833 invalid_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f err=block number is lower than the last finalized block number #1985833 2024-10-22T06:00:40.674292Z WARN Bad block with hash hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f header=Header { parent_hash: 0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xf654e2cbaaf85bff1751711de985dc2a5a080892cc284d6b1fa5af86a383b480, transactions_root: 0x7949f1cd25f597978932d4d16b3f6e5ecca994e679668c8a4762685dc4232357, receipts_root: 0x53d508521b7f5be34fe673aed3c7d59087e1a2e746fef41050a37a074830acb1, withdrawals_root: None, logs_bloom: 0x00201000000002000000000080000000000008100008000000840008000000800000000020000040000400100000000000000000000000000000000001200000800000240000400000000118000000200000000000402100000000008010000008400000080440080000000000000040010000000a0004100000201000600000000208000000000000120200002800000002120108200000000000400800800002004002004400040c00000000200000010080000800000001080208a00000008000000200000100000000040000080002002080005002000000000201000000001000000000002000000000010411020000000000000040000800000c000000, difficulty: 0, number: 1985833, gas_limit: 30000000, gas_used: 1304308, timestamp: 1690761013, mix_hash: 0x48b5afb9cedd5b522582acf10dff992885b8ac17b6cc56e8ff90f7c9166a5acd, nonce: 0x0000000000000000, base_fee_per_gas: Some(77), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, requests_hash: None, extra_data: 0x }

Steps to reproduce

Normal op-reth run with --full

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

reth-optimism-cli Version: 1.1.0-dev Commit SHA: f25cceb9f93fcceaae59a030f81f088e29a5f0c1 Build Timestamp: 2024-10-21T12:39:05.541893344Z Build Features: asm_keccak,jemalloc,optimism Build Profile: maxperf

What database version are you on?

Current database version: 2 Local database version: 2

Which chain / network are you on?

Base

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

CRossel87a commented 2 hours ago

Attempt fix: op-reth stage unwind --datadir /root/base --chain base num-blocks 100

Result:

2024-10-22T06:06:40.547935Z INFO Canonical chain committed number=1985829 hash=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7 elapsed=13.281116ms 2024-10-22T06:06:40.549489Z INFO Block is already canonical, ignoring. block_hash=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7 2024-10-22T06:06:40.549585Z INFO Canonical chain committed number=1985829 hash=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7 elapsed=62.124µs 2024-10-22T06:06:40.549711Z INFO New payload job created id=0x5016d6a1de2cbfce parent=0xebfd5bf6294242f7e372cd0c9f32ff442de47f7997f7f57693a50f2a8bc1c2e7 2024-10-22T06:06:40.563416Z INFO Block added to canonical chain number=1985830 hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 peers=2 txs=10 gas=2.11 Mgas gas_throughput=365.67 Mgas/second full=7.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=5.757786ms 2024-10-22T06:06:40.573192Z INFO Canonical chain committed number=1985830 hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 elapsed=9.222479ms 2024-10-22T06:06:40.574463Z INFO Block is already canonical, ignoring. block_hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 2024-10-22T06:06:40.574524Z INFO Canonical chain committed number=1985830 hash=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 elapsed=55.134µs 2024-10-22T06:06:40.574635Z INFO New payload job created id=0xdfc5b8e7260edf01 parent=0x4ee7c06f5f94f51557757f8aefcdbefbc2cbaa09cf9076d81e5fe6a40eb29401 2024-10-22T06:06:40.586913Z INFO Block added to canonical chain number=1985831 hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a peers=2 txs=10 gas=1.20 Mgas gas_throughput=257.46 Mgas/second full=4.0% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.65812ms 2024-10-22T06:06:40.597081Z INFO Canonical chain committed number=1985831 hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a elapsed=9.78652ms 2024-10-22T06:06:40.598732Z INFO Block is already canonical, ignoring. block_hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a 2024-10-22T06:06:40.598806Z INFO Canonical chain committed number=1985831 hash=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a elapsed=69.675µs 2024-10-22T06:06:40.598943Z INFO New payload job created id=0x13ed0e7d144b3e80 parent=0x234af71049f5e122a3cedcfeccdbc75e1a0a234d5cfd61c5a43de2a7c8cf5b5a 2024-10-22T06:06:40.598984Z INFO Pruner started tip_block_number=1985831 2024-10-22T06:06:40.599151Z INFO Pruner finished tip_block_number=1985831 elapsed=161.153µs stats=[] 2024-10-22T06:06:40.618532Z INFO Block added to canonical chain number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 peers=2 txs=17 gas=3.75 Mgas gas_throughput=477.89 Mgas/second full=12.5% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=7.848649ms 2024-10-22T06:06:40.636331Z INFO Canonical chain committed number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 elapsed=17.335485ms 2024-10-22T06:06:40.638037Z INFO Block is already canonical, ignoring. block_hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 2024-10-22T06:06:40.638131Z INFO Canonical chain committed number=1985832 hash=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 elapsed=69.125µs 2024-10-22T06:06:40.638253Z INFO New payload job created id=0x07579ffccfc6a862 parent=0x86a9d9f2501a417c9387312614ebcd09f05ad9e628f8df9145683ddeec832370 2024-10-22T06:06:40.651047Z INFO Block added to canonical chain number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f peers=2 txs=7 gas=1.30 Mgas gas_throughput=264.75 Mgas/second full=4.3% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=4.926644ms 2024-10-22T06:06:40.662012Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=10.589528ms 2024-10-22T06:06:40.663565Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f 2024-10-22T06:06:40.663634Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=59.729µs 2024-10-22T06:06:40.663727Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9: add_branch key Nibbles("0c05") self.key Nibbles("0d0f") stack backtrace: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. 2024-10-22T06:06:43.532761Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f 2024-10-22T06:06:43.532926Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=145.469µs 2024-10-22T06:06:43.533065Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9: add_branch key Nibbles("0c05") self.key Nibbles("0d0f") stack backtrace: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. 2024-10-22T06:06:47.786829Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f 2024-10-22T06:06:47.786988Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=139.05µs 2024-10-22T06:06:47.787170Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9: add_branch key Nibbles("0c05") self.key Nibbles("0d0f") stack backtrace: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. 2024-10-22T06:06:57.785464Z INFO Block is already canonical, ignoring. block_hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f 2024-10-22T06:06:57.785617Z INFO Canonical chain committed number=1985833 hash=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f elapsed=161.273µs 2024-10-22T06:06:57.785790Z INFO New payload job created id=0x95ee6cbb1a6e18b7 parent=0xb2f51472ae880ec6199792c2886fc75729340f715beb517c4c0698057d72ce3f thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/alloy-trie-0.7.2/src/hash_builder/mod.rs:123:9: add_branch key Nibbles("0c05") self.key Nibbles("0d0f") stack backtrace: note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

CRossel87a commented 2 hours ago

Trying new fresh run without -full but with engine experimental