paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
4.03k stars 1.24k forks source link

Unable to sync base-mainnet with op-reth || op-node hangs whilst op-reth is 'pruning' #7500

Closed dallonby closed 4 months ago

dallonby commented 8 months ago

Describe the bug

Trying to sync op-reth on base-mainnet. Using op-node, op-node often hangs when op-reth is 'pruning' (despite this being an archive node sync).

op-node shows following:

INFO [04-07|09:44:54.169] inserted block                           hash=66aadb..89cc2b number=58586 state_root=9b13d9..8889ac timestamp=1,686,906,519 parent=481ef4..e20181 prev_randao=0f039b..bc8f94 fee_recipient=0x4200000000000000000000000000000000000
011 txs=1 update_safe=true
INFO [04-07|09:44:54.170] Sync progress                            reason="processed safe block derived from L1" l2_finalized=df6d3d..9e8c4e:48857 l2_safe=66aadb..89cc2b:58586 l2_pending_safe=66aadb..89cc2b:58586 l2_unsafe=66aadb..89cc2b:58586 l2_backu
p_unsafe=000000..000000:0 l2_time=1,686,906,519 l1_derived=4d7b5c..499d92:17491420
INFO [04-07|09:44:54.170] Found next batch                         batch_type=SingularBatch batch_timestamp=1,686,906,521 parent_hash=66aadb..89cc2b batch_epoch=70d79f..43110c:17491408 txs=0
INFO [04-07|09:44:54.170] generated attributes in payload queue    txs=1 timestamp=1,686,906,521
INFO [04-07|09:44:54.178] inserted block                           hash=3a4ede..5c6eb7 number=58587 state_root=7cfbc1..657780 timestamp=1,686,906,521 parent=66aadb..89cc2b prev_randao=0f039b..bc8f94 fee_recipient=0x4200000000000000000000000000000000000
011 txs=1 update_safe=true
INFO [04-07|09:44:54.178] Sync progress                            reason="processed safe block derived from L1" l2_finalized=df6d3d..9e8c4e:48857 l2_safe=3a4ede..5c6eb7:58587 l2_pending_safe=3a4ede..5c6eb7:58587 l2_unsafe=3a4ede..5c6eb7:58587 l2_backu
p_unsafe=000000..000000:0 l2_time=1,686,906,521 l1_derived=4d7b5c..499d92:17491420
INFO [04-07|09:44:54.178] Found next batch                         batch_type=SingularBatch batch_timestamp=1,686,906,523 parent_hash=3a4ede..5c6eb7 batch_epoch=70d79f..43110c:17491408 txs=0
INFO [04-07|09:44:54.179] generated attributes in payload queue    txs=1 timestamp=1,686,906,523
ERROR[04-07|09:44:54.184] cancelling old block sealing job         payload=0x14ddafb523b060c7
WARN [04-07|09:44:54.186] could not process payload derived from L1 data, dropping batch err="failed to complete building on top of L2 chain 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7:58587, id: 0x14ddafb523b060c7, error (3): up
dated forkchoice, but node is syncing"
ERROR[04-07|09:44:54.186] deposit only block was invalid           parent=3a4ede..5c6eb7:58587 err="failed to complete building on top of L2 chain 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7:58587, id: 0x14ddafb523b060c7, error (
3): updated forkchoice, but node is syncing"
ERROR[04-07|09:44:54.186] Derivation process critical error        err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc7
33a2a9f84f5c6eb7:58587, id: 0x14ddafb523b060c7, error (3): updated forkchoice, but node is syncing"
INFO [04-07|09:44:54.186] State loop returned

N.B. setting block_interval = 5 under the prune setting in reth.toml to something obscene like 5000000000 seems to reduce the frequency of the error. When set at 5 I will see the error every 30 seconds or so, and have to restart op-node. With it set at 5000000000 I've not seen the error after ~30 mins of testing.

N.N.B. even setting log level to -vvvv in op-reth made it take much longer for the error to appear in op-node. Maybe 5 mins instead of every 30 seconds.

Steps to reproduce

start op-node ./bin/op-node --network="base-mainnet" --l1=http://127.0.0.1:8545 --l1.beacon=http://127.0.0.1:5052 --l2=http://localhost:9551 --l2.jwt-secret=/data/mev-geth/secret.jwt --rpc.addr=0.0.0.0 --rpc.port=9545

start op-reth ./target/release/op-reth node --chain base --rollup.sequencer-http https://sequencer.base.org --http --ws --authrpc.port 9551 --authrpc.jwtsecret /data/mev-geth/secret.jwt --port 30333 --discovery.port 30333 --ws.port 18546 --http.port 18545 --datadir /data/op-reth

Node logs

2024-04-07T09:44:54.178032Z DEBUG consensus::engine: Canonicalized new head hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 number=58587
2024-04-07T09:44:54.178253Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.178256Z  INFO reth_node_core::events::node: Canonical chain committed number=58587 hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 elapsed=1.403013ms
2024-04-07T09:44:54.178510Z DEBUG payload_builder: building new payload id=0x55e9d3849224cfdf parent_hash=0x0eaeddce536a5301cc5369efa7db499143af1568d961fae94195e3ceb902ae6c parent_number=58538
2024-04-07T09:44:54.178706Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58539
2024-04-07T09:44:54.179034Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xd2b64a930e12aec0cde6fef6ecfa24f21bd73cbac954f0eed98031aae385e117, header: Header { parent_hash: 0x0eaeddce536a5301cc5369efa7db499143af1568d961fae94195e3ceb902ae6c, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x7b5395807646415057b4b085ff51741a3b261695c995a29268d2a32e4bfb579e, transactions_root: 0xa8cc9befd38c48d3d9f88b9bbfd001748a25ea8c87cc48a1c501b51b017d51d4, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58539, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906425, mix_hash: 0x2e6b97ce5835ca7e380b8c6733bc6e0cf2c60531890cc83078eed19028eb389b, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x4f5a1a203b52be0944a69b3b90450dfac20dffe991c2d7f4f67d32d2ca729240, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc1bfc19c8c012764e34c8b9b346eca1153364114c97561d3d6809690e6e744d4, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c800000000000000000000000000000000000000000000000000000000648c25eb0000000000000000000000000000000000000000000000000000000362225146eae384e874a4720dadfabbb8792958faf7cfb7995718da1b5b52d55426ff9c1b00000000000000000000000000000000000000000000000000000000000000030000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.179235Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.179725Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7
2024-04-07T09:44:54.179918Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=58587 current_head_num=58587
2024-04-07T09:44:54.179995Z  INFO reth_node_core::events::node: Canonical chain committed number=58587 hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 elapsed=550.815µs
2024-04-07T09:44:54.180002Z DEBUG pruner: Minimum pruning interval reached previous_tip_block_number=Some(58582) tip_block_number=58587
2024-04-07T09:44:54.180036Z DEBUG pruner: Pruner started tip_block_number=58587
2024-04-07T09:44:54.180041Z  INFO reth_node_core::events::node: Pruner started tip_block_number=58587
2024-04-07T09:44:54.180427Z  INFO reth_payload_builder::service: New payload job created id=0x14ddafb523b060c7 parent=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7
2024-04-07T09:44:54.180015Z DEBUG consensus::engine::hooks: Polled next hook hook="Prune" result=PolledHook { name: "Prune", event: Started, db_access_level: ReadWrite }
2024-04-07T09:44:54.181030Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.181068Z DEBUG payload_builder: building new payload id=0x14ddafb523b060c7 parent_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 parent_number=58587
2024-04-07T09:44:54.181511Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329, header: Header { parent_hash: 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x02c620a6df954aa2fb456d7abb7fba3fdbdea2269a05efeb560091ea8957ea95, transactions_root: 0x5ae1634b08b4f90478bee4ddf84f37ec6069a6511fae22c1a80d7ee69c157ba2, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58588, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906523, mix_hash: 0x0f039b421747a6c6132b14b9178a1fc2c7633f17e974ed3ec36a958e35bc8f94, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x3ed7acd44814f15321c310098d5d2bf288084eade5b78f281ecdcea2141cfd71, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc4d9bf18c0a7b02f2cb6ef25fbb70f0f99e07dba9f36825b0844c11a1fd81cf2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5d000000000000000000000000000000000000000000000000000000000648c264b0000000000000000000000000000000000000000000000000000000358e19b7270d79f07b4b2d8db4a3a2dc3ad1297d988c6f8bed5059556c9ef26d42043110c00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.181538Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.181588Z DEBUG payload_builder: building new payload id=0x14ddafb523b060c7 parent_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 parent_number=58587
2024-04-07T09:44:54.181955Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329, header: Header { parent_hash: 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x02c620a6df954aa2fb456d7abb7fba3fdbdea2269a05efeb560091ea8957ea95, transactions_root: 0x5ae1634b08b4f90478bee4ddf84f37ec6069a6511fae22c1a80d7ee69c157ba2, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58588, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906523, mix_hash: 0x0f039b421747a6c6132b14b9178a1fc2c7633f17e974ed3ec36a958e35bc8f94, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x3ed7acd44814f15321c310098d5d2bf288084eade5b78f281ecdcea2141cfd71, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc4d9bf18c0a7b02f2cb6ef25fbb70f0f99e07dba9f36825b0844c11a1fd81cf2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5d000000000000000000000000000000000000000000000000000000000648c264b0000000000000000000000000000000000000000000000000000000358e19b7270d79f07b4b2d8db4a3a2dc3ad1297d988c6f8bed5059556c9ef26d42043110c00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.181980Z DEBUG payload_builder: resolving fallback payload as best payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.182291Z DEBUG payload_builder: building new payload id=0x1afa30385c2a133b parent_hash=0x42d573473e2e53140ed1335ef660ad403179c99ac4cf8142b04c03c00be4dce6 parent_number=58201
2024-04-07T09:44:54.182404Z DEBUG payload_builder: building new payload id=0x9eb78b76c133d825 parent_hash=0x3e97904bbd92511c38bc3597c4c9aa0e4c5707c30f92404d21ab666cdf889898 parent_number=58487
2024-04-07T09:44:54.182806Z DEBUG payload_builder: resolving better payload
2024-04-07T09:44:54.183649Z DEBUG blockchain_tree: Appending block to canonical chain head=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 parent=(58587, 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7)
2024-04-07T09:44:54.183863Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T09:44:54.184265Z DEBUG blockchain_tree::chain: Validated state root number=58588 hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 elapsed=438.048µs
2024-04-07T09:44:54.184308Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.184323Z  INFO reth_node_core::events::node: Block added to canonical chain number=58588 hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 peers=0 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=976.4µs
2024-04-07T09:44:54.184681Z  WARN consensus::engine: Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator. hook=Prune head_block_hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 safe_block_hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 finalized_block_hash=0xdf6d3d5c2496182bf82f59ea7fa1ce75f8f4012e1c2aa44b9b9b3d4e859e8c4e
2024-04-07T09:44:54.184695Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.185550Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.185597Z DEBUG payload_builder: building new payload id=0x14ddafb523b060c7 parent_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 parent_number=58587
2024-04-07T09:44:54.186031Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329, header: Header { parent_hash: 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x02c620a6df954aa2fb456d7abb7fba3fdbdea2269a05efeb560091ea8957ea95, transactions_root: 0x5ae1634b08b4f90478bee4ddf84f37ec6069a6511fae22c1a80d7ee69c157ba2, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58588, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906523, mix_hash: 0x0f039b421747a6c6132b14b9178a1fc2c7633f17e974ed3ec36a958e35bc8f94, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x3ed7acd44814f15321c310098d5d2bf288084eade5b78f281ecdcea2141cfd71, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc4d9bf18c0a7b02f2cb6ef25fbb70f0f99e07dba9f36825b0844c11a1fd81cf2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5d000000000000000000000000000000000000000000000000000000000648c264b0000000000000000000000000000000000000000000000000000000358e19b7270d79f07b4b2d8db4a3a2dc3ad1297d988c6f8bed5059556c9ef26d42043110c00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.186073Z DEBUG payload_builder: resolving fallback payload as best payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.186091Z DEBUG payload_builder: resolving best payload
2024-04-07T09:44:54.188357Z DEBUG payload_builder: building new payload id=0xba9b204a7803fbd3 parent_hash=0xd2b64a930e12aec0cde6fef6ecfa24f21bd73cbac954f0eed98031aae385e117 parent_number=58539
2024-04-07T09:44:54.188503Z DEBUG payload_builder: building new payload id=0xf215c90466ac9926 parent_hash=0x87b6beab35817901b2dc15f0686a1864bf8a0450de5e949d4033a5f951d3109e parent_number=58488
2024-04-07T09:44:54.188576Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58540
2024-04-07T09:44:54.188921Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xa4165e67354c3022630d96d8cf3fa31190c0aaa68869089dff428bc9b78444a4, header: Header { parent_hash: 0xd2b64a930e12aec0cde6fef6ecfa24f21bd73cbac954f0eed98031aae385e117, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6df72d55185e73207564fbbf2ec515af8971d8f14d35bdb77b778b7ab50d1722, transactions_root: 0xb20b44f5397c71bb924858e586c47de1b6aa4a21c7878aa8d875f3adc3108f27, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58540, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906427, mix_hash: 0x2e6b97ce5835ca7e380b8c6733bc6e0cf2c60531890cc83078eed19028eb389b, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x920ffd8bc230bdfe1f61b3bed7712d067b5f137f03c3f4503029f6e390e4c7d5, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x75e36bd155036249db3de9f81d84a17c18fca568da2893673c285822a2fad3ce, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c800000000000000000000000000000000000000000000000000000000648c25eb0000000000000000000000000000000000000000000000000000000362225146eae384e874a4720dadfabbb8792958faf7cfb7995718da1b5b52d55426ff9c1b00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.189091Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.191100Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=48857 prune_mode=Before(48858)
2024-04-07T09:44:54.191146Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=48857 prune_mode=Before(48858) output.pruned=0
2024-04-07T09:44:54.191153Z DEBUG pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=48857 prune_mode=Before(48858)
2024-04-07T09:44:54.191162Z DEBUG pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=48857 prune_mode=Before(48858) output.pruned=0
2024-04-07T09:44:54.191169Z DEBUG pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=48857 prune_mode=Before(48858)
2024-04-07T09:44:54.191175Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=48857 prune_mode=Before(48858) output.pruned=0
2024-04-07T09:44:54.191205Z DEBUG storage::db::mdbx: Commit total_duration=18.763µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T09:44:54.191231Z DEBUG pruner: Pruner finished tip_block_number=58587 elapsed=11.179791ms delete_limit=8500 progress=Finished stats={}
2024-04-07T09:44:54.191270Z  INFO reth_node_core::events::node: Pruner finished tip_block_number=58587 elapsed=11.179791ms stats={}
2024-04-07T09:44:54.191282Z DEBUG consensus::engine::hooks: Polled running hook with db write access hook="Prune" result=PolledHook { name: "Prune", event: Finished(Ok(())), db_access_level: ReadWrite }
2024-04-07T09:44:54.191468Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.193700Z DEBUG payload_builder: building new payload id=0x1a524539692c8fa0 parent_hash=0xba8e557c674771dec883112af0e9e230a5aeb2ca3e5e815eb2e5a73ce824cc39 parent_number=58202
2024-04-07T09:44:54.195250Z DEBUG payload_builder: building new payload id=0xbe169363e05f7bb1 parent_hash=0xdc1b728ec2cac433e3423d2c9c48ed5e81ff597bde5f80d51914cfffe883a903 parent_number=58258
2024-04-07T09:44:54.196768Z DEBUG payload_builder: building new payload id=0x1d7ec23d848a530c parent_hash=0xea290dda7668ccd114aa6e5099ef8d43c1dd6ccbc88a5c2b0aca6301cdbf1b74 parent_number=58489
2024-04-07T09:44:54.198178Z DEBUG payload_builder: building new payload id=0x260a466b18dc9663 parent_hash=0x6975478d2b5c2fb6a658697cccd5a537b476fc66b434c2e5f19744f7e4aaa1f3 parent_number=58259
2024-04-07T09:44:54.198328Z DEBUG payload_builder: building new payload id=0x73ffe70330eb135b parent_hash=0xa4165e67354c3022630d96d8cf3fa31190c0aaa68869089dff428bc9b78444a4 parent_number=58540
2024-04-07T09:44:54.198472Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58541
2024-04-07T09:44:54.198804Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x9a1637cd5266813e2c1552652693b6e62e6a55ce92fcbceef7b909f3c7e53871, header: Header { parent_hash: 0xa4165e67354c3022630d96d8cf3fa31190c0aaa68869089dff428bc9b78444a4, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x8cf98309b43cfca46d63f10844fc6e1dd864693f769721b71c9ef9025e4ce03b, transactions_root: 0x430eb5394508edbb1f2f73928a5c6120f2cfce0b9a650704405c761f62c97c89, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58541, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906429, mix_hash: 0x2e6b97ce5835ca7e380b8c6733bc6e0cf2c60531890cc83078eed19028eb389b, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xc766d7dd3d1a8089d03f2e3649e5b5b5f124053cded78d41a42d749b3eea2e04, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xfb486b24abc723b921ba6a61d15edd59e06ae27a185a8b29c59c9445291623fe, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c800000000000000000000000000000000000000000000000000000000648c25eb0000000000000000000000000000000000000000000000000000000362225146eae384e874a4720dadfabbb8792958faf7cfb7995718da1b5b52d55426ff9c1b00000000000000000000000000000000000000000000000000000000000000050000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.199055Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.204277Z DEBUG payload_builder: building new payload id=0x4bba997a7c12b9e5 parent_hash=0x4bc58cc28e8b6c0746dfc99a62472c7e6c3089b8fcb1deac2c545f139fbb96ca parent_number=58490
2024-04-07T09:44:54.205958Z DEBUG payload_builder: building new payload id=0x369b307a0a932359 parent_hash=0x5735beadd4944533f6e167419f86a8bec195d9316e4a1b92021e2c195383410b parent_number=58260
2024-04-07T09:44:54.208306Z DEBUG payload_builder: building new payload id=0x14ec4c52c4c4d2ee parent_hash=0x06922fe25a277ae3c01b9effaeefcb8d163bcf4017bd36b9859c20147950f3b2 parent_number=58203
2024-04-07T09:44:54.208339Z DEBUG payload_builder: building new payload id=0x7011b34027e78fdb parent_hash=0x7776dedeef4d550e8e1f4b65d51b547b2349ebe03697eb0f3f2bf116efa74039 parent_number=58261
2024-04-07T09:44:54.208428Z DEBUG payload_builder: building new payload id=0xb2860c84d5470f0c parent_hash=0x9a1637cd5266813e2c1552652693b6e62e6a55ce92fcbceef7b909f3c7e53871 parent_number=58541
2024-04-07T09:44:54.208581Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58542
2024-04-07T09:44:54.208943Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xcd6385f63a0443e3fd57e101332ecb88e11a18a70013467cc55b889fced40966, header: Header { parent_hash: 0x9a1637cd5266813e2c1552652693b6e62e6a55ce92fcbceef7b909f3c7e53871, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x8ceb5ec4c4b080e39437dc61191c35acfbe1c1dc16d41155a423faa1e952d919, transactions_root: 0x35519e7c1d2ed2678c7c4fd0686a9874640d0bce0174e1814dc055ba9f6c5df8, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58542, gas_limit: 30000000, gas_used: 50501, timestamp: 1686906431, mix_hash: 0x49ccf354359ba9aa65866d292b5f461c54a3605a70b06db21687afd260154706, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xa4d3f6bd3f2be0e300390658d41a6ffc340ee6f7ed92ee32c8c9e8da85286500, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xa30069c0f9b2a074d348962442cea4bc1132c17ef666d5ed0896e17efe3acd97, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c900000000000000000000000000000000000000000000000000000000648c25f700000000000000000000000000000000000000000000000000000003432d7dbdd10eaf160ab24585f4b9e061e958ecd2d567a9b34faa0793d4ed740382f485db00000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.209097Z DEBUG payload_builder: building new payload id=0x0d14f4c100545f9d parent_hash=0x9c18dae0ac7e83b893ee489aae89acd1d01e52b463d3350bde3b7c279d1c4300 parent_number=58491
2024-04-07T09:44:54.209237Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.213710Z DEBUG payload_builder: building new payload id=0x441c548376a6baef parent_hash=0xaa2d17125ec3088f5f856ac718e3bc622d6a5df1f52138ae671104cf716efb92 parent_number=58262
2024-04-07T09:44:54.215146Z DEBUG payload_builder: building new payload id=0xa97a415220c9e159 parent_hash=0x193e5c2c9f3e8650616d73b80b773b376978d74de1830e946f2fd89e6c03eae2 parent_number=58492
2024-04-07T09:44:54.216510Z DEBUG payload_builder: building new payload id=0x750e432b897393ae parent_hash=0x87331051e41c45d6c86fb7a053a43b352c6e0b2b833259aa931a8b44e8de71c1 parent_number=58263
2024-04-07T09:44:54.219919Z DEBUG payload_builder: building new payload id=0x7d18e3dbf094bd23 parent_hash=0xcd6385f63a0443e3fd57e101332ecb88e11a18a70013467cc55b889fced40966 parent_number=58542
2024-04-07T09:44:54.220145Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58543
2024-04-07T09:44:54.220201Z DEBUG payload_builder: building new payload id=0x6f18699b1180833c parent_hash=0x04deb25b5a95d78e460b0e13429ff94c396ced02396777b7b279f4e57b1bd5d6 parent_number=58406
2024-04-07T09:44:54.220496Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x72fc8d58f91fd3f09b5091a5dd4121873eb46a49808dbf71aeb848ab91e1163b, header: Header { parent_hash: 0xcd6385f63a0443e3fd57e101332ecb88e11a18a70013467cc55b889fced40966, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x8b546fe63827af69faf63a53d4475652ff585db568756b93fc91097b69cc469b, transactions_root: 0xc3f81934a3e121d72ae7da996d14a847efc01c5c23fdfabce9c0c563c6ac2c2a, receipts_root: 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58543, gas_limit: 30000000, gas_used: 64013, timestamp: 1686906433, mix_hash: 0x49ccf354359ba9aa65866d292b5f461c54a3605a70b06db21687afd260154706, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x50272fb4bc55ff1b6df13040eddaaea8d1a580243ee736250429ffec37e350bd, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc43bf4a3e7c9777891d52297f2eda3b81cfabfeb0094a109b410aec9deb60e87, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c900000000000000000000000000000000000000000000000000000000648c25f700000000000000000000000000000000000000000000000000000003432d7dbdd10eaf160ab24585f4b9e061e958ecd2d567a9b34faa0793d4ed740382f485db00000000000000000000000000000000000000000000000000000000000000010000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.220641Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.222903Z DEBUG payload_builder: building new payload id=0x73e5f84fac2f7ce2 parent_hash=0xf9fd5d17664fb8a936483f069b58859884d7612a4b9e2927fd14742230644fbc parent_number=58493
2024-04-07T09:44:54.224437Z DEBUG payload_builder: building new payload id=0x71b84955ebbda7f0 parent_hash=0xf2a28233d35e569739d1e61d5cf928264e4a8f62a6b71dd43e6c0418015e003b parent_number=58264
2024-04-07T09:44:54.226899Z DEBUG payload_builder: building new payload id=0x9d5c805739102ce9 parent_hash=0xe3a372e7583af9122492e4d93aeaf5198f34daf9e3828d765e23459a576ef677 parent_number=58204
2024-04-07T09:44:54.226899Z DEBUG payload_builder: building new payload id=0x88922a230887bb78 parent_hash=0x7b213573e3d35904660632fb79c2afd4fc105c72e519f2d874da0444f9616b3c parent_number=58407
2024-04-07T09:44:54.227091Z DEBUG payload_builder: building new payload id=0x03f10b6d36148495 parent_hash=0x7da5fb624f431f93708d6364237da2276bb595c18eaec525c3a2987489ec656b parent_number=58265
2024-04-07T09:44:54.227112Z DEBUG payload_builder: building new payload id=0xb4dda4e18a04abe7 parent_hash=0x72fc8d58f91fd3f09b5091a5dd4121873eb46a49808dbf71aeb848ab91e1163b parent_number=58543
2024-04-07T09:44:54.227252Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58544
2024-04-07T09:44:54.227595Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x87c160a636c53b96574226dbf8a673a99bc9e73f30848f656143ea5cdae7694c, header: Header { parent_hash: 0x72fc8d58f91fd3f09b5091a5dd4121873eb46a49808dbf71aeb848ab91e1163b, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xad50be7c814f8079fd6b230169788c7566907674716e5685e1839bbb9b0d1c7d, transactions_root: 0x99090cac9b4e48e0bbdf46120fde5319a8c6e7866126eddcf6fbfba090457b48, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58544, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906435, mix_hash: 0x49ccf354359ba9aa65866d292b5f461c54a3605a70b06db21687afd260154706, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x566c6b54ee1a6a912fbc0310b2723a01ba27e37d2a52cf8c725cbf17ff1f8fb4, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xb4cc0e07ec6f39695e04d11a8da90786fa34b8925251429502d2e88ec73c2621, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c900000000000000000000000000000000000000000000000000000000648c25f700000000000000000000000000000000000000000000000000000003432d7dbdd10eaf160ab24585f4b9e061e958ecd2d567a9b34faa0793d4ed740382f485db00000000000000000000000000000000000000000000000000000000000000020000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.227775Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.232022Z DEBUG payload_builder: building new payload id=0xe2075cf95b7dec75 parent_hash=0x22e0b6bcccecb0dfb863482ffcbddbe9884a0075e2847cbd3c9fcf9c440822ae parent_number=58408
2024-04-07T09:44:54.233551Z DEBUG payload_builder: building new payload id=0x73493bfdb8915fa1 parent_hash=0x0ab4185c8fbf71734efd995f585a1bb5052647f3b3baeb805fa0c7a79796fef2 parent_number=58266
2024-04-07T09:44:54.238005Z DEBUG payload_builder: building new payload id=0x918ff1adaa7a89bc parent_hash=0xcf1148ff3427f8c1c1e33a373b21596eb5c46fced2154e16d2efeb31c7505725 parent_number=58409
......
......
......
2024-04-07T09:44:54.366736Z DEBUG payload_builder: building new payload id=0x013e88ed859f72ad parent_hash=0xb4e82429cf20ef15ea60b159b7ffaff8b85de1614f9e111411d8673421f063c9 parent_number=58220
2024-04-07T09:44:54.369550Z DEBUG payload_builder: building new payload id=0x69c641703ff6b29e parent_hash=0xd67a6094661ca2052f71247655bf4e7f8c53bf1e16576f0b3366eb2404e804ce parent_number=58350
2024-04-07T09:44:54.371029Z DEBUG payload_builder: building new payload id=0x087fbff9a979b7fd parent_hash=0x96f3e254da22c7131537b8810bd0094d244e69374798d5fa1d9ad47613a85e7e parent_number=58221
2024-04-07T09:44:54.371276Z DEBUG payload_builder: building new payload id=0xa35c374ff84ffa5d parent_hash=0x62b37e19857462686b74df0f795072cad3dc761598669b33f0b49d3dfdda8302 parent_number=58351
2024-04-07T09:44:54.373679Z DEBUG payload_builder: building new payload id=0x4071fe2e79d61fac parent_hash=0xa8989cdd042c3268f00dbac3a65f5e6c6046e79d07f93d344ac906cf03e10842 parent_number=58222
2024-04-07T09:44:54.378480Z DEBUG payload_builder: building new payload id=0x26eaf4207312fe83 parent_hash=0x91d0f29a54be9358809160531d98de9c2d3f81afcc7c7fcc7fb605735ef5a32c parent_number=58223
2024-04-07T09:44:54.378480Z DEBUG payload_builder: building new payload id=0xcb56b34e39b1423b parent_hash=0x2fdf5627712726df13271293b1f1457f015ba8e3bcaf2443662992eef4fcafd2 parent_number=58352

Platform(s)

Linux (x86)

What version/commit are you on?

./target/release/op-reth --version reth Version: 0.2.0-beta.4 Commit SHA: c04dbe6e9 Build Timestamp: 2024-04-07T07:19:12.388844682Z Build Features: jemalloc,optimism Build Profile: release

What database version are you on?

NA

What type of node are you running?

Archive (default)

What prune config do you use, if any?

[prune] block_interval = 5

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

dallonby commented 8 months ago

After these log events, op-reth spams messages such as

2024-04-07T09:45:00.634527Z DEBUG payload_builder: building new payload id=0x1561046804b861b6 parent_hash=0xde6639ef3f31d04233f0921fca06a4f112714c448bc8a65500479a55cb12edd1 parent_number=58446
2024-04-07T09:45:00.634613Z DEBUG payload_builder: building new payload id=0x2262a9cf1fa79729 parent_hash=0xb27241ef8c569b13fa40f15d5cea1ca57b15a328bd38c620706a6ca738ca64a5 parent_number=58310
2024-04-07T09:45:00.636130Z DEBUG payload_builder: building new payload id=0x8beb4a469d4fddd2 parent_hash=0xf0502c09642001d123dd0104e75960887d87d096b8015fe437d7b2bbe11c53d6 parent_number=58311
2024-04-07T09:45:00.636295Z DEBUG payload_builder: building new payload id=0xdc1fdcc90a999af7 parent_hash=0x3321d5f1497d97c4cd23c5a46faac889faf453eaee07358c25418c6ff2fa5d49 parent_number=58517
2024-04-07T09:45:00.640562Z DEBUG payload_builder: building new payload id=0x12e6dd8e2508b5d6 parent_hash=0xc8dae84337bf08385ff6dcea566284c0284ec5f551598095cea1344e7b5a11f8 parent_number=58312
2024-04-07T09:45:00.648222Z DEBUG payload_builder: building new payload id=0x60633afcc3fc664c parent_hash=0x7c01e99d31cce9964e97e0f71e99f7b76d18534a9065704dc06ff600263fc288 parent_number=58448
2024-04-07T09:45:00.653814Z DEBUG payload_builder: building new payload id=0x5e14d70db8b596e0 parent_hash=0x8c6ed433b65d4324d4a60b60f3e37b349f1c35344e627d8fb88d40d322ff190b parent_number=58313
2024-04-07T09:45:00.653874Z DEBUG payload_builder: building new payload id=0x0d7842228438f96d parent_hash=0x1d17daf6fc831f4835ad0fd473d06ebf68237c9d20229880bae732e7017d583c parent_number=58519
2024-04-07T09:45:00.656764Z DEBUG payload_builder: building new payload id=0xad917f67321186e1 parent_hash=0x096dcad704c5e2ffeb88f0835047f4f3bbc4bba4c0134126481522071b76ff0e parent_number=58314
2024-04-07T09:45:00.659500Z DEBUG payload_builder: building new payload id=0x38e1f1ce49997479 parent_hash=0xc9c688436abcd92750ead9aca48e2425ee84739ba232065be82705b27db9d6e7 parent_number=58449
2024-04-07T09:45:00.663310Z DEBUG payload_builder: building new payload id=0xf898662c492e2432 parent_hash=0x60c4a80ec809536e39557917122e0b224494db9ddd3044fab9bffc284724af89 parent_number=58520
2024-04-07T09:45:00.663392Z DEBUG payload_builder: building new payload id=0xed79f11671c454ab parent_hash=0x392598169cbc2c82b2e13cc8533c5f2035e56141f1b6ecacb25a538014cff743 parent_number=58316
2024-04-07T09:45:00.664223Z DEBUG payload_builder: building new payload id=0x40c08df0a246e6bf parent_hash=0xc02093fabcdddb6df32c83aa5efc8e28f9d968049e2a69752e26475e6c7890cf parent_number=58521

with the same set of payload ids (ie 0x40c08df0a246e6bf) repeating themselves over and over

dallonby commented 8 months ago

Possibly duplicates https://github.com/paradigmxyz/reth/issues/7477

zilayo commented 7 months ago

facing a similar issue.

base archive node sync with no pruning config, but it still attempts to prune every 5 blocks anyway (nothing is actually pruned).

for me the error seems to be due to block 50390.

op-node runs into a critical error so further derivation stops completely - https://github.com/ethereum-optimism/optimism/blob/8f516faf42da416c02355f9981add3137a3db190/op-node/rollup/derive/engine_queue.go#L639-L645

for blocks where the only txs are deposit-txs op-node expects that the NewPayload status will always be valid, otherwise it crit errors as above - https://github.com/ethereum-optimism/optimism/blob/8f516faf42da416c02355f9981add3137a3db190/op-node/rollup/derive/engine_update.go#L168-L170

since reth returns FCU status "syncing" when there's an active db write hook, this seems to be causing the error - https://github.com/paradigmxyz/reth/blob/c6857efa8a10e2e86396ac6385d1008b8128e694/crates/consensus/beacon/src/engine/mod.rs#L360-L372

For me, the pruner starts and then takes 13ms to finish (compared to the usual sub 200 µs prune durations). Whilst the pruner is running, reth is also processing op-node's engine API messages for block 50390 (payload id 0x6d5c5904c42aee43 in the logs).

It seems that during the time the pruner has a write hook on the db, op-node errors out due to the syncing status.

Seen a few people mention in tg or in issues about the op node error: msg="failed to notify engine driver of L1 head change" err="context deadline exceeded" - the crit error kills the op-node driver event loop so this will be the cause of that error message.

After all this, op-node loses all peers and further peer connections are refused.

Reth then continually tries building new payloads, but since op-node is essentially dead, there's nothing else it can do.

op node logs

op-node-1  | t=2024-04-07T16:28:06+0000 lvl=info msg="Sync progress" reason="processed safe block derived from L1" l2_finalized=0xe1a452f10433d50c9cf87b5b9703507fe5fe910cf8ccea31d9d0b927b86a6015:50318 l2_safe=0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390 l2_pending_safe=0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390 l2_unsafe=0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390 l2_time=1686890127 l1_derived=0xf74258f232380be257fa7c19c379b96f9d3b25a20debe74f06ff1772258402b5:17490062
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=info msg="Found next batch" batch_type=SingularBatch batch_timestamp=1686890129 parent_hash=0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df batch_epoch=0xffd93b6dcc14289d573e9729ea69177a80432a1b4e1ce12c119fc6040987d804:17490053 txs=0
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1686890129
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=eror msg="cancelling old block sealing job" payload=0x6d5c5904c42aee43
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=warn msg="could not process payload derived from L1 data, dropping batch" err="failed to complete building on top of L2 chain 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390, id: 0x6d5c5904c42aee43, error (3): updated forkchoice, but node is syncing"
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=eror msg="deposit only block was invalid" parent=0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390 err="failed to complete building on top of L2 chain 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390, id: 0x6d5c5904c42aee43, error (3): updated forkchoice, but node is syncing"
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=eror msg="Derivation process critical error" err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df:50390, id: 0x6d5c5904c42aee43, error (3): updated forkchoice, but node is syncing"
op-node-1  | t=2024-04-07T16:28:06+0000 lvl=info msg="State loop returned"
op-node-1  | t=2024-04-07T16:28:07+0000 lvl=info msg="Received signed execution payload from p2p" id=0x05467a95e44070a2f758995a430e8753851327742464ac55c2ebbc229991de8d:12858970 peer=16Uiu2HAmQBkteadMT9oxAXHnNewQVJvgnbJ2WonTCgVXc1ffFaMP
op-node-1  | t=2024-04-07T16:28:08+0000 lvl=info msg="disconnected from peer" peer=16Uiu2HAm4wN4epAyuj82ix2QC1Uc2vQd833szwvsNpLqhubPz8xa addr=/ip4/3.231.138.188/tcp/9222
op-node-1  | t=2024-04-07T16:28:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0x9c8fb2768fc7d1446caca4c76eefccee494851e35dc4dde1851cd628c325f1cd:12858971 peer=16Uiu2HAm17riA2iLFzUf5jJzfNAYi8oibBR1gYdb27g7E8pTmMeH
op-node-1  | t=2024-04-07T16:28:10+0000 lvl=info msg="disconnected from peer" peer=16Uiu2HAkwjBmkSB21QMHKjfAoFoN49UPveFVEbzh9Po6MKPJB3Nk addr=/ip4/148.251.235.60/tcp/9222
op-node-1  | t=2024-04-07T16:28:11+0000 lvl=info msg="Received signed execution payload from p2p" id=0xfa72a3c82184c1ff54d132e79e912e2053bc3d87b0820d1f28394a9b17a8cbda:12858972 peer=16Uiu2HAmQBkteadMT9oxAXHnNewQVJvgnbJ2WonTCgVXc1ffFaMP

reth logs from relevant time

{"timestamp":"2024-04-07T16:28:06.616207Z","level":"INFO","fields":{"message":"New payload job created","id":"0x6d5c5904c42aee43","parent":"0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df"},"target":"reth_payload_builder::service"}
{"timestamp":"2024-04-07T16:28:06.618972Z","level":"DEBUG","fields":{"message":"Pruner started","tip_block_number":"50390"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.619134Z","level":"INFO","fields":{"message":"Pruner started","tip_block_number":50390},"target":"reth_node_core::events::node"}
{"timestamp":"2024-04-07T16:28:06.623023Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x3eb9ad75ca21fa37","parent_hash":"0x50df631ba9cb6a1cf8d8ee5512a98d0b9c8edc38446f32dbd148438606df430a","parent_number":50053},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623018Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x6d5c5904c42aee43","parent_hash":"0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df","parent_number":50390},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623084Z","level":"DEBUG","fields":{"message":"building new payload","id":"0xb54780f914b88d7d","parent_hash":"0xb96d0775c80379361d6504fb4437704f7ebb5caf2df3e224ae896795f0bdb2a0","parent_number":49686},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623232Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x4678521d25a5078d","parent_hash":"0x62f6c03d8d20bf5ac719cbcfe6effd5f818691182be4f5b0af8aa1b4759a7f39","parent_number":49760},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623373Z","level":"DEBUG","fields":{"message":"building new payload","id":"0xb9fd4edd603ff9ca","parent_hash":"0x54cb9f9bc7513629cd3120db8dc1480f95e913e105b9b00bb58295f7445e47b0","parent_number":50317},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623399Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x450ccd7a398637e9","parent_hash":"0x2685ceb846d843c9575d816dca485b8531def21b8b112525593c1cf4b5e69830","parent_number":49897},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623489Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x615c95700540ba7e","parent_hash":"0x8f5a7841044921dc5599538e8f89cb85375f17f72bdce748f1bfc2b0143486f2","parent_number":49685},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623591Z","level":"WARN","fields":{"message":"Attempt to calculate state root for an old block might result in OOM, tread carefully","target":50318},"target":"provider::historical_sp"}
{"timestamp":"2024-04-07T16:28:06.623637Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x0a7dd1cb23fa658d","parent_hash":"0xa8a8ab5b44ed92e230d15fb8614fa55c57970b731cf0a33a247ba558b05ddaab","parent_number":49980},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623781Z","level":"DEBUG","fields":{"message":"sealed built block","sealed_block":"SealedBlock { header: SealedHeader { hash: 0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a, header: Header { parent_hash: 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b
033386a6452a9c63df, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xc8a83fbafb96f282f088414a2ace1bed25a70229c0e2ffe885a85654589262f3, transactions_root: 0x978e98959d7ce140b1e31ecd5fe3aabbc025dd3dd4f443
a3c3d4b717467c593e, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 50391, gas_limit: 30000000, gas_used: 50501, timestamp: 1686890129, mix_hash: 0xfbece8ca943c50771abb487d7c61e3e3a364061571b6a3d7d29cf79b2929a776, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, pa
rent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xc60d935ac4d467f711f3f7a3b3fdc9e8f3646c40b78a9610b9673aa868a1ec4e, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xd2d7b3c81a852418c71cda71d340e1be7f
d529f543304079e1edf801fc1d26f2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae085000000000
00000000000000000000000000000000000000000000000648be64f000000000000000000000000000000000000000000000000000000035e503fe5ffd93b6dcc14289d573e9729ea69177a80432a1b4e1ce12c119fc6040987d80400000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e
5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623908Z","level":"DEBUG","fields":{"message":"no best payload yet to resolve, building empty payload","id":"0x6d5c5904c42aee43"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.623953Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x6d5c5904c42aee43","parent_hash":"0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df","parent_number":50390},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.624198Z","level":"DEBUG","fields":{"message":"sealed built block","sealed_block":"SealedBlock { header: SealedHeader { hash: 0xe1a452f10433d50c9cf87b5b9703507fe5fe910cf8ccea31d9d0b927b86a6015, header: Header { parent_hash: 0x54cb9f9bc7513629cd3120db8dc1480f95e913e105b9b0
0bb58295f7445e47b0, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6960aee4154da8f239346efb2be373544e0ac21b30d04e030d7c47706ef2c381, transactions_root: 0xabf2ed447e69c2f042154d47dd1447bc2f43ba3b28accc
3202f6bd26981b0778, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 50318, gas_limit: 30000000, gas_used: 46913, timestamp: 1686889983, mix_hash: 0x6410069cd9059139dea46be8c2160661f0405bddcb78f65e01963c9f45071127, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, pa
rent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x739d055a5cad32a5be1202cd18f6af319b8076f30fd51f78e5fe81a558211dfb, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x935ddbe773d536ead8b5b923aa565059f1
073338fa52c09d8a51d24dfd26aa59, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae078000000000
00000000000000000000000000000000000000000000000648be5b30000000000000000000000000000000000000000000000000000000372bf4cd08e236d6534fcedef18ec50e526a9341031ba3859abe3ea7861f97951b6b83b4000000000000000000000000000000000000000000000000000000000000000050000000000000000000000005050f69a9786f081509234f1a7f4684b5e
5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.624682Z","level":"DEBUG","fields":{"message":"sealed built block","sealed_block":"SealedBlock { header: SealedHeader { hash: 0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a, header: Header { parent_hash: 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b
033386a6452a9c63df, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xc8a83fbafb96f282f088414a2ace1bed25a70229c0e2ffe885a85654589262f3, transactions_root: 0x978e98959d7ce140b1e31ecd5fe3aabbc025dd3dd4f443
a3c3d4b717467c593e, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 50391, gas_limit: 30000000, gas_used: 50501, timestamp: 1686890129, mix_hash: 0xfbece8ca943c50771abb487d7c61e3e3a364061571b6a3d7d29cf79b2929a776, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, pa
rent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xc60d935ac4d467f711f3f7a3b3fdc9e8f3646c40b78a9610b9673aa868a1ec4e, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xd2d7b3c81a852418c71cda71d340e1be7f
d529f543304079e1edf801fc1d26f2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae085000000000
00000000000000000000000000000000000000000000000648be64f000000000000000000000000000000000000000000000000000000035e503fe5ffd93b6dcc14289d573e9729ea69177a80432a1b4e1ce12c119fc6040987d80400000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e
5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.624733Z","level":"DEBUG","fields":{"message":"resolving fallback payload as best payload","id":"0x6d5c5904c42aee43"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.624746Z","level":"DEBUG","fields":{"message":"resolving better payload"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.625129Z","level":"DEBUG","fields":{"message":"built better payload","value":"0"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.625337Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x4cfbbab5d344b563","parent_hash":"0x196f84a3ddd48e1db08eb1d874a2a14d1a841cf8fcd66eaf92e167b3e856cda4","parent_number":49898},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.625755Z","level":"DEBUG","fields":{"message":"Appending block to canonical chain","head":"0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a","parent":"(50390, 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df)"},"target":"blockchain_tree
"}
{"timestamp":"2024-04-07T16:28:06.626009Z","level":"DEBUG","fields":{"message":"pre-calculating storage roots","len":2},"target":"trie::parallel_state_root"}
{"timestamp":"2024-04-07T16:28:06.626786Z","level":"DEBUG","fields":{"message":"Validated state root","number":50391,"hash":"0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a","elapsed":"820.343µs"},"target":"blockchain_tree::chain"}
{"timestamp":"2024-04-07T16:28:06.626849Z","level":"DEBUG","fields":{"message":"Next hook is not ready","hook":"StaticFile"},"target":"consensus::engine::hooks"}
{"timestamp":"2024-04-07T16:28:06.626921Z","level":"INFO","fields":{"message":"Block added to canonical chain","number":50391,"hash":"0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a","peers":0,"txs":1,"mgas":"0.051","full":"0.2%","base_fee":"0.00gwei","blobs":0,"excess_blobs":0,"elapse
d":"1.130387ms"},"target":"reth_node_core::events::node"}
{"timestamp":"2024-04-07T16:28:06.626983Z","level":"DEBUG","fields":{"message":"building new payload","id":"0xaa90638fdb644a47","parent_hash":"0x25a767dbf44cd602c9a7bc9fc76e3adde2ed01204cb12d5fa54d43195b82cdf3","parent_number":50054},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.627108Z","level":"DEBUG","fields":{"message":"building new payload","id":"0xca5ad958a19ebd8d","parent_hash":"0x92be6bc3569dd876e80342bd4ab51f8309c5defdcc40459798542944fddf021c","parent_number":49981},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.627743Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x4db86a638b83cacf","parent_hash":"0xe6b625b56bf606b04850b003dc3c56e81e0d03ae8b9e0e6f703c642d1bd56e4b","parent_number":49761},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.627797Z","level":"WARN","fields":{"message":"Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator.","hook":"Prune","head_block_hash":"0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a","safe_block_h
ash":"0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a","finalized_block_hash":"0xe1a452f10433d50c9cf87b5b9703507fe5fe910cf8ccea31d9d0b927b86a6015"},"target":"consensus::engine"}
{"timestamp":"2024-04-07T16:28:06.627814Z","level":"DEBUG","fields":{"message":"Next hook is not ready","hook":"StaticFile"},"target":"consensus::engine::hooks"}
{"timestamp":"2024-04-07T16:28:06.627814Z","level":"DEBUG","fields":{"message":"building new payload","id":"0xaae355f449e9bbb5","parent_hash":"0xdcdb4d33278d97b349082b3c7fdfe9801776ce5e725866c0e1ba30ec3dc193a4","parent_number":49899},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.629081Z","level":"DEBUG","fields":{"message":"Segment pruning started","segment":"Transactions","purpose":"StaticFile","to_block":"50318","prune_mode":"Before(50319)"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.629118Z","level":"DEBUG","fields":{"message":"Segment pruning finished","segment":"Transactions","purpose":"StaticFile","to_block":"50318","prune_mode":"Before(50319)","output.pruned":"0"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.629127Z","level":"DEBUG","fields":{"message":"Segment pruning started","segment":"Headers","purpose":"StaticFile","to_block":"50318","prune_mode":"Before(50319)"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.629135Z","level":"DEBUG","fields":{"message":"Segment pruning finished","segment":"Headers","purpose":"StaticFile","to_block":"50318","prune_mode":"Before(50319)","output.pruned":"0"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.629141Z","level":"DEBUG","fields":{"message":"Segment pruning started","segment":"Receipts","purpose":"StaticFile","to_block":"50318","prune_mode":"Before(50319)"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.629149Z","level":"DEBUG","fields":{"message":"Segment pruning finished","segment":"Receipts","purpose":"StaticFile","to_block":"50318","prune_mode":"Before(50319)","output.pruned":"0"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.629757Z","level":"DEBUG","fields":{"message":"no best payload yet to resolve, building empty payload","id":"0x6d5c5904c42aee43"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.629794Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x6d5c5904c42aee43","parent_hash":"0xe46398ae41b327d57a79b63662a124dde795438f0ae12b033386a6452a9c63df","parent_number":50390},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.630445Z","level":"DEBUG","fields":{"message":"sealed built block","sealed_block":"SealedBlock { header: SealedHeader { hash: 0xd104c14a115a64e368c265dd080d6243896dca23ec055ed69d5023198fd3c62a, header: Header { parent_hash: 0xe46398ae41b327d57a79b63662a124dde795438f0ae12b
033386a6452a9c63df, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xc8a83fbafb96f282f088414a2ace1bed25a70229c0e2ffe885a85654589262f3, transactions_root: 0x978e98959d7ce140b1e31ecd5fe3aabbc025dd3dd4f443
a3c3d4b717467c593e, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 50391, gas_limit: 30000000, gas_used: 50501, timestamp: 1686890129, mix_hash: 0xfbece8ca943c50771abb487d7c61e3e3a364061571b6a3d7d29cf79b2929a776, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, pa
rent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xc60d935ac4d467f711f3f7a3b3fdc9e8f3646c40b78a9610b9673aa868a1ec4e, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xd2d7b3c81a852418c71cda71d340e1be7f
d529f543304079e1edf801fc1d26f2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae085000000000
00000000000000000000000000000000000000000000000648be64f000000000000000000000000000000000000000000000000000000035e503fe5ffd93b6dcc14289d573e9729ea69177a80432a1b4e1ce12c119fc6040987d80400000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e
5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.630487Z","level":"DEBUG","fields":{"message":"resolving fallback payload as best payload","id":"0x6d5c5904c42aee43"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.630511Z","level":"DEBUG","fields":{"message":"resolving best payload"},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.630979Z","level":"DEBUG","fields":{"message":"building new payload","id":"0x2d1eb8a769325f21","parent_hash":"0xbd225b0d00bb0ed99b766330dd206624809c3f8ac993611ee2dc3c532d906346","parent_number":49687},"target":"payload_builder"}
{"timestamp":"2024-04-07T16:28:06.632130Z","level":"DEBUG","fields":{"message":"Commit","total_duration":"2.972825ms","commit_latency":"Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__
bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } }))","is_re
ad_only":false},"target":"storage::db::mdbx"}
{"timestamp":"2024-04-07T16:28:06.632159Z","level":"DEBUG","fields":{"message":"Pruner finished","tip_block_number":"50390","elapsed":"13.172668ms","deleted_entries":"0","limiter":"PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 8500, deleted: 0 }), time_limit: Some(PruneTime
Limit { limit: 100ms, start: Instant { tv_sec: 1111193, tv_nsec: 476061308 } }) }","progress":"Finished","stats":"{}"},"target":"pruner"}
{"timestamp":"2024-04-07T16:28:06.632224Z","level":"INFO","fields":{"message":"Pruner finished","tip_block_number":50390,"elapsed":"13.172668ms","stats":"{}"},"target":"reth_node_core::events::node"}
{"timestamp":"2024-04-07T16:28:06.632678Z","level":"DEBUG","fields":{"message":"Polled running hook with db write access","hook":"Prune","result":"PolledHook { name: \"Prune\", event: Finished(Ok(())), db_access_level: ReadWrite }"},"target":"consensus::engine::hooks"}
zilayo commented 7 months ago

related https://github.com/paradigmxyz/reth/issues/6796 https://github.com/paradigmxyz/reth/pull/6792#issuecomment-1964289389

slordua commented 7 months ago

related #6796 #6792 (comment)

yep.. subbing this issue you guys' is more thorough than mine

dallonby commented 7 months ago

Alexey requested further logs. On this run, op-node crashed at 18:24:23.701Z with...

INFO [04-07|18:24:23.693] Found next batch                         batch_type=SingularBatch batch_timestamp=1,690,594,543 parent_hash=872d2d..ecda21 batch_epoch=ced210..937d43:17795429 txs=0
INFO [04-07|18:24:23.694] generated attributes in payload queue    txs=1   timestamp=1,690,594,543
ERROR[04-07|18:24:23.700] cancelling old block sealing job         payload=0x6fe61e7beeb51d1d
WARN [04-07|18:24:23.701] could not process payload derived from L1 data, dropping batch err="failed to complete building on top of L2 chain 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21:1902597, id: 0x6fe61e7beeb51d1d, error (3):
updated forkchoice, but node is syncing"
ERROR[04-07|18:24:23.701] deposit only block was invalid           parent=872d2d..ecda21:1902597 err="failed to complete building on top of L2 chain 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21:1902597, id: 0x6fe61e7beeb51d1d, err
or (3): updated forkchoice, but node is syncing"
ERROR[04-07|18:24:23.701] Derivation process critical error        err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d
835fa3b94decda21:1902597, id: 0x6fe61e7beeb51d1d, error (3): updated forkchoice, but node is syncing"
INFO [04-07|18:24:23.701] State loop returned
INFO [04-07|18:24:25.733] Received signed execution payload from p2p id=7394ab..00a27a:12862459 peer=16Uiu2HAkwdGmzLtcAvX1aXCeWD2qD1CMsA5N8PaBHK3Xj4dFjhdJ
INFO [04-07|18:24:27.432] Received signed execution payload from p2p id=a61242..b87f94:12862460 peer=16Uiu2HAmVapL5XMp9hLSCi8aT3aMooVRCq6v3ZPKZEXQc8z7Wsac

op-reth logs from this period:

2024-04-07T18:24:22.642989Z DEBUG payload_builder: building new payload id=0x28605e35487c6759 parent_hash=0xd448e882b0c7f030971a241f697552d1bdc1931e2b2e4c8cf1eb8f1797524496 parent_number=1902567
2024-04-07T18:24:22.647424Z DEBUG payload_builder: building new payload id=0x3c7cdd94c2f0339f parent_hash=0xa5db8010f311e6cd19090ffaf721e8972739eda5f479688949b10da57ea5874f parent_number=1902568
2024-04-07T18:24:22.650832Z DEBUG payload_builder: building new payload id=0x7376ad8deb1c2966 parent_hash=0xd29d941b3e9bd1550035af681f2de21ff0706f947fea1d613d27142c12d82a5e parent_number=1902569
2024-04-07T18:24:22.662223Z DEBUG payload_builder: building new payload id=0x6139c2b26265a52e parent_hash=0x5c102bd0d50f598929c49cdd98ddd2ba883398e90189e4ee6c3d509b9373a4ba parent_number=1902570
2024-04-07T18:24:22.664629Z DEBUG payload_builder: building new payload id=0x0c403d1d8413d311 parent_hash=0x3d0b87144f235116cdbb59a9376b6cf34ff412d66476bd89204826a303678c17 parent_number=1902571
2024-04-07T18:24:22.700327Z DEBUG payload_builder: building new payload id=0x98a97e1c27afba46 parent_hash=0x9c52c4e39d3667e75120e7fac33f3e3b9ebfd1b2f9faa4cad4a844d795ad8ae8 parent_number=1902572
2024-04-07T18:24:22.707847Z DEBUG payload_builder: building new payload id=0x2ccd19df244c5d34 parent_hash=0x3fc268ab7896e865eb21e189658d298b39ce921466caea7e1146303094f8a844 parent_number=1902573
2024-04-07T18:24:22.716277Z DEBUG payload_builder: building new payload id=0x46b7d300ffeea3ce parent_hash=0x192af96983f950034f1621402118d770d3336cea84617dde3b68c80274c480c1 parent_number=1902574
2024-04-07T18:24:22.725721Z DEBUG payload_builder: building new payload id=0xb213796cd2d86427 parent_hash=0xd9054bcc9fc06efe4e69dfebf5304feeb2a1e995517d69b1a7630dca31f5e643 parent_number=1902575
2024-04-07T18:24:22.736248Z DEBUG payload_builder: building new payload id=0xd5f8214c429b85c2 parent_hash=0xeaa906d9a98a0cc355763fa989a9093c1236c871414927630c08945e6a83991c parent_number=1902576
2024-04-07T18:24:22.871932Z DEBUG payload_builder: building new payload id=0xb1b8ef2f1b11e0b7 parent_hash=0x7ed10333a5fe6ed25ac20b2132254bc53fb14a4d4da6c11fc203e93c69356f68 parent_number=1902577
2024-04-07T18:24:22.877444Z DEBUG payload_builder: building new payload id=0x69b2f279d4e69a17 parent_hash=0xdfa4f2518a5f8ab1adfcb424e33bdfa7c02fc07a82711bc2e8ba660723b139f3 parent_number=1902578
2024-04-07T18:24:22.885916Z DEBUG payload_builder: building new payload id=0x7cfc0cff0cab1c7e parent_hash=0xb6acec19b52eff94a13b1bdd9a82b066088c534c4caf1ac11d276bb483279500 parent_number=1902579
2024-04-07T18:24:22.898374Z DEBUG payload_builder: building new payload id=0xc3416a9aece08284 parent_hash=0xe6f7b6609f9ca6fe4735cd787a9f69e3d16f1208334a4d1f1d652b069fc19f1b parent_number=1902580
2024-04-07T18:24:22.903813Z DEBUG payload_builder: building new payload id=0xee7cc41ec6e68d34 parent_hash=0x56205d5bcb7a9a5c59a08da8531c4711c74d5331bac135b6ed7517c0a3d83f03 parent_number=1902581
2024-04-07T18:24:23.018245Z DEBUG payload_builder: building new payload id=0x4c27eb7f395ec725 parent_hash=0x37509b9cdc618b227d1c392179df4ca6f03416168cafdc6af45aa7f1c7981da9 parent_number=1902582
2024-04-07T18:24:23.031554Z DEBUG payload_builder: building new payload id=0xe30f3dfa943a7fe5 parent_hash=0xa5d40df5b3861a038258cade09a9770e6046e5ae09c9d91edb0e70d75d318824 parent_number=1902583
2024-04-07T18:24:23.036884Z DEBUG payload_builder: building new payload id=0xacb236c1b6b64177 parent_hash=0x088e70922178d731db03150f07394e83b6a38c2034d7068c87e3b7de0342d809 parent_number=1902584
2024-04-07T18:24:23.053180Z DEBUG payload_builder: building new payload id=0x88721efec332ee5b parent_hash=0xd025c56fac86df927da18b7890d536350cc910bfa7d6b5d49baf53523771c7a0 parent_number=1902585
2024-04-07T18:24:23.059466Z DEBUG payload_builder: building new payload id=0x75b2b50d964b9096 parent_hash=0x4526ece61869809199cd95bb6141abf2261f0f3415ff21a10318f78e076bc3c9 parent_number=1902586
2024-04-07T18:24:23.127856Z DEBUG payload_builder: building new payload id=0xfb380cc4c48a990c parent_hash=0x3731cce76cad54e487e6bc797abd251da6eee5a647768ca506fa6a6778dc8ae6 parent_number=1902587
2024-04-07T18:24:23.144547Z DEBUG payload_builder: building new payload id=0x1b62fca6c83aa91c parent_hash=0x920881b2f59d0b7b53868052e3009dbdc0803e99c6cba0b4a81497fe302ac1a4 parent_number=1902588
2024-04-07T18:24:23.157143Z DEBUG payload_builder: building new payload id=0x75e77f9330ea556d parent_hash=0x28e2abb5b1f41f16b631e1ddde91db621c34620f2645a182cf5419f61d7f9eee parent_number=1902589
2024-04-07T18:24:23.169778Z DEBUG payload_builder: building new payload id=0x7745f908471a2142 parent_hash=0xf397b1c31df2cb1f7b204a26d279e3e9c49d29cd2c9735b75af6f21dc3155d55 parent_number=1902590
2024-04-07T18:24:23.181453Z DEBUG payload_builder: building new payload id=0x1e3df767d5d3c53c parent_hash=0xa045515191f274f671b88e36c9386a5dd5facdc786ff311d15b1a4066619f243 parent_number=1902591
2024-04-07T18:24:23.350599Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680
2024-04-07T18:24:23.350623Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902592 current_head_num=1902592
2024-04-07T18:24:23.350660Z  INFO reth_node_core::events::node: Canonical chain committed number=1902592 hash=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680 elapsed=86.042µs
2024-04-07T18:24:23.350680Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.350756Z  INFO reth_payload_builder::service: New payload job created id=0x5c43970a27c2df69 parent=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680
2024-04-07T18:24:23.351005Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x5c43970a27c2df69
2024-04-07T18:24:23.351031Z DEBUG payload_builder: building new payload id=0x5c43970a27c2df69 parent_hash=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680 parent_number=1902592
2024-04-07T18:24:23.351594Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee, header: Header { parent_hash: 0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x82445a2fb070f15d9fb8bb4b7eff2b417c5a4b3674c52779c54e9908237055e3, transactions_root: 0x59ceba73b602149c51d663335d7b1ebd2ec2a434c52df9743956f284157b81b7, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902593, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594533, mix_hash: 0xd2acd772a7a1ff457d3b76442f31be63c13214f42f9dba4f0f5ddf7256d557c1, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xe91680c6894c36cc3a12890c696c3596a05fb574306bf264d3042e71d71c05ab, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x4339dcdd95918b8b45104f42c1ebc6005027fb2c3557f66f2100d5407241c028, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89640000000000000000000000000000000000000000000000000000000064c46ba300000000000000000000000000000000000000000000000000000003c9bcc74afc5974bccdbe9a708b2d2c9fade2eff3c55e7b88f107ae4a0898630a36cb422400000000000000000000000000000000000000000000000000000000000000050000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.351617Z DEBUG payload_builder: resolving fallback payload as best payload id=0x5c43970a27c2df69
2024-04-07T18:24:23.351624Z DEBUG payload_builder: resolving best payload
2024-04-07T18:24:23.351868Z DEBUG payload_builder: building new payload id=0x5c43970a27c2df69 parent_hash=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680 parent_number=1902592
2024-04-07T18:24:23.351939Z DEBUG blockchain_tree: Appending block to canonical chain head=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee parent=(1902592, 0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680)
2024-04-07T18:24:23.352048Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T18:24:23.352463Z DEBUG blockchain_tree::chain: Validated state root number=1902593 hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee elapsed=434.802µs
2024-04-07T18:24:23.352490Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.352521Z  INFO reth_node_core::events::node: Block added to canonical chain number=1902593 hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee peers=0 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=576.944µs
2024-04-07T18:24:23.352662Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee, header: Header { parent_hash: 0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x82445a2fb070f15d9fb8bb4b7eff2b417c5a4b3674c52779c54e9908237055e3, transactions_root: 0x59ceba73b602149c51d663335d7b1ebd2ec2a434c52df9743956f284157b81b7, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902593, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594533, mix_hash: 0xd2acd772a7a1ff457d3b76442f31be63c13214f42f9dba4f0f5ddf7256d557c1, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xe91680c6894c36cc3a12890c696c3596a05fb574306bf264d3042e71d71c05ab, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x4339dcdd95918b8b45104f42c1ebc6005027fb2c3557f66f2100d5407241c028, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89640000000000000000000000000000000000000000000000000000000064c46ba300000000000000000000000000000000000000000000000000000003c9bcc74afc5974bccdbe9a708b2d2c9fade2eff3c55e7b88f107ae4a0898630a36cb422400000000000000000000000000000000000000000000000000000000000000050000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.352704Z DEBUG payload_builder: built better payload value=0
2024-04-07T18:24:23.352744Z DEBUG blockchain_tree: Committing new canonical chain: [(1902593, 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee)]
2024-04-07T18:24:23.352757Z DEBUG blockchain_tree: Using cached trie updates blocks=[(1902593, 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee)]
2024-04-07T18:24:23.352905Z DEBUG providers::db: Inserted block block_number=1902593 actions=[(InsertCanonicalHeaders, 20.163µs), (InsertHeaders, 14.388µs), (InsertHeaderNumbers, 7.508µs), (GetParentTD, 698ns), (InsertHeaderTerminalDifficulties, 5.814µs), (GetNextTxNum, 3.605µs), (InsertTransactionSenders, 11.126µs), (InsertTransactions, 30.509µs), (InsertTransactionHashNumbers, 17.575µs), (InsertBlockBodyIndices, 10.714µs), (InsertTransactionBlocks, 4.002µs)]
2024-04-07T18:24:23.353069Z DEBUG providers::db: Appended blocks range=1902593..=1902593 actions=[(InsertBlock, 137.102µs), (InsertState, 51.333µs), (InsertHashes, 49.981µs), (InsertHistoryIndices, 45.528µs), (UpdatePipelineStages, 10.29µs)]
2024-04-07T18:24:23.353388Z DEBUG storage::db::mdbx: Commit total_duration=312.916µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 1, gc_wallclock: 1, audit: 0, write: 0, sync: 16, ending: 0, whole: 19, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T18:24:23.353401Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 7.147µs), (FindCanonicalHeader, 7.24µs), (SplitChain, 1.221µs), (SplitChainForks, 906ns), (MergeAllChains, 696ns), (UpdateCanonicalIndex, 2.837µs), (RetrieveStateTrieUpdates, 17.026µs), (CommitCanonicalChainToDatabase, 636.017µs)]
2024-04-07T18:24:23.353411Z DEBUG consensus::engine: Canonicalized new head hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee number=1902593
2024-04-07T18:24:23.353453Z  INFO reth_node_core::events::node: Canonical chain committed number=1902593 hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee elapsed=687.793µs
2024-04-07T18:24:23.353459Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.354191Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee
2024-04-07T18:24:23.354219Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902593 current_head_num=1902593
2024-04-07T18:24:23.354240Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.354247Z  INFO reth_node_core::events::node: Canonical chain committed number=1902593 hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee elapsed=47.059µs
2024-04-07T18:24:23.354292Z  INFO reth_payload_builder::service: New payload job created id=0x2615eab975a12f4a parent=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee
2024-04-07T18:24:23.354447Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x2615eab975a12f4a
2024-04-07T18:24:23.354465Z DEBUG payload_builder: building new payload id=0x2615eab975a12f4a parent_hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee parent_number=1902593
2024-04-07T18:24:23.354976Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9, header: Header { parent_hash: 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x4a57069e2eb43cfbd58f858d0a067becaea15550a88092b641d94136a7533d1e, transactions_root: 0xd7f51ddd1979ac75a156e8d6c73ec2e811a3bf2fc10ab1bed93ecb9406d0c81b, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902594, gas_limit: 30000000, gas_used: 50501, timestamp: 1690594535, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xe5db581db4c0dd9ba74ac9cf46a31efc8aa8bbc059b2724c41a7bd126f98204d, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x6b6f1f43376bdd15ee393940ac753639bf99606042f809202f99fe4c0a3658f8, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.355003Z DEBUG payload_builder: resolving fallback payload as best payload id=0x2615eab975a12f4a
2024-04-07T18:24:23.355009Z DEBUG payload_builder: resolving best payload
2024-04-07T18:24:23.355306Z DEBUG payload_builder: building new payload id=0x2615eab975a12f4a parent_hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee parent_number=1902593
2024-04-07T18:24:23.355352Z DEBUG blockchain_tree: Appending block to canonical chain head=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 parent=(1902593, 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee)
2024-04-07T18:24:23.355470Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T18:24:23.355831Z DEBUG blockchain_tree::chain: Validated state root number=1902594 hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 elapsed=377.346µs
2024-04-07T18:24:23.355851Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.355883Z  INFO reth_node_core::events::node: Block added to canonical chain number=1902594 hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 peers=0 txs=1 mgas=0.051 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=527.802µs
2024-04-07T18:24:23.355877Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9, header: Header { parent_hash: 0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x4a57069e2eb43cfbd58f858d0a067becaea15550a88092b641d94136a7533d1e, transactions_root: 0xd7f51ddd1979ac75a156e8d6c73ec2e811a3bf2fc10ab1bed93ecb9406d0c81b, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902594, gas_limit: 30000000, gas_used: 50501, timestamp: 1690594535, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xe5db581db4c0dd9ba74ac9cf46a31efc8aa8bbc059b2724c41a7bd126f98204d, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x6b6f1f43376bdd15ee393940ac753639bf99606042f809202f99fe4c0a3658f8, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.355908Z DEBUG payload_builder: built better payload value=0
2024-04-07T18:24:23.356099Z DEBUG blockchain_tree: Committing new canonical chain: [(1902594, 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9)]
2024-04-07T18:24:23.356112Z DEBUG blockchain_tree: Using cached trie updates blocks=[(1902594, 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9)]
2024-04-07T18:24:23.356228Z DEBUG providers::db: Inserted block block_number=1902594 actions=[(InsertCanonicalHeaders, 18.448µs), (InsertHeaders, 8.07µs), (InsertHeaderNumbers, 7.183µs), (GetParentTD, 761ns), (InsertHeaderTerminalDifficulties, 6.95µs), (GetNextTxNum, 3.375µs), (InsertTransactionSenders, 4.973µs), (InsertTransactions, 20.823µs), (InsertTransactionHashNumbers, 16.245µs), (InsertBlockBodyIndices, 5.348µs), (InsertTransactionBlocks, 6.159µs)]
2024-04-07T18:24:23.356558Z DEBUG providers::db: Appended blocks range=1902594..=1902594 actions=[(InsertBlock, 106.118µs), (InsertState, 52.751µs), (InsertHashes, 60.588µs), (InsertHistoryIndices, 200.344µs), (UpdatePipelineStages, 9.544µs)]
2024-04-07T18:24:23.356860Z DEBUG storage::db::mdbx: Commit total_duration=297.02µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 1, audit: 0, write: 0, sync: 17, ending: 0, whole: 18, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T18:24:23.356871Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 5.635µs), (FindCanonicalHeader, 5.362µs), (SplitChain, 2.761µs), (SplitChainForks, 624ns), (MergeAllChains, 814ns), (UpdateCanonicalIndex, 2.777µs), (RetrieveStateTrieUpdates, 15.946µs), (CommitCanonicalChainToDatabase, 752.816µs)]
2024-04-07T18:24:23.356879Z DEBUG consensus::engine: Canonicalized new head hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 number=1902594
2024-04-07T18:24:23.356914Z  INFO reth_node_core::events::node: Canonical chain committed number=1902594 hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 elapsed=798.076µs
2024-04-07T18:24:23.356928Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.357587Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9
2024-04-07T18:24:23.357599Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902594 current_head_num=1902594
2024-04-07T18:24:23.357618Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.357620Z  INFO reth_node_core::events::node: Canonical chain committed number=1902594 hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 elapsed=30.089µs
2024-04-07T18:24:23.357680Z  INFO reth_payload_builder::service: New payload job created id=0xa7a9618ec35cbbd7 parent=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9
2024-04-07T18:24:23.357852Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0xa7a9618ec35cbbd7
2024-04-07T18:24:23.357876Z DEBUG payload_builder: building new payload id=0xa7a9618ec35cbbd7 parent_hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 parent_number=1902594
2024-04-07T18:24:23.358481Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe, header: Header { parent_hash: 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xd2a6ae7ffb98aa71ff4a1c9a2a19adbb575b418b52218285ccaae090ea826b74, transactions_root: 0x5ae19c3c67c91a47d2de7208bea997ffb0a5255dc7156424c80540769cf8bbbe, receipts_root: 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902595, gas_limit: 30000000, gas_used: 64013, timestamp: 1690594537, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x1625b75f21a5a488f86be6b83012c644523a0617a28861881b1b716724abb8db, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xfa3953827988df5ba80b782f57d66688abd2e50ff65565d9ee5efef34a48d2b3, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000010000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.358505Z DEBUG payload_builder: resolving fallback payload as best payload id=0xa7a9618ec35cbbd7
2024-04-07T18:24:23.358514Z DEBUG payload_builder: resolving best payload
2024-04-07T18:24:23.358725Z DEBUG payload_builder: building new payload id=0xa7a9618ec35cbbd7 parent_hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 parent_number=1902594
2024-04-07T18:24:23.358792Z DEBUG blockchain_tree: Appending block to canonical chain head=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe parent=(1902594, 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9)
2024-04-07T18:24:23.358886Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T18:24:23.359092Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe, header: Header { parent_hash: 0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xd2a6ae7ffb98aa71ff4a1c9a2a19adbb575b418b52218285ccaae090ea826b74, transactions_root: 0x5ae19c3c67c91a47d2de7208bea997ffb0a5255dc7156424c80540769cf8bbbe, receipts_root: 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902595, gas_limit: 30000000, gas_used: 64013, timestamp: 1690594537, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x1625b75f21a5a488f86be6b83012c644523a0617a28861881b1b716724abb8db, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xfa3953827988df5ba80b782f57d66688abd2e50ff65565d9ee5efef34a48d2b3, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000010000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.359116Z DEBUG payload_builder: built better payload value=0
2024-04-07T18:24:23.359174Z DEBUG blockchain_tree::chain: Validated state root number=1902595 hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe elapsed=302.811µs
2024-04-07T18:24:23.359192Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.359221Z  INFO reth_node_core::events::node: Block added to canonical chain number=1902595 hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe peers=0 txs=1 mgas=0.064 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=420.979µs
2024-04-07T18:24:23.359378Z DEBUG blockchain_tree: Committing new canonical chain: [(1902595, 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe)]
2024-04-07T18:24:23.359393Z DEBUG blockchain_tree: Using cached trie updates blocks=[(1902595, 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe)]
2024-04-07T18:24:23.359497Z DEBUG providers::db: Inserted block block_number=1902595 actions=[(InsertCanonicalHeaders, 12.747µs), (InsertHeaders, 9.344µs), (InsertHeaderNumbers, 10.383µs), (GetParentTD, 576ns), (InsertHeaderTerminalDifficulties, 4.675µs), (GetNextTxNum, 2.715µs), (InsertTransactionSenders, 4.544µs), (InsertTransactions, 21.013µs), (InsertTransactionHashNumbers, 11.81µs), (InsertBlockBodyIndices, 5.27µs), (InsertTransactionBlocks, 4.035µs)]
2024-04-07T18:24:23.359642Z DEBUG providers::db: Appended blocks range=1902595..=1902595 actions=[(InsertBlock, 95.588µs), (InsertState, 45.582µs), (InsertHashes, 43.564µs), (InsertHistoryIndices, 42.332µs), (UpdatePipelineStages, 7.526µs)]
2024-04-07T18:24:23.359850Z DEBUG storage::db::mdbx: Commit total_duration=204.381µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 11, ending: 0, whole: 12, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T18:24:23.359860Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 4.068µs), (FindCanonicalHeader, 3.711µs), (SplitChain, 1.291µs), (SplitChainForks, 627ns), (MergeAllChains, 570ns), (UpdateCanonicalIndex, 2.311µs), (RetrieveStateTrieUpdates, 18.492µs), (CommitCanonicalChainToDatabase, 460.717µs)]
2024-04-07T18:24:23.359867Z DEBUG consensus::engine: Canonicalized new head hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe number=1902595
2024-04-07T18:24:23.359900Z  INFO reth_node_core::events::node: Canonical chain committed number=1902595 hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe elapsed=502.244µs
2024-04-07T18:24:23.359901Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.421844Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe
2024-04-07T18:24:23.421875Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902595 current_head_num=1902595
2024-04-07T18:24:23.421934Z  INFO reth_node_core::events::node: Canonical chain committed number=1902595 hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe elapsed=108.966µs
2024-04-07T18:24:23.422031Z DEBUG consensus::engine::hooks: Polled next hook hook="StaticFile" result=PolledHook { name: "StaticFile", event: Started, db_access_level: ReadOnly }
2024-04-07T18:24:23.422042Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.422111Z DEBUG static_file: StaticFileProducer started targets=StaticFileTargets { headers: Some(1902513..=1902595), receipts: Some(1902513..=1902595), transactions: Some(1902513..=1902595) }
2024-04-07T18:24:23.422139Z  INFO reth_node_core::events::node: Static File Producer started targets=StaticFileTargets { headers: Some(1902513..=1902595), receipts: Some(1902513..=1902595), transactions: Some(1902513..=1902595) }
2024-04-07T18:24:23.422176Z DEBUG static_file: StaticFileProducer segment segment=Transactions block_range=1902513..=1902595
2024-04-07T18:24:23.422185Z DEBUG static_file: StaticFileProducer segment segment=Headers block_range=1902513..=1902595
2024-04-07T18:24:23.422210Z DEBUG static_file: StaticFileProducer segment segment=Receipts block_range=1902513..=1902595
2024-04-07T18:24:23.422446Z DEBUG static_file: Finished StaticFileProducer segment segment=Headers block_range=1902513..=1902595 elapsed=249.844µs
2024-04-07T18:24:23.423576Z DEBUG static_file: Finished StaticFileProducer segment segment=Transactions block_range=1902513..=1902595 elapsed=1.383711ms
2024-04-07T18:24:23.423849Z DEBUG static_file: Finished StaticFileProducer segment segment=Receipts block_range=1902513..=1902595 elapsed=1.613738ms
2024-04-07T18:24:23.424693Z DEBUG provider::static_file: Commit segment=Headers path="/data/op-reth/static_files/static_file_headers_1500000_1999999" duration=798.188µs
2024-04-07T18:24:23.425509Z DEBUG provider::static_file: Commit segment=Transactions path="/data/op-reth/static_files/static_file_transactions_1500000_1999999" duration=559.891µs
2024-04-07T18:24:23.426151Z DEBUG provider::static_file: Commit segment=Receipts path="/data/op-reth/static_files/static_file_receipts_1500000_1999999" duration=432.386µs
2024-04-07T18:24:23.426402Z DEBUG static_file: StaticFileProducer finished targets=StaticFileTargets { headers: Some(1902513..=1902595), receipts: Some(1902513..=1902595), transactions: Some(1902513..=1902595) } elapsed=4.273458ms
2024-04-07T18:24:23.426494Z  INFO reth_node_core::events::node: Static File Producer finished targets=StaticFileTargets { headers: Some(1902513..=1902595), receipts: Some(1902513..=1902595), transactions: Some(1902513..=1902595) } elapsed=4.273458ms
2024-04-07T18:24:23.644576Z DEBUG payload_builder: building new payload id=0x28605e35487c6759 parent_hash=0xd448e882b0c7f030971a241f697552d1bdc1931e2b2e4c8cf1eb8f1797524496 parent_number=1902567
2024-04-07T18:24:23.649023Z DEBUG payload_builder: building new payload id=0x3c7cdd94c2f0339f parent_hash=0xa5db8010f311e6cd19090ffaf721e8972739eda5f479688949b10da57ea5874f parent_number=1902568
2024-04-07T18:24:23.652457Z DEBUG payload_builder: building new payload id=0x7376ad8deb1c2966 parent_hash=0xd29d941b3e9bd1550035af681f2de21ff0706f947fea1d613d27142c12d82a5e parent_number=1902569
2024-04-07T18:24:23.662848Z DEBUG payload_builder: building new payload id=0x6139c2b26265a52e parent_hash=0x5c102bd0d50f598929c49cdd98ddd2ba883398e90189e4ee6c3d509b9373a4ba parent_number=1902570
2024-04-07T18:24:23.666136Z DEBUG payload_builder: building new payload id=0x0c403d1d8413d311 parent_hash=0x3d0b87144f235116cdbb59a9376b6cf34ff412d66476bd89204826a303678c17 parent_number=1902571
2024-04-07T18:24:23.673632Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe
2024-04-07T18:24:23.673707Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902595 current_head_num=1902595
2024-04-07T18:24:23.673773Z  INFO reth_node_core::events::node: Canonical chain committed number=1902595 hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe elapsed=192.213µs
2024-04-07T18:24:23.673806Z DEBUG consensus::engine::hooks: Polled next hook hook="StaticFile" result=PolledHook { name: "StaticFile", event: Finished(Ok(())), db_access_level: ReadOnly }
2024-04-07T18:24:23.673826Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.673984Z  INFO reth_payload_builder::service: New payload job created id=0xfee5be46ae7df940 parent=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe
2024-04-07T18:24:23.674513Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0xfee5be46ae7df940
2024-04-07T18:24:23.674557Z DEBUG payload_builder: building new payload id=0xfee5be46ae7df940 parent_hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe parent_number=1902595
2024-04-07T18:24:23.675751Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81, header: Header { parent_hash: 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x7128899ca049180f0902da70fce843ce9bce280406f2de7cf6bafb0d0f5f585e, transactions_root: 0xb04f9ffc022dbe03407fea40dd2b28f040450875faed38422fd1c094e7417bc1, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902596, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594539, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xfba89bbe89c4b4384fd63715758ff32a4ed453265fbe3b0d2c625d214c6105a3, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x7b68c6697f13ba37012ff3dd523fd2d694bca375fa32ecea24b001b78140cb1f, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000020000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.675794Z DEBUG payload_builder: resolving fallback payload as best payload id=0xfee5be46ae7df940
2024-04-07T18:24:23.675815Z DEBUG payload_builder: resolving best payload
2024-04-07T18:24:23.675926Z DEBUG payload_builder: building new payload id=0xfee5be46ae7df940 parent_hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe parent_number=1902595
2024-04-07T18:24:23.676706Z DEBUG blockchain_tree: Appending block to canonical chain head=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 parent=(1902595, 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe)
2024-04-07T18:24:23.677112Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T18:24:23.677877Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81, header: Header { parent_hash: 0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x7128899ca049180f0902da70fce843ce9bce280406f2de7cf6bafb0d0f5f585e, transactions_root: 0xb04f9ffc022dbe03407fea40dd2b28f040450875faed38422fd1c094e7417bc1, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902596, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594539, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xfba89bbe89c4b4384fd63715758ff32a4ed453265fbe3b0d2c625d214c6105a3, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x7b68c6697f13ba37012ff3dd523fd2d694bca375fa32ecea24b001b78140cb1f, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000020000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.678025Z DEBUG payload_builder: built better payload value=0
2024-04-07T18:24:23.678277Z DEBUG blockchain_tree::chain: Validated state root number=1902596 hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 elapsed=1.234657ms
2024-04-07T18:24:23.678329Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.678455Z  INFO reth_node_core::events::node: Block added to canonical chain number=1902596 hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 peers=0 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=1.694137ms
2024-04-07T18:24:23.678974Z DEBUG blockchain_tree: Committing new canonical chain: [(1902596, 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81)]
2024-04-07T18:24:23.679022Z DEBUG blockchain_tree: Using cached trie updates blocks=[(1902596, 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81)]
2024-04-07T18:24:23.679539Z DEBUG providers::db: Inserted block block_number=1902596 actions=[(InsertCanonicalHeaders, 53.015µs), (InsertHeaders, 61.258µs), (InsertHeaderNumbers, 30.112µs), (GetParentTD, 1.75µs), (InsertHeaderTerminalDifficulties, 17.665µs), (GetNextTxNum, 9.475µs), (InsertTransactionSenders, 15.858µs), (InsertTransactions, 59.547µs), (InsertTransactionHashNumbers, 36.698µs), (InsertBlockBodyIndices, 16.652µs), (InsertTransactionBlocks, 21.354µs)]
2024-04-07T18:24:23.680122Z DEBUG providers::db: Appended blocks range=1902596..=1902596 actions=[(InsertBlock, 358.763µs), (InsertState, 163.483µs), (InsertHashes, 187.209µs), (InsertHistoryIndices, 165.877µs), (UpdatePipelineStages, 38.596µs)]
2024-04-07T18:24:23.681220Z DEBUG storage::db::mdbx: Commit total_duration=1.078982ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 1, gc_wallclock: 3, audit: 0, write: 0, sync: 59, ending: 0, whole: 64, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T18:24:23.681275Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 18.09µs), (FindCanonicalHeader, 35.535µs), (SplitChain, 2.889µs), (SplitChainForks, 5.782µs), (MergeAllChains, 2.121µs), (UpdateCanonicalIndex, 8.548µs), (RetrieveStateTrieUpdates, 58.849µs), (CommitCanonicalChainToDatabase, 2.234563ms)]
2024-04-07T18:24:23.681302Z DEBUG consensus::engine: Canonicalized new head hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 number=1902596
2024-04-07T18:24:23.681417Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.681463Z DEBUG txpool: cleaning up blob store finalized_block=1902595
2024-04-07T18:24:23.681478Z DEBUG txpool::blob: Removing blobs from disk num_blobs=0
2024-04-07T18:24:23.681516Z  INFO reth_node_core::events::node: Canonical chain committed number=1902596 hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 elapsed=2.404681ms
2024-04-07T18:24:23.683039Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81
2024-04-07T18:24:23.683088Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902596 current_head_num=1902596
2024-04-07T18:24:23.683152Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.683218Z  INFO reth_node_core::events::node: Canonical chain committed number=1902596 hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 elapsed=121.337µs
2024-04-07T18:24:23.683230Z  INFO reth_payload_builder::service: New payload job created id=0x8e570a56f6904988 parent=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81
2024-04-07T18:24:23.683770Z DEBUG payload_builder: building new payload id=0x8e570a56f6904988 parent_hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 parent_number=1902596
2024-04-07T18:24:23.683812Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x8e570a56f6904988
2024-04-07T18:24:23.683872Z DEBUG payload_builder: building new payload id=0x8e570a56f6904988 parent_hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 parent_number=1902596
2024-04-07T18:24:23.684896Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21, header: Header { parent_hash: 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x2bd926d9b62ce68cd0794c8d9f5838d7d4dfe650574d1798820d1c12dbb1aaa2, transactions_root: 0x1bbace239c3b0427c5ba21c0bc3f0c08741b5c00b1c3ba57598e9956317a66e0, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902597, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594541, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xde3de060516a5de0ba7cbd7c5405c10ddfe2bcaefa1f3e9a43ebbf90a5264c65, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x22962f3e11c1f8452f50339a0be89aeb4b3b1d050553bc4f06e572a7ab2ef545, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000030000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.685884Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21, header: Header { parent_hash: 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x2bd926d9b62ce68cd0794c8d9f5838d7d4dfe650574d1798820d1c12dbb1aaa2, transactions_root: 0x1bbace239c3b0427c5ba21c0bc3f0c08741b5c00b1c3ba57598e9956317a66e0, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902597, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594541, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xde3de060516a5de0ba7cbd7c5405c10ddfe2bcaefa1f3e9a43ebbf90a5264c65, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x22962f3e11c1f8452f50339a0be89aeb4b3b1d050553bc4f06e572a7ab2ef545, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000030000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.685965Z DEBUG payload_builder: resolving fallback payload as best payload id=0x8e570a56f6904988
2024-04-07T18:24:23.685998Z DEBUG payload_builder: resolving better payload
2024-04-07T18:24:23.687163Z DEBUG blockchain_tree: Appending block to canonical chain head=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 parent=(1902596, 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81)
2024-04-07T18:24:23.687527Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T18:24:23.689068Z DEBUG blockchain_tree::chain: Validated state root number=1902597 hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 elapsed=1.592822ms
2024-04-07T18:24:23.689147Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-07T18:24:23.689231Z  INFO reth_node_core::events::node: Block added to canonical chain number=1902597 hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 peers=0 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=2.057495ms
2024-04-07T18:24:23.689924Z DEBUG blockchain_tree: Committing new canonical chain: [(1902597, 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21)]
2024-04-07T18:24:23.689997Z DEBUG blockchain_tree: Using cached trie updates blocks=[(1902597, 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21)]
2024-04-07T18:24:23.690458Z DEBUG providers::db: Inserted block block_number=1902597 actions=[(InsertCanonicalHeaders, 61.387µs), (InsertHeaders, 39.21µs), (InsertHeaderNumbers, 36.215µs), (GetParentTD, 1.716µs), (InsertHeaderTerminalDifficulties, 23.481µs), (GetNextTxNum, 11.613µs), (InsertTransactionSenders, 34.616µs), (InsertTransactions, 63.284µs), (InsertTransactionHashNumbers, 43.872µs), (InsertBlockBodyIndices, 24.06µs), (InsertTransactionBlocks, 36.828µs)]
2024-04-07T18:24:23.691433Z DEBUG providers::db: Appended blocks range=1902597..=1902597 actions=[(InsertBlock, 409.293µs), (InsertState, 357.308µs), (InsertHashes, 309.835µs), (InsertHistoryIndices, 238.057µs), (UpdatePipelineStages, 45.657µs)]
2024-04-07T18:24:23.692602Z DEBUG storage::db::mdbx: Commit total_duration=1.149551ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 1, gc_wallclock: 7, audit: 0, write: 0, sync: 62, ending: 0, whole: 70, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T18:24:23.692649Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 14.577µs), (FindCanonicalHeader, 18.876µs), (SplitChain, 2.971µs), (SplitChainForks, 1.899µs), (MergeAllChains, 1.972µs), (UpdateCanonicalIndex, 7.663µs), (RetrieveStateTrieUpdates, 86.667µs), (CommitCanonicalChainToDatabase, 2.628765ms)]
2024-04-07T18:24:23.692685Z DEBUG consensus::engine: Canonicalized new head hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 number=1902597
2024-04-07T18:24:23.692727Z  INFO reth_node_core::events::node: Canonical chain committed number=1902597 hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 elapsed=2.812435ms
2024-04-07T18:24:23.692819Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.694914Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21
2024-04-07T18:24:23.694960Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=1902597 current_head_num=1902597
2024-04-07T18:24:23.694990Z  INFO reth_node_core::events::node: Canonical chain committed number=1902597 hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 elapsed=109.347µs
2024-04-07T18:24:23.695028Z DEBUG pruner: Minimum pruning interval reached previous_tip_block_number=Some(1902592) tip_block_number=1902597
2024-04-07T18:24:23.695047Z DEBUG consensus::engine::hooks: Polled next hook hook="Prune" result=PolledHook { name: "Prune", event: Started, db_access_level: ReadWrite }
2024-04-07T18:24:23.695061Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.695074Z  INFO reth_payload_builder::service: New payload job created id=0x6fe61e7beeb51d1d parent=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21
2024-04-07T18:24:23.695076Z DEBUG pruner: Pruner started tip_block_number=1902597
2024-04-07T18:24:23.695129Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=1902595 prune_mode=Before(1902596)
2024-04-07T18:24:23.695130Z  INFO reth_node_core::events::node: Pruner started tip_block_number=1902597
2024-04-07T18:24:23.695622Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x6fe61e7beeb51d1d
2024-04-07T18:24:23.695686Z DEBUG payload_builder: building new payload id=0x6fe61e7beeb51d1d parent_hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 parent_number=1902597
2024-04-07T18:24:23.697273Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29, header: Header { parent_hash: 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6f7226e4a663f58a6d33c1fe79d002a8f4924b58fd8196b2ef3eb24477a25ab7, transactions_root: 0xca2031ce7a9393d6b23a9b484d3da4a2e0e15ce160b593d3664d63f2414504b1, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902598, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594543, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x18584753a7c7897035af6c10c466ec68494f5590ded406cccdf23aa2f6003f17, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xb6c895078798b4640a8f871cc63e01e90a82128559128d97c7b0f927883c14a4, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.697342Z DEBUG payload_builder: resolving fallback payload as best payload id=0x6fe61e7beeb51d1d
2024-04-07T18:24:23.697369Z DEBUG payload_builder: resolving best payload
2024-04-07T18:24:23.697478Z DEBUG payload_builder: building new payload id=0x6fe61e7beeb51d1d parent_hash=0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21 parent_number=1902597
2024-04-07T18:24:23.698244Z DEBUG blockchain_tree: Appending block to canonical chain head=0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29 parent=(1902597, 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21)
2024-04-07T18:24:23.698602Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T18:24:23.698802Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29, header: Header { parent_hash: 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6f7226e4a663f58a6d33c1fe79d002a8f4924b58fd8196b2ef3eb24477a25ab7, transactions_root: 0xca2031ce7a9393d6b23a9b484d3da4a2e0e15ce160b593d3664d63f2414504b1, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902598, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594543, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x18584753a7c7897035af6c10c466ec68494f5590ded406cccdf23aa2f6003f17, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xb6c895078798b4640a8f871cc63e01e90a82128559128d97c7b0f927883c14a4, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:23.698892Z DEBUG payload_builder: built better payload value=0
2024-04-07T18:24:23.700062Z DEBUG blockchain_tree::chain: Validated state root number=1902598 hash=0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29 elapsed=1.512172ms
2024-04-07T18:24:23.700141Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.700166Z  INFO reth_node_core::events::node: Block added to canonical chain number=1902598 hash=0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29 peers=0 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=1.957677ms
2024-04-07T18:24:23.700708Z  WARN consensus::engine: Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator. hook=Prune head_block_hash=0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29 safe_block_hash=0x77458a215a82a72f0de8e5080939bb6f7b89263fdd34074676c5243d3a9a9a29 finalized_block_hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe
2024-04-07T18:24:23.700732Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.700795Z DEBUG payload_builder: building new payload id=0x98a97e1c27afba46 parent_hash=0x9c52c4e39d3667e75120e7fac33f3e3b9ebfd1b2f9faa4cad4a844d795ad8ae8 parent_number=1902572
2024-04-07T18:24:23.701176Z DEBUG payload_builder: resolving best payload
2024-04-07T18:24:23.709411Z DEBUG payload_builder: building new payload id=0x2ccd19df244c5d34 parent_hash=0x3fc268ab7896e865eb21e189658d298b39ce921466caea7e1146303094f8a844 parent_number=1902573
2024-04-07T18:24:23.716889Z DEBUG payload_builder: building new payload id=0x46b7d300ffeea3ce parent_hash=0x192af96983f950034f1621402118d770d3336cea84617dde3b68c80274c480c1 parent_number=1902574
2024-04-07T18:24:23.727372Z DEBUG payload_builder: building new payload id=0xb213796cd2d86427 parent_hash=0xd9054bcc9fc06efe4e69dfebf5304feeb2a1e995517d69b1a7630dca31f5e643 parent_number=1902575
2024-04-07T18:24:23.736896Z DEBUG payload_builder: building new payload id=0xd5f8214c429b85c2 parent_hash=0xeaa906d9a98a0cc355763fa989a9093c1236c871414927630c08945e6a83991c parent_number=1902576
2024-04-07T18:24:23.745607Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=1902595 prune_mode=Before(1902596) output.pruned=8500
2024-04-07T18:24:23.745748Z DEBUG storage::db::mdbx: Commit total_duration=129.86µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 1, audit: 0, write: 0, sync: 4, ending: 0, whole: 6, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T18:24:23.745765Z DEBUG pruner: Pruner finished tip_block_number=1902597 elapsed=50.673333ms delete_limit=0 progress=HasMoreData stats={Transactions: (HasMoreData, 8500)}
2024-04-07T18:24:23.745807Z  INFO reth_node_core::events::node: Pruner finished tip_block_number=1902597 elapsed=50.673333ms stats={Transactions: (HasMoreData, 8500)}
2024-04-07T18:24:23.745879Z DEBUG consensus::engine::hooks: Polled running hook with db write access hook="Prune" result=PolledHook { name: "Prune", event: Finished(Ok(())), db_access_level: ReadWrite }
2024-04-07T18:24:23.746469Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T18:24:23.873790Z DEBUG payload_builder: building new payload id=0xb1b8ef2f1b11e0b7 parent_hash=0x7ed10333a5fe6ed25ac20b2132254bc53fb14a4d4da6c11fc203e93c69356f68 parent_number=1902577
2024-04-07T18:24:23.879241Z DEBUG payload_builder: building new payload id=0x69b2f279d4e69a17 parent_hash=0xdfa4f2518a5f8ab1adfcb424e33bdfa7c02fc07a82711bc2e8ba660723b139f3 parent_number=1902578
2024-04-07T18:24:23.887746Z DEBUG payload_builder: building new payload id=0x7cfc0cff0cab1c7e parent_hash=0xb6acec19b52eff94a13b1bdd9a82b066088c534c4caf1ac11d276bb483279500 parent_number=1902579
2024-04-07T18:24:23.900329Z DEBUG payload_builder: building new payload id=0xc3416a9aece08284 parent_hash=0xe6f7b6609f9ca6fe4735cd787a9f69e3d16f1208334a4d1f1d652b069fc19f1b parent_number=1902580
2024-04-07T18:24:23.904888Z DEBUG payload_builder: building new payload id=0xee7cc41ec6e68d34 parent_hash=0x56205d5bcb7a9a5c59a08da8531c4711c74d5331bac135b6ed7517c0a3d83f03 parent_number=1902581
2024-04-07T18:24:24.019568Z DEBUG payload_builder: building new payload id=0x4c27eb7f395ec725 parent_hash=0x37509b9cdc618b227d1c392179df4ca6f03416168cafdc6af45aa7f1c7981da9 parent_number=1902582
2024-04-07T18:24:24.033235Z DEBUG payload_builder: building new payload id=0xe30f3dfa943a7fe5 parent_hash=0xa5d40df5b3861a038258cade09a9770e6046e5ae09c9d91edb0e70d75d318824 parent_number=1902583
2024-04-07T18:24:24.038758Z DEBUG payload_builder: building new payload id=0xacb236c1b6b64177 parent_hash=0x088e70922178d731db03150f07394e83b6a38c2034d7068c87e3b7de0342d809 parent_number=1902584
2024-04-07T18:24:24.054286Z DEBUG payload_builder: building new payload id=0x88721efec332ee5b parent_hash=0xd025c56fac86df927da18b7890d536350cc910bfa7d6b5d49baf53523771c7a0 parent_number=1902585
2024-04-07T18:24:24.060827Z DEBUG payload_builder: building new payload id=0x75b2b50d964b9096 parent_hash=0x4526ece61869809199cd95bb6141abf2261f0f3415ff21a10318f78e076bc3c9 parent_number=1902586
2024-04-07T18:24:24.129450Z DEBUG payload_builder: building new payload id=0xfb380cc4c48a990c parent_hash=0x3731cce76cad54e487e6bc797abd251da6eee5a647768ca506fa6a6778dc8ae6 parent_number=1902587
2024-04-07T18:24:24.146091Z DEBUG payload_builder: building new payload id=0x1b62fca6c83aa91c parent_hash=0x920881b2f59d0b7b53868052e3009dbdc0803e99c6cba0b4a81497fe302ac1a4 parent_number=1902588
2024-04-07T18:24:24.158665Z DEBUG payload_builder: building new payload id=0x75e77f9330ea556d parent_hash=0x28e2abb5b1f41f16b631e1ddde91db621c34620f2645a182cf5419f61d7f9eee parent_number=1902589
2024-04-07T18:24:24.171132Z DEBUG payload_builder: building new payload id=0x7745f908471a2142 parent_hash=0xf397b1c31df2cb1f7b204a26d279e3e9c49d29cd2c9735b75af6f21dc3155d55 parent_number=1902590
2024-04-07T18:24:24.183689Z DEBUG payload_builder: building new payload id=0x1e3df767d5d3c53c parent_hash=0xa045515191f274f671b88e36c9386a5dd5facdc786ff311d15b1a4066619f243 parent_number=1902591
2024-04-07T18:24:24.354248Z DEBUG payload_builder: building new payload id=0x5c43970a27c2df69 parent_hash=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680 parent_number=1902592
2024-04-07T18:24:24.357497Z DEBUG payload_builder: building new payload id=0x2615eab975a12f4a parent_hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee parent_number=1902593
2024-04-07T18:24:24.359834Z DEBUG payload_builder: building new payload id=0xa7a9618ec35cbbd7 parent_hash=0x541707ae36111123ad6dab2acfa0b13bddaa1de3045589fb01b91df487cd07b9 parent_number=1902594
2024-04-07T18:24:24.678951Z DEBUG payload_builder: building new payload id=0xfee5be46ae7df940 parent_hash=0x3355e5568c863471328071809c67677174168e03cf9d41714951ea1962e86bfe parent_number=1902595
2024-04-07T18:24:24.684278Z DEBUG payload_builder: building new payload id=0x8e570a56f6904988 parent_hash=0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81 parent_number=1902596
2024-04-07T18:24:24.686170Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x872d2d3f3ccdd935565e55778018aec8cd50d7121ffa9f2d835fa3b94decda21, header: Header { parent_hash: 0x89d45ac987e734cec7ff6644af95ba6cc9847f32603912094e24015ca326fe81, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x2bd926d9b62ce68cd0794c8d9f5838d7d4dfe650574d1798820d1c12dbb1aaa2, transactions_root: 0x1bbace239c3b0427c5ba21c0bc3f0c08741b5c00b1c3ba57598e9956317a66e0, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1902597, gas_limit: 30000000, gas_used: 46913, timestamp: 1690594541, mix_hash: 0x8ea1b8213a1a0f3734c946330584cca3fc9ed31645516d59fc9aaf3c0b0041c0, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xde3de060516a5de0ba7cbd7c5405c10ddfe2bcaefa1f3e9a43ebbf90a5264c65, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x22962f3e11c1f8452f50339a0be89aeb4b3b1d050553bc4f06e572a7ab2ef545, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010f89650000000000000000000000000000000000000000000000000000000064c46baf0000000000000000000000000000000000000000000000000000000442ed92dbced21062c92a98112c908b1f54e58d996286b36396b3dec8f465100e87937d4300000000000000000000000000000000000000000000000000000000000000030000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T18:24:24.686297Z DEBUG payload_builder: built better payload value=0
2024-04-07T18:24:24.701891Z DEBUG payload_builder: building new payload id=0x98a97e1c27afba46 parent_hash=0x9c52c4e39d3667e75120e7fac33f3e3b9ebfd1b2f9faa4cad4a844d795ad8ae8 parent_number=1902572
2024-04-07T18:24:24.711154Z DEBUG payload_builder: building new payload id=0x2ccd19df244c5d34 parent_hash=0x3fc268ab7896e865eb21e189658d298b39ce921466caea7e1146303094f8a844 parent_number=1902573
2024-04-07T18:24:24.718551Z DEBUG payload_builder: building new payload id=0x46b7d300ffeea3ce parent_hash=0x192af96983f950034f1621402118d770d3336cea84617dde3b68c80274c480c1 parent_number=1902574
2024-04-07T18:24:24.729546Z DEBUG payload_builder: building new payload id=0xb213796cd2d86427 parent_hash=0xd9054bcc9fc06efe4e69dfebf5304feeb2a1e995517d69b1a7630dca31f5e643 parent_number=1902575
2024-04-07T18:24:24.738530Z DEBUG payload_builder: building new payload id=0xd5f8214c429b85c2 parent_hash=0xeaa906d9a98a0cc355763fa989a9093c1236c871414927630c08945e6a83991c parent_number=1902576
2024-04-07T18:24:24.875568Z DEBUG payload_builder: building new payload id=0xb1b8ef2f1b11e0b7 parent_hash=0x7ed10333a5fe6ed25ac20b2132254bc53fb14a4d4da6c11fc203e93c69356f68 parent_number=1902577
2024-04-07T18:24:24.879999Z DEBUG payload_builder: building new payload id=0x69b2f279d4e69a17 parent_hash=0xdfa4f2518a5f8ab1adfcb424e33bdfa7c02fc07a82711bc2e8ba660723b139f3 parent_number=1902578
2024-04-07T18:24:24.889260Z DEBUG payload_builder: building new payload id=0x7cfc0cff0cab1c7e parent_hash=0xb6acec19b52eff94a13b1bdd9a82b066088c534c4caf1ac11d276bb483279500 parent_number=1902579
2024-04-07T18:24:24.902485Z DEBUG payload_builder: building new payload id=0xc3416a9aece08284 parent_hash=0xe6f7b6609f9ca6fe4735cd787a9f69e3d16f1208334a4d1f1d652b069fc19f1b parent_number=1902580
2024-04-07T18:24:24.906040Z DEBUG payload_builder: building new payload id=0xee7cc41ec6e68d34 parent_hash=0x56205d5bcb7a9a5c59a08da8531c4711c74d5331bac135b6ed7517c0a3d83f03 parent_number=1902581
2024-04-07T18:24:25.021107Z DEBUG payload_builder: building new payload id=0x4c27eb7f395ec725 parent_hash=0x37509b9cdc618b227d1c392179df4ca6f03416168cafdc6af45aa7f1c7981da9 parent_number=1902582
2024-04-07T18:24:25.034184Z DEBUG payload_builder: building new payload id=0xe30f3dfa943a7fe5 parent_hash=0xa5d40df5b3861a038258cade09a9770e6046e5ae09c9d91edb0e70d75d318824 parent_number=1902583
2024-04-07T18:24:25.040625Z DEBUG payload_builder: building new payload id=0xacb236c1b6b64177 parent_hash=0x088e70922178d731db03150f07394e83b6a38c2034d7068c87e3b7de0342d809 parent_number=1902584
2024-04-07T18:24:25.055158Z DEBUG payload_builder: building new payload id=0x88721efec332ee5b parent_hash=0xd025c56fac86df927da18b7890d536350cc910bfa7d6b5d49baf53523771c7a0 parent_number=1902585
2024-04-07T18:24:25.062869Z DEBUG payload_builder: building new payload id=0x75b2b50d964b9096 parent_hash=0x4526ece61869809199cd95bb6141abf2261f0f3415ff21a10318f78e076bc3c9 parent_number=1902586
2024-04-07T18:24:25.131251Z DEBUG payload_builder: building new payload id=0xfb380cc4c48a990c parent_hash=0x3731cce76cad54e487e6bc797abd251da6eee5a647768ca506fa6a6778dc8ae6 parent_number=1902587
2024-04-07T18:24:25.147312Z DEBUG payload_builder: building new payload id=0x1b62fca6c83aa91c parent_hash=0x920881b2f59d0b7b53868052e3009dbdc0803e99c6cba0b4a81497fe302ac1a4 parent_number=1902588
2024-04-07T18:24:25.159914Z DEBUG payload_builder: building new payload id=0x75e77f9330ea556d parent_hash=0x28e2abb5b1f41f16b631e1ddde91db621c34620f2645a182cf5419f61d7f9eee parent_number=1902589
2024-04-07T18:24:25.172002Z DEBUG payload_builder: building new payload id=0x7745f908471a2142 parent_hash=0xf397b1c31df2cb1f7b204a26d279e3e9c49d29cd2c9735b75af6f21dc3155d55 parent_number=1902590
2024-04-07T18:24:25.186155Z DEBUG payload_builder: building new payload id=0x1e3df767d5d3c53c parent_hash=0xa045515191f274f671b88e36c9386a5dd5facdc786ff311d15b1a4066619f243 parent_number=1902591
2024-04-07T18:24:25.355031Z DEBUG payload_builder: building new payload id=0x5c43970a27c2df69 parent_hash=0x28c1b9d31e013d6f014a45c993db260efd04bda847971200b581b147c82c4680 parent_number=1902592
2024-04-07T18:24:25.359399Z DEBUG payload_builder: building new payload id=0x2615eab975a12f4a parent_hash=0x1783ac78d3c4663ec6589a889c85dadca51b9a7e8a4397cc7ae9bf5e60775fee parent_number=1902593
emhane commented 7 months ago

since v0.2.0-beta.5, pruning is being interrupted by timeout https://github.com/paradigmxyz/reth/pull/6958, assuming upgrading the node solves the issue. if not, please re-open @dallonby .

dallonby commented 7 months ago

This occurs running both v0.2.0-beta.5 and head

dallonby commented 7 months ago

I appear unable to reopen

emhane commented 7 months ago

related https://github.com/paradigmxyz/reth/issues/7505

jankjr commented 7 months ago

I can verify this still happens on latest HEAD 711de3b08d62857d2c3959468eb327c443f776ca

Logs op-node:

parent_hash=0xcc271f0a219e863a1383e1dddaff54d80574b2a045bb8040ee79cd0fd43432fa batch_epoch=0xe8ca726a1a92fcb6822526536096dc55de2670ac56606f1921d42a48e465472b:17792388 txs=0 parent=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920:1884073 parent_time=1690557493
t=2024-04-16T10:49:37+0200 lvl=info msg="Generating next batch" epoch=0x9e1d2ebfe5f0aaa4903655776c433b91ef7bb9a019aa2dc4a5d33f57b3fbe364:17792394 timestamp=1690557495
t=2024-04-16T10:49:37+0200 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1690557495
t=2024-04-16T10:49:37+0200 lvl=info msg="inserted block" hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 number=1884074 state_root=0x924daa236c26ae48767a1375f4f76db828e3f0babbe0a2849c869a17b190d6d9 timestamp=1690557495 parent=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 prev_randao=0x75be4217e571ba7459592cc119a099c98f9e79b595b08c3701e5e60e766ab338 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 update_safe=true
t=2024-04-16T10:49:37+0200 lvl=info msg="Sync progress" reason="processed safe block derived from L1" l2_finalized=0xb9fa1d4a1d109d025f749ae7e0f129f5a2836e40f155c7d8b735367d04c99834:1883777 l2_safe=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 l2_pending_safe=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 l2_unsafe=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1690557495 l1_derived=0x6d449d294c8950471694e7b9cf2c2f3443ab76d27a5286ad8cd5a9113df6e3b0:17795994
t=2024-04-16T10:49:37+0200 lvl=warn msg="ignoring batch with mismatching parent hash" batch_index=0 batch_type=SingularBatch batch_timestamp=1690557497 parent_hash=0xa89bf2fa0f65975a9265cfc107b5807e3d4b9b241b2b72775b10dbd9168659c1 batch_epoch=0xe8ca726a1a92fcb6822526536096dc55de2670ac56606f1921d42a48e465472b:17792388 txs=0 current_safe_head=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22
t=2024-04-16T10:49:37+0200 lvl=warn msg="Dropping batch" batch_type=SingularBatch batch_timestamp=1690557497 parent_hash=0xa89bf2fa0f65975a9265cfc107b5807e3d4b9b241b2b72775b10dbd9168659c1 batch_epoch=0xe8ca726a1a92fcb6822526536096dc55de2670ac56606f1921d42a48e465472b:17792388 txs=0 parent=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 parent_time=1690557495
t=2024-04-16T10:49:37+0200 lvl=info msg="Generating next batch" epoch=0x9e1d2ebfe5f0aaa4903655776c433b91ef7bb9a019aa2dc4a5d33f57b3fbe364:17792394 timestamp=1690557497
t=2024-04-16T10:49:37+0200 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1690557497
t=2024-04-16T10:49:37+0200 lvl=error msg="cancelling old block sealing job" payload=0x2b2c84d225fe8a46
t=2024-04-16T10:49:37+0200 lvl=warn msg="Failed to get payload" payload_id=0x2b2c84d225fe8a46 payload_id=0x2b2c84d225fe8a46 err="Unknown payload"
t=2024-04-16T10:49:37+0200 lvl=error msg="failed to cancel block building job" payload=0x2b2c84d225fe8a46 err="unrecognized rpc error: Unknown payload"
t=2024-04-16T10:49:37+0200 lvl=warn msg="could not process payload derived from L1 data, dropping batch" err="failed to complete building on top of L2 chain 0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074, id: 0x2b2c84d225fe8a46, error (3): updated forkchoice, but node is syncing"
t=2024-04-16T10:49:37+0200 lvl=error msg="deposit only block was invalid" parent=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 err="failed to complete building on top of L2 chain 0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074, id: 0x2b2c84d225fe8a46, error (3): updated forkchoice, but node is syncing"
t=2024-04-16T10:49:37+0200 lvl=error msg="Derivation process critical error" err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074, id: 0x2b2c84d225fe8a46, error (3): updated forkchoice, but node is syncing"
t=2024-04-16T10:49:37+0200 lvl=info msg="State loop returned"

Logs reth:

2024-04-16T08:49:37.252495Z  INFO Block added to canonical chain number=1884073 hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 peers=3 txs=1 mgas=0.064 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=478.423µs
2024-04-16T08:49:37.253398Z  INFO Canonical chain committed number=1884073 hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 elapsed=731.866µs
2024-04-16T08:49:37.303133Z  INFO Block is already canonical, ignoring. block_hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920
2024-04-16T08:49:37.303168Z  INFO Canonical chain committed number=1884073 hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 elapsed=46.607µs
2024-04-16T08:49:37.303210Z  INFO New payload job created id=0x8b627a82aa609590 parent=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920
2024-04-16T08:49:37.304555Z  INFO Block added to canonical chain number=1884074 hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 peers=3 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=539.216µs
2024-04-16T08:49:37.305426Z  INFO Canonical chain committed number=1884074 hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 elapsed=761.801µs
2024-04-16T08:49:37.371254Z  INFO Block is already canonical, ignoring. block_hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22
2024-04-16T08:49:37.371586Z  INFO Pruner started tip_block_number=1884074
2024-04-16T08:49:37.371602Z  INFO Canonical chain committed number=1884074 hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 elapsed=51.606µs
2024-04-16T08:49:37.371678Z  INFO New payload job created id=0x2b2c84d225fe8a46 parent=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22
2024-04-16T08:49:37.374554Z  INFO Block added to canonical chain number=1884075 hash=0xbd3b7e03cf9e9e5cfaf0c6fd58160032b24baecc6051f9359a1794005762860d peers=3 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=1.231837ms
2024-04-16T08:49:37.374999Z  WARN Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator. hook=Prune head_block_hash=0xbd3b7e03cf9e9e5cfaf0c6fd58160032b24baecc6051f9359a1794005762860d safe_block_hash=0xbd3b7e03cf9e9e5cfaf0c6fd58160032b24baecc6051f9359a1794005762860d finalized_block_hash=0xb9fa1d4a1d109d025f749ae7e0f129f5a2836e40f155c7d8b735367d04c99834
2024-04-16T08:49:37.377031Z  INFO Pruner finished tip_block_number=1884074 elapsed=5.681868ms stats={}
2024-04-16T08:50:01.192094Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:50:26.192002Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:50:51.191779Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:51:16.191845Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:51:41.191966Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:52:06.191916Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:52:31.192208Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:52:56.191791Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:53:21.191788Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:53:46.191789Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:54:08.183008Z  WARN Beacon client online, but no consensus updates received for a while. Please fix your beacon client to follow the chain! period=270.808010513s
0xZerohero commented 7 months ago

I ran into this problem too. Didn't have time to look into it in detail, but can confirm that after patching my archive node to disable the pruner, syncing is running without any issues. @emhane do we even need the pruner hook for archive nodes ?

emhane commented 7 months ago

@0xZerohero thanks, good to know syncing is smooth with disabled pruner. archive full nodes prune history. atm timeout is only soft for history pruning (is only checked when history pruning returns). so atm timeout only helps on full archive nodes, not archive full nodes.

dallonby commented 7 months ago

[prune] block_interval = 999999999 in reth.toml also 'resolves'

popconst commented 7 months ago

I ran into this problem too. Didn't have time to look into it in detail, but can confirm that after patching my archive node to disable the pruner, syncing is running without any issues. @emhane do we even need the pruner hook for archive nodes ?

Would you mind sharing how you patched reth to disable the pruning? I'm having the same issue I think.

slordua commented 7 months ago

I ran into this problem too. Didn't have time to look into it in detail, but can confirm that after patching my archive node to disable the pruner, syncing is running without any issues. @emhane do we even need the pruner hook for archive nodes ?

Would you mind sharing how you patched reth to disable the pruning? I'm having the same issue I think.

+1

popconst commented 7 months ago

Noticed that if I restart the op-node after it gets stuck, it's able to resume the sync.

popconst commented 7 months ago

Thank you @dallonby for sharing the tip on how to solve it.

What I did is make a copy of reth.toml and add in

[prune]
block_interval = 999999999

Then I started op-geth with --config PATH/TO/MY/reth.toml

I ran into this problem too. Didn't have time to look into it in detail, but can confirm that after patching my archive node to disable the pruner, syncing is running without any issues. @emhane do we even need the pruner hook for archive nodes ?

Would you mind sharing how you patched reth to disable the pruning? I'm having the same issue I think.

+1

andreclaro commented 6 months ago

I have tested op-reth on Base mainnet and sepolia networks in the last days.

The node syncs faster compare with op-geth and op-erigon, however the following issues were observed on both base mainnet and sepolia:

  1. It was only possible to start the node using [v0.2.0-beta.6](https://github.com/paradigmxyz/reth/tree/v0.2.0-beta.6). However the Reth team recommended to use the main branch, but the node was unable to start processing blocks due to the errors:
    1. op-reth:
      1. WARN Error while processing payload error=Failed to insert block
      2. WARN Invalid block error on new payload invalid_hash
      3. WARN Bad block with hash hash=
    2. op-node:
      1. lvl=error msg="cancelling old block sealing job"
      2. lvl=warn msg="could not process payload derived from L1 data, dropping batch"
      3. lvl=error msg="deposit only block was invalid"
      4. lvl=error msg="Derivation process critical error" err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain
  2. Multiple times on both network, mainnet and sepolia, the node stopped processing new blocks and only after restarting both clients (op-reth and op-node) it starts to process blocks again. The errors observed were the same from point 1.
  3. It was only possible to use the main branch after the node was started using [v0.2.0-beta.6](https://github.com/paradigmxyz/reth/tree/v0.2.0-beta.6).

Configuration used:

[prune]
# Minimum pruning interval measured in blocks
# https://github.com/paradigmxyz/reth/pull/8308
block_interval = 500_000

[prune.parts]
# Sender Recovery pruning configuration
sender_recovery = { distance = 1_300_000 } # Prune all transaction senders before the block `head-100000`, i.e. keep transaction senders for the last 100001 blocks

# Transaction Lookup pruning configuration
transaction_lookup = { distance = 1_300_000 } # Prune all TxNumber => TxHash mappings

# Receipts pruning configuration. This setting overrides `receipts_log_filter`.
receipts = { distance = 1_300_000 } # Prune all receipts from transactions before the block 1920000, i.e. keep receipts from the block 1920000

# Account History pruning configuration
account_history = { distance = 1_300_000 } # Prune all historical account states before the block `head-100000`

# Storage History pruning configuration
storage_history = { distance = 1_300_000 } # Prune all historical storage states before the block `head-100000`

Errors

As described above, these errors were observed multiple times, using the main branch and also using the latest release [v0.2.0-beta.6](https://github.com/paradigmxyz/reth/tree/v0.2.0-beta.6).

op-reth

May 29 07:42:51 m-base-02 op-reth[2595147]: 2024-05-29T07:42:51.216258Z  INFO Canonical chain committed number=0 hash=0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd elapsed=124.414µs
May 29 07:42:51 m-base-02 op-reth[2595147]: 2024-05-29T07:42:51.216352Z  INFO New payload job created id=0x2544c58091e5276b parent=0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd
May 29 07:42:51 m-base-02 op-reth[2595147]: 2024-05-29T07:42:51.264295Z  WARN Error while processing payload error=Failed to insert block (hash=0x07eee59d552f482e0f350933b8e96f824aa0bfa28267de53dcb0ddbe868a4bf9, number=1, parent_hash=0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd): receipt root mismatch: got 0x99d7563fd44dac89d8516b5ba56e109cf073641e9b58a1b8601f80757b2b6d74, expected 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85
May 29 07:42:51 m-base-02 op-reth[2595147]: 2024-05-29T07:42:51.264339Z  WARN Invalid block error on new payload invalid_hash=0x07eee59d552f482e0f350933b8e96f824aa0bfa28267de53dcb0ddbe868a4bf9 invalid_number=1 error=receipt root mismatch: got 0x99d7563fd44dac89d8516b5ba56e109cf073641e9b58a1b8601f80757b2b6d74, expected 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85
May 29 07:42:51 m-base-02 op-reth[2595147]: 2024-05-29T07:42:51.264366Z  WARN Bad block with hash hash=0x07eee59d552f482e0f350933b8e96f824aa0bfa28267de53dcb0ddbe868a4bf9 header=Header { parent_hash: 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xe3e10cb3963f6b88b7ba14cc76c34de65308fc0524dca48410ee53df2a43700b, transactions_root: 0xee45a6283d626ebed087ac705c71740d7ecb276d81b745284a68b88925dc91ae, receipts_root: 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 1, gas_limit: 30000000, gas_used: 64013, timestamp: 1686789349, mix_hash: 0xae36673a68b9370a513a06698115fd86cc7faf4d6c248a982fbc3ae29c442da2, nonce: 0, base_fee_per_gas: Some(980000000), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, requests_root: None, extra_data: 0x }
May 29 07:42:58 m-base-02 op-reth[2595147]: 2024-05-29T07:42:58.476573Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:43:23 m-base-02 op-reth[2595147]: 2024-05-29T07:43:23.476462Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:43:48 m-base-02 op-reth[2595147]: 2024-05-29T07:43:48.477067Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:44:13 m-base-02 op-reth[2595147]: 2024-05-29T07:44:13.477023Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:44:38 m-base-02 op-reth[2595147]: 2024-05-29T07:44:38.476535Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:45:03 m-base-02 op-reth[2595147]: 2024-05-29T07:45:03.477192Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:45:28 m-base-02 op-reth[2595147]: 2024-05-29T07:45:28.477243Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:45:53 m-base-02 op-reth[2595147]: 2024-05-29T07:45:53.477076Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:46:18 m-base-02 op-reth[2595147]: 2024-05-29T07:46:18.477294Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:46:43 m-base-02 op-reth[2595147]: 2024-05-29T07:46:43.477112Z  INFO Status connected_peers=0 freelist=2 latest_block=0
May 29 07:47:05 m-base-02 op-reth[2595147]: 2024-05-29T07:47:05.477104Z  WARN Beacon client online, but no consensus updates received for a while. This may be because of a reth error, or an error in the beacon client! Please investigate reth and beacon client logs! period=254.261006858s
May 29 07:47:08 m-base-02 op-reth[2595147]: 2024-05-29T07:47:08.476428Z  INFO Status connected_peers=0 freelist=2 latest_block=0

op-node

May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=info msg="created new channel" origin=0xb0beed7a7e12eaf620fcf9cd6f6674ff08d3bc1e4851978ba0824fd43509ddf1:17482370 channel=47d44b9235706648a325ead11fa5121a length=105637 frame_number=0 is_last=true
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=info msg="Reading channel" channel=47d44b9235706648a325ead11fa5121a frames=1
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=info msg="Found next batch" batch_type=SingularBatch batch_timestamp=1686789349 parent_hash=0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd batch_epoch=0x5c13d307623a926cd31415036c8b7fa14572f9dac64528e857a470511fc30771:17481768 txs=0
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1686789349
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=error msg="cancelling old block sealing job" payload=0x2544c58091e5276b
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=warn msg="could not process payload derived from L1 data, dropping batch" err="failed to complete building on top of L2 chain 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd:0, id: 0x2544c58091e5276b, error (3): execution payload 0x07eee59d552f482e0f350933b8e96f824aa0bfa28267de53dcb0ddbe868a4bf9:1 was INVALID! Latest valid hash is 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd, ignoring bad block: 0xc0198e7d90"
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=error msg="deposit only block was invalid" parent=0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd:0 err="failed to complete building on top of L2 chain 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd:0, id: 0x2544c58091e5276b, error (3): execution payload 0x07eee59d552f482e0f350933b8e96f824aa0bfa28267de53dcb0ddbe868a4bf9:1 was INVALID! Latest valid hash is 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd, ignoring bad block: 0xc0198e7d90"
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=error msg="Derivation process critical error" err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd:0, id: 0x2544c58091e5276b, error (3): execution payload 0x07eee59d552f482e0f350933b8e96f824aa0bfa28267de53dcb0ddbe868a4bf9:1 was INVALID! Latest valid hash is 0xf712aa9241cc24369b143cf6dce85f0902a9731e70d66818a3a5845b296c73dd, ignoring bad block: 0xc0198e7d90"
May 29 07:42:51 m-base-02 op-node[2595204]: t=2024-05-29T07:42:51+0000 lvl=info msg="State loop returned"
May 29 07:45:13 m-base-02 op-node[2595204]: t=2024-05-29T07:45:13+0000 lvl=warn msg="failed to notify engine driver of L1 head change" err="context deadline exceeded"
May 29 07:45:25 m-base-02 op-node[2595204]: t=2024-05-29T07:45:25+0000 lvl=warn msg="failed to notify engine driver of L1 head change" err="context deadline exceeded"
May 29 07:45:37 m-base-02 op-node[2595204]: t=2024-05-29T07:45:37+0000 lvl=warn msg="failed to notify engine driver of L1 head change" err="context deadline exceeded"
emhane commented 6 months ago

I would advise syncing with --debug.tip tbh, getting the hash to pass to that flag from latest block on https://optimistic.etherscan.io . think that's the absolute fastest, since no engine messages to process then. when it has synced to that fixed tip, then connect it to op-node, to track moving tip.

emhane commented 6 months ago

try this op-node flag to improve speed, then context deadline exceeded error should be less likely to occur. make sure the op-node and op-reth nodes are connected over localhost for security reason.

    --l1.trustrpc                       (default: false)                   ($OP_NODE_L1_TRUST_RPC)
          Trust the L1 RPC, sync faster at risk of malicious/buggy RPC providing bad or
          inconsistent L1 data

@andreclaro

github-actions[bot] commented 5 months ago

This issue is stale because it has been open for 21 days with no activity.

github-actions[bot] commented 4 months ago

This issue was closed because it has been inactive for 7 days since being marked as stale.

jun0tpyrc commented 1 month ago

see same issiue on v0.10.2 with/without --l1.trustrpc , a node slowdown for this (after getting in sync within 1 block delay) for this gradually

emhane commented 1 month ago

@jun0tpyrc the error logs show that it's due to pruning as well?

jun0tpyrc commented 1 month ago

found seem not quite the samething although some error /warn may be similar in my case i found default for reth

[stages.merkle]
# The threshold in number of blocks before the stage starts from scratch
# and re-computes the state root, discarding the trie that has already been built,
# as opposed to incrementally updating the trie.
clean_threshold = 5000

maybe too io heavy for mainnets ike base-mainnet we do archive nodes without pruning

@jun0tpyrc the error logs show that it's due to pruning as well?

suggest if we could include some suggestions for those tunings in doc/tweak tips in repo

i am still not consistently stay in sync for long enough period (maybe some slow down after sometimes) sometimes i have to reboot op-node to make things better (or some compatibility issue with beacon node lighthouse in other thread)

emhane commented 1 month ago

could you please open a new issue @jun0tpyrc ?