paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.91k stars 1.16k forks source link

Failed to fetch finalized head in op-node #10284

Closed AntonieDavid closed 1 month ago

AntonieDavid commented 2 months ago

Describe the bug

I have followed the documentation regarding exporting/importing the prebedrock chain and right after starting the op-reth and op-node services, the op-node logs show the same error detailed in the node logs below. It's been 2 days since startup and nothing has changed. Op-reth logs looked all right at the begining, but for some time now (more than 1/2 a day) it has been stuck showing the logs presented below. Probably a restart would solve the op-reth issue, but I think it's related to op-node and it will get stuck again eventually. I am attaching the service files used to run the nodes. Worth mentioning that I am running 2 nodes with the only difference being that one has --debug.tip 0xef96c413427e0b350953039ca23c49a28f80cd22c87a61c0b25ae27da1ae769d in the op-reth configs and the other doesn't, but still showing the exact same error on both.

ExecStart=/usr/local/bin/op-reth node \ --ws --ws.port=8546 --ws.addr=0.0.0.0 --ws.origins="*" \ --http --http.port=8545 --http.addr=0.0.0.0 \ --http.corsdomain="*" \ --authrpc.addr=0.0.0.0 \ --authrpc.jwtsecret=/node/data/secret.jwt \ --authrpc.port=8551 \ --datadir=/node/data/reth/ \ --rollup.sequencer-http=https://mainnet-sequencer.optimism.io/ \ --http.api=admin,debug,eth,net,trace,web3,rpc,reth,ots,eth-call-bundle \ --ws.api=admin,debug,eth,net,trace,web3,rpc,reth,ots,eth-call-bundle \ --metrics 0.0.0.0:6060 \ --chain optimism \ --rpc.gascap 18446744073709551615 \ --rpc.max-connections 429496729 \ --rpc.max-logs-per-response 0 \ --rpc.max-subscriptions-per-connection 10000 \ --rollup.disable-tx-pool-gossip \ --debug.tip 0xef96c413427e0b350953039ca23c49a28f80cd22c87a61c0b25ae27da1ae769d

ExecStart=/usr/local/bin/op-node --l1.trustrpc=true --l1=https://eth-mainnet.... \ --l2=ws://localhost:8551 --rpc.addr=0.0.0.0 \ --rpc.port=9545 --l2.jwt-secret=/node/data/secret.jwt \ --metrics.enabled --network op-mainnet --l1.beacon http://...... \ --syncmode=execution-layer

Steps to reproduce

Starting the node with the mentioned conditions. Main issue is in op-node logs.

Node logs

Op-reth logs:
Aug 13 07:18:23 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:18:23.768889Z  WARN Post-merge network, but never seen beacon client. Please launch one to follow the chain!
Aug 13 07:18:26 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:18:26.774895Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0
Aug 13 07:18:51 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:18:51.774814Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0
Aug 13 07:19:16 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:19:16.774709Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0
Aug 13 07:19:41 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:19:41.773921Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0
Aug 13 07:20:06 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:20:06.774077Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0
Aug 13 07:20:31 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:20:31.774491Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0
Aug 13 07:20:56 optimism-archive-mainnet-1-euc1 op-reth[746862]: 2024-08-13T07:20:56.774953Z  INFO Status connected_peers=84 freelist=5992404 latest_block=0

Op-node logs:
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=info msg="Received signed execution payload from p2p" id=0xba01628a69239c4f51556a455f3b6d3f59b4d571040e3bd16492af51e709cf85:123967467 peer=16Uiu2HAmRhsnEigxSwu4vDGed6h8AtWcGHTRmRf2jrYt33emZ5g8
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0xba01628a69239c4f51556a455f3b6d3f59b4d571040e3bd16492af51e709cf85:123967467
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=warn msg="Failed to insert unsafe payload for EL sync" id=0xba01628a69239c4f51556a455f3b6d3f59b4d571040e3bd16492af51e709cf85:123967467 err="temp: failed to fetch finalized head: failed to determine L2BlockRef of finalized, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm1AMWdJHLXSbuXqZwCxhsssyGLE8qpu8Wja71CbcriXEF
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmRAw82BdrF58ToB8ZYkgpotfwP3EitGcVvjg9fXTYXtJY
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=info msg="connected to peer" peer=16Uiu2HAm1AMWdJHLXSbuXqZwCxhsssyGLE8qpu8Wja71CbcriXEF addr=/ip4/34.65.100.167/tcp/9003
Aug 13 07:21:51 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:51+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAm1AMWdJHLXSbuXqZwCxhsssyGLE8qpu8Wja71CbcriXEF
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmJ69FgNT2Nn9Bm7HvrUQ598VxP2hhMZK1WAvDvZNkNGoV
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm38DvhVVYBBnjGBA71Ycdh9zMc1ThSbFkcArbmMG7sqdL
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmKsgJ2ptZHrg7LtGvs5HoVzmKryYUEeoJmfAGd6scJxCW
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmRAw82BdrF58ToB8ZYkgpotfwP3EitGcVvjg9fXTYXtJY addr=/ip4/208.115.202.178/tcp/9222
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmRAw82BdrF58ToB8ZYkgpotfwP3EitGcVvjg9fXTYXtJY
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmF8M2hudw25Jg2744zzLxNdAdQJEDEdZh8GA7VQt2rgs3
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkwWrzYrLh4f6ihEW4XbFA82okQ5QvcAGNQtoi2aZZXWHC
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmKsgJ2ptZHrg7LtGvs5HoVzmKryYUEeoJmfAGd6scJxCW addr=/ip4/173.231.44.87/tcp/9222
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmKsgJ2ptZHrg7LtGvs5HoVzmKryYUEeoJmfAGd6scJxCW
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="connected to peer" peer=16Uiu2HAkwWrzYrLh4f6ihEW4XbFA82okQ5QvcAGNQtoi2aZZXWHC addr=/ip4/51.81.107.127/tcp/30222
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAkwWrzYrLh4f6ihEW4XbFA82okQ5QvcAGNQtoi2aZZXWHC
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmJN3VbzSTFfDsnXErEFcD2Hozcbmq33B4ugfy9iy1DMFH
Aug 13 07:21:52 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:52+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmRawUcpgjTc86XgA94bReBTkV4LxgGgAK6EiUq4a8Fpim
Aug 13 07:21:53 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:53+0000 lvl=info msg="Received signed execution payload from p2p" id=0x3f2da1a664100f23e179b027db41149e1aa1f401bbb2898b16dc3794622ec0fe:123967468 peer=16Uiu2HAmRhsnEigxSwu4vDGed6h8AtWcGHTRmRf2jrYt33emZ5g8
Aug 13 07:21:53 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:53+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0x3f2da1a664100f23e179b027db41149e1aa1f401bbb2898b16dc3794622ec0fe:123967468
Aug 13 07:21:53 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:53+0000 lvl=warn msg="Failed to insert unsafe payload for EL sync" id=0x3f2da1a664100f23e179b027db41149e1aa1f401bbb2898b16dc3794622ec0fe:123967468 err="temp: failed to fetch finalized head: failed to determine L2BlockRef of finalized, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
Aug 13 07:21:55 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:55+0000 lvl=info msg="Received signed execution payload from p2p" id=0x9d94d226cc20aecb59d137eb20f8627db6ca0d130d8a60e8a3e949f00f8021ef:123967469 peer=16Uiu2HAmRhsnEigxSwu4vDGed6h8AtWcGHTRmRf2jrYt33emZ5g8
Aug 13 07:21:55 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:55+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0x9d94d226cc20aecb59d137eb20f8627db6ca0d130d8a60e8a3e949f00f8021ef:123967469
Aug 13 07:21:55 optimism-archive-mainnet-1-euc1 op-node[746863]: t=2024-08-13T07:21:55+0000 lvl=warn msg="Failed to insert unsafe payload for EL sync" id=0x9d94d226cc20aecb59d137eb20f8627db6ca0d130d8a60e8a3e949f00f8021ef:123967469 err="temp: failed to fetch finalized head: failed to determine L2BlockRef of finalized, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"

Platform(s)

Linux (x86)

What version/commit are you on?

op-reth --version reth Version: 1.0.4 Commit SHA: e24e4c773d7571a5a54dba7854643c02d0b0a841 Build Timestamp: 2024-08-07T14:53:14.988149886Z Build Features: asm_keccak,jemalloc,optimism Build Profile: maxperf

What database version are you on?

op-reth db version 2024-08-13T07:04:51.376236Z INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/optimism Current database version: 2 Local database version: 2

Which chain / network are you on?

optimism mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

[stages.prune] commit_threshold = 1000000

If you've built Reth from source, provide the full command you used

source "$HOME/.cargo/env" && RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf --features jemalloc,asm-keccak,optimism --bin op-reth

Code of Conduct

joshieDo commented 2 months ago

There is an extra flag which is mentioned on running optimism, but went amiss on the specific op-mainnet page: https://github.com/paradigmxyz/reth/pull/10289

--l2.enginekind=reth should also be added to the running command.

At the same time, can you double check your import ? it shows the latest block as 0, but it should be higher

AntonieDavid commented 2 months ago

I made a mistake regarding the logs provided (i copied logs from the wrong node which as you mentioned had incomplete import of blocks), the correct op-reth logs looked similar to this, except they were at 100% for a long time before restarting the service (can't regenerate those logs as I restarted the service). Aug 07 01:34:07 optimism-archive-mainnet-0-euc1 op-reth[1020504]: 2024-08-07T01:34:07.021001Z INFO Status connected_peers=15 freelist=3098403 stage=Bodies checkpoint=108992000 target=123681893 stage_progress=88.12% Regarding the mentioned flag for op-node, I couldn't find the reference to it in op-node --help and also it returns an undefined flag error: t=2024-08-14T00:20:16+0000 lvl=crit msg="Application failed" message="flag provided but not defined: -l2.enginekind"

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 21 days with no activity.

github-actions[bot] commented 1 month ago

This issue was closed because it has been inactive for 7 days since being marked as stale.