search

ethereum-optimism / developers

This repository is to serve as a place where builders on every level of the OP Stack can come to collaborate.
Creative Commons Zero v1.0 Universal
73 stars 45 forks source link

[Node Operator Issue] Granite Release bricks node, full re-sync required #565

Closed moorow closed 2 months ago

moorow commented 2 months ago

Discussed in https://github.com/ethereum-optimism/developers/discussions/564

Originally posted by **moorow** August 30, 2024 ### Are you running the most up to date node software? - [X] Yes ### Did you check the documentation? - [X] Yes ### Did you check for duplicate questions? - [X] Yes ### Issue Description Updating to the Granite release seems to break the state of a node. Here are steps I followed and logs: 1) Graceful stop op-node 2) Graceful stop op-geth 3) Start op-geth (New version) ``` INFO [08-30|09:27:51.999] Post-Merge hard forks (timestamp based): INFO [08-30|09:27:51.999] - Shanghai: @1704992401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md) INFO [08-30|09:27:51.999] - Cancun: @1710374401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md) INFO [08-30|09:27:51.999] - Regolith: @0 INFO [08-30|09:27:51.999] - Canyon: @1704992401 INFO [08-30|09:27:51.999] - Ecotone: @1710374401 INFO [08-30|09:27:51.999] - Fjord: @1720627201 INFO [08-30|09:27:51.999] - Granite: @1726070401 INFO [08-30|09:27:51.999] INFO [08-30|09:27:51.999] --------------------------------------------------------------------------------------------------------------------------------------------------------- INFO [08-30|09:27:51.999] INFO [08-30|09:27:52.000] Loaded most recent local block number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=1m49s INFO [08-30|09:27:52.000] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=26m31s INFO [08-30|09:27:52.000] Loaded last snap-sync pivot marker number=123,961,326 WARN [08-30|09:27:52.000] Head state missing, repairing number=124,705,593 hash=953dd8..ba9ede snaproot=2a2e50..f5c381 INFO [08-30|09:28:03.101] Block state missing, rewinding further number=124,703,029 hash=baeb46..e4fa7a elapsed=11.100s INFO [08-30|09:28:11.101] Block state missing, rewinding further number=124,612,195 hash=441943..78c159 elapsed=19.100s INFO [08-30|09:28:19.101] Block state missing, rewinding further number=124,466,580 hash=de38e2..34caa6 elapsed=27.100s INFO [08-30|09:28:27.101] Block state missing, rewinding further number=124,251,092 hash=62041b..f1a58b elapsed=35.101s INFO [08-30|09:28:35.101] Block state missing, rewinding further number=124,005,258 hash=798f45..1b1643 elapsed=43.101s INFO [08-30|09:28:36.505] Pivot block reached, resetting to genesis number=123,961,326 hash=5678e6..fd8b23 INFO [08-30|09:28:36.507] Chain is stateless, wait state sync number=0 hash=7ca38a..3fa48b INFO [08-30|09:28:36.508] Loaded most recent local header number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s INFO [08-30|09:28:36.508] Loaded most recent local block number=0 hash=7ca38a..3fa48b td=1 age=55y5mo2w INFO [08-30|09:28:36.508] Loaded most recent local snap block number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s INFO [08-30|09:28:36.508] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=27m15s INFO [08-30|09:28:36.508] Loaded last snap-sync pivot marker number=123,961,326 WARN [08-30|09:28:36.509] Enabling snapshot recovery chainhead=0 diskbase=124,705,308 WARN [08-30|09:28:36.526] Snapshot is not continuous with chain snaproot=1a96d0..894acc chainroot=eddb4c..a568eb INFO [08-30|09:28:36.526] Initialized transaction indexer range="last 2350000 blocks" INFO [08-30|09:28:36.526] Initialising Ethereum protocol network=10 dbversion=8 INFO [08-30|09:28:36.526] Loaded local transaction journal transactions=0 dropped=0 INFO [08-30|09:28:36.526] Enabled snap sync head=0 hash=7ca38a..3fa48b INFO [08-30|09:28:36.526] Gasprice oracle is ignoring threshold set threshold=2 WARN [08-30|09:28:36.527] Engine API enabled protocol=eth ``` 3) Start op-node (New version) ``` t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got " ``` ### Protocol Description network: op-mainnet chainid: 10 L1 chain: mainnet **Old Versions** op-geth: v1.101315.2 op-node: v1.7.7 **New Versions** op-geth: v1.101408.0 op-node: v1.9.1 ### Node Logs _No response_ ### Additional Information _No response_ ### Feedback _No response_
moorow commented 2 months ago

Based on some discussion on Discord it looks like this may be related to Path Based (PBSS)?

Someone said that this only affects Full nodes, not Archive.

INFO [08-30|09:27:51.974] State schema set to default              scheme=path

This line was printed in the logs

sbvegan commented 2 months ago

Thanks for the flag, can you share your runtime configuration you're using for op-mode and op-geth?

moorow commented 2 months ago

Sure thing:

op-geth

      - --ws
      - --ws.port=8550
      - --ws.addr=0.0.0.0
      - --ws.origins="*"
      - --http
      - --http.port=8549
      - --http.addr=0.0.0.0
      - --http.vhosts="*"
      - --http.corsdomain="*"
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/data/jwt.txt
      - --authrpc.port=9333
      - --authrpc.vhosts="*"
      - --datadir=/data/geth
      - --verbosity=3
      - --rollup.disabletxpoolgossip=true
      - --rollup.sequencerhttp=https://mainnet-sequencer.optimism.io
      - --op-network=op-mainnet
      - --syncmode=snap
      - --rpc.gascap=0
      - --rpc.evmtimeout=10s
      - --rpc.batch-response-max-size=50000000
      - --rpc.batch-request-limit=2000
      - --nat=extip:${HOST_IP_ADDRESS}

op-node

      - --l1=${L1_EXECUTION_RPC_URL}
      - --l1.rpckind=alchemy
      - --l1.trustrpc
      - --l1.beacon=${L1_BEACON_RPC_URL}
      - --l2=ws://op-geth:9333
      - --network=op-mainnet
      - --rpc.addr=127.0.0.1
      - --rpc.port=9545
      - --l2.jwt-secret=/data/jwt.txt
      - --syncmode=execution-layer
      - --p2p.advertise.ip=${HOST_IP_ADDRESS}
      - --p2p.listen.ip=0.0.0.0
      - --p2p.listen.tcp=9003
      - --p2p.listen.udp=9003
      - --p2p.advertise.tcp=9003
      - --p2p.advertise.udp=9003
      - --metrics.enabled
      - --metrics.addr=0.0.0.0
      - --metrics.port=7300