Closed 0x366 closed 1 week ago
I encountered the same problem...
WARN [04-23|09:07:33.612] Unclean shutdown detected booted=2024-04-23T08:16:02+0000 age=51m31s
WARN [04-23|09:07:33.612] Unclean shutdown detected booted=2024-04-23T08:40:28+0000 age=27m5s
WARN [04-23|09:07:33.612] Unclean shutdown detected booted=2024-04-23T08:44:13+0000 age=23m20s
https://docs.optimism.io/builders/chain-operators/management/operations#stop-op-geth
There may be some kind of corruption in the database, see this link for the cause.
One suggestion is to always keep op-geth and op-node running one-to-one.
Synchronous mode is usually very fast, it is also recommended that you delete the database (geth folder) and start from scratch
@opfocus seems like its the root issue. But was wondering what's the cause, as on mainnet geth is able to recover itself in such cases.
always keep op-geth and op-node running one-to-one
Do you mean you need to restart both op-node and op-geth, if for example you only need to restart one of them?
Following the startup sequence in the documentation will reduce error messages in the log. But I don't mean to say that it must be done. The original text of the document is: Always run op-node and op-geth in a one-to-one configuration. Don't run multiple op-geth instances behind one op-node, or vice versa.
I am encountering the same issue:
INFO [05-03|13:32:51.975] Starting geth on an OP network... network=op-mainnet
INFO [05-03|13:32:51.975] Bumping default cache on mainnet provided=1024 updated=4096
INFO [05-03|13:32:51.976] Maximum peer count ETH=100 total=100
INFO [05-03|13:32:51.977] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-03|13:32:51.981] The flag --txlookuplimit is deprecated and will be removed, please use --history.transactions
INFO [05-03|13:32:51.981] Set global gas cap cap=50,000,000
INFO [05-03|13:32:51.982] Initializing the KZG library backend=gokzg
INFO [05-03|13:32:52.002] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB
INFO [05-03|13:32:52.139] Using pebble as the backing database
INFO [05-03|13:32:52.139] Allocated cache and file handles database=/opt/data/geth/chaindata cache=2.00GiB handles=524,288
INFO [05-03|13:32:55.216] Opened ancient database database=/opt/data/geth/chaindata/ancient/chain readonly=false
INFO [05-03|13:32:55.228] State schema set to default scheme=hash
INFO [05-03|13:32:55.849]
INFO [05-03|13:32:55.849] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [05-03|13:32:55.849] Chain ID: 10 (OP-Mainnet)
INFO [05-03|13:32:55.849] Consensus: Optimism
INFO [05-03|13:32:55.849]
INFO [05-03|13:32:55.849] Pre-Merge hard forks (block based):
INFO [05-03|13:32:55.849] - Homestead: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [05-03|13:32:55.849] - Tangerine Whistle (EIP 150): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [05-03|13:32:55.849] - Spurious Dragon/1 (EIP 155): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [05-03|13:32:55.849] - Spurious Dragon/2 (EIP 158): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [05-03|13:32:55.849] - Byzantium: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [05-03|13:32:55.849] - Constantinople: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [05-03|13:32:55.849] - Petersburg: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [05-03|13:32:55.849] - Istanbul: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [05-03|13:32:55.849] - Muir Glacier: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [05-03|13:32:55.849] - Berlin: #3950000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [05-03|13:32:55.849] - London: #105235063 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [05-03|13:32:55.849] - Arrow Glacier: #105235063 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [05-03|13:32:55.849] - Gray Glacier: #105235063 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [05-03|13:32:55.849]
INFO [05-03|13:32:55.849] Merge configured:
INFO [05-03|13:32:55.849] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [05-03|13:32:55.849] - Network known to be merged: true
INFO [05-03|13:32:55.849] - Total terminal difficulty: 0
INFO [05-03|13:32:55.849] - Merge netsplit block: #105235063
INFO [05-03|13:32:55.849]
INFO [05-03|13:32:55.849] Post-Merge hard forks (timestamp based):
INFO [05-03|13:32:55.849] - Shanghai: @1704992401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [05-03|13:32:55.849] - Cancun: @1710374401
INFO [05-03|13:32:55.849] - Regolith: @0
INFO [05-03|13:32:55.849] - Canyon: @1704992401
INFO [05-03|13:32:55.849] - Ecotone: @1710374401
INFO [05-03|13:32:55.849]
INFO [05-03|13:32:55.849] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [05-03|13:32:55.849]
INFO [05-03|13:32:55.852] Loaded most recent local header number=119,395,391 hash=f0d82e..e3615f td=210,470,125 age=4d2h13m
INFO [05-03|13:32:55.852] Loaded most recent local block number=119,375,892 hash=a4cc1b..6a6dae td=210,470,125 age=4d13h3m
INFO [05-03|13:32:55.852] Loaded most recent local snap block number=119,395,391 hash=f0d82e..e3615f td=210,470,125 age=4d2h13m
INFO [05-03|13:32:55.852] Loaded most recent local finalized block number=119,394,792 hash=a5ea7d..2f960d td=210,470,125 age=4d2h33m
INFO [05-03|13:32:55.852] Loaded last snap-sync pivot marker number=119,269,703
WARN [05-03|13:32:55.855] Enabling snapshot recovery chainhead=119,375,892 diskbase=119,395,112
WARN [05-03|13:32:55.892] Snapshot is not continuous with chain snaproot=3a6707..8d2fb3 chainroot=1365fd..3278ec
INFO [05-03|13:32:55.892] Initialized transaction indexer range="entire chain"
INFO [05-03|13:32:55.892] Initialising Ethereum protocol network=10 dbversion=8
INFO [05-03|13:32:55.892] Loaded local transaction journal transactions=0 dropped=0
INFO [05-03|13:32:55.892] Regenerated local transaction journal transactions=0 accounts=0
INFO [05-03|13:32:55.892] Chain post-merge, sync via beacon client
INFO [05-03|13:32:55.893] Gasprice oracle is ignoring threshold set threshold=2
WARN [05-03|13:32:55.894] Unclean shutdown detected booted=2024-04-29T11:17:57+0000 age=4d2h14m
WARN [05-03|13:32:55.895] Engine API enabled protocol=eth
INFO [05-03|13:32:55.895] Starting peer-to-peer node instance=Geth/v0.1.0-unstable/linux-amd64/go1.21.7
INFO [05-03|13:32:55.936] New local node record seq=xxxxxxxx id=xxxxxxxx ip=127.0.0.1 udp=30303 tcp=30303
INFO [05-03|13:32:55.936] Started P2P networking self=enode://xxxxxxxx
INFO [05-03|13:32:55.938] Loaded JWT secret file path=/secrets/auth_rpc_jwt_secret crc32=xxxxxxxx
INFO [05-03|13:32:55.938] HTTP server started endpoint=[::]:8545 auth=false prefix= cors=* vhosts=*
INFO [05-03|13:32:55.938] WebSocket enabled url=ws://[::]:8551
INFO [05-03|13:32:55.938] HTTP server started endpoint=[::]:8551 auth=true prefix= cors=localhost vhosts=*
INFO [05-03|13:33:06.464] Looking for peers peercount=0 tried=0 static=0
INFO [05-03|13:33:16.469] Looking for peers peercount=2 tried=162 static=0
INFO [05-03|13:33:26.828] Looking for peers peercount=2 tried=83 static=0
INFO [05-03|13:34:14.076] New local node record seq=xxxxxxxx id=xxxxxxxx ip=xxxxxxxx udp=30303 tcp=30303
INFO [05-03|13:44:21.992] Starting work on payload id=0x037240beadac8b1f
WARN [05-03|13:44:22.001] Ignoring already known beacon payload number=119,375,893 hash=b9eea1..93d66e age=4d13h14m
ERROR[05-03|13:44:22.013] Failed to create sealing context err="missing trie node 6eab2ff4bb552b28de78f1b62849bd376642a68cb45143d85d425ecf79fe832e (path ) state 0x6eab2ff4bb552b28de78f1b62849bd376642a68cb45143d85d425ecf79fe832e is not available, not found"
ERROR[05-03|13:44:22.013] Failed to build payload err="missing trie node 6eab2ff4bb552b28de78f1b62849bd376642a68cb45143d85d425ecf79fe832e (path ) state 0x6eab2ff4bb552b28de78f1b62849bd376642a68cb45143d85d425ecf79fe832e is not available, not found"
WARN [05-03|13:44:22.013] Served engine_forkchoiceUpdatedV3 reqid=45663 duration=1.577977ms err="Invalid payload attributes" errdata="{Error:missing trie node 6eab2ff4bb552b28de78f1b62849bd376642a68cb45143d85d425ecf79fe832e (path ) state 0x6eab2ff4bb552b28de78f1b62849bd376642a68cb45143d85d425ecf79fe832e is not available, not found}"
It seems the unclean shutdown caused it, however our other Geth nodes for ETH mainnet run just fine with unclean shutdowns.
Bug Description Can't sync node after restart
Steps to Reproduce Snapsync on Base Restart a few times (not sure about that steps, but eventually at some restart it happens).
even if all restarts are graceful
Node will switch to fullsync And sometimes it will never sync after thatExpected behavior Node should catch the tip after restart
Environment Information: Ubuntu 22 us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:v1.7.3 us-docker.pkg.dev/oplabs-tools-artifacts/images/op-geth:v1.101311.0
Configurations: https://github.com/smartcontracts/simple-optimism-node/blob/main/scripts/start-op-node.sh https://github.com/smartcontracts/simple-optimism-node/blob/main/scripts/start-op-geth.sh
Logs: Geth:
There are the logs of node during trie missing error
Additional context L1 rpc and beacon are pingable from inside containers, node and geth p2p ports are opened