search

ethereum-optimism / op-geth

GNU Lesser General Public License v3.0
287 stars 746 forks source link

Issues Snap Syncing OP-Mainnet with v1.101411.0-rc.1 #404

Closed jelias2 closed 2 hours ago

jelias2 commented 3 days ago

System information

Geth version: geth version CL client & version: e.g. lighthouse/nimbus/prysm@v1.0.0 OS & Version: OSX/Linux

Expected behaviour

Actual behaviour

Actual behavior, op-geth is unable to sync due to a incompatable genesis. 

....
t=2024-10-16T19:05:42+0000 lvl=info msg="Starting geth on an OP network..." network=op-mainnet
t=2024-10-16T19:05:42+0000 lvl=info msg="Enabling metrics collection"
t=2024-10-16T19:05:42+0000 lvl=info msg="Enabling stand-alone metrics HTTP endpoint" address=0.0.0.0:6060
t=2024-10-16T19:05:42+0000 lvl=info msg="Starting metrics server" addr=http://0.0.0.0:6060/debug/metrics
t=2024-10-16T19:05:42+0000 lvl=info msg="Maximum peer count" ETH=30 total=30
t=2024-10-16T19:05:42+0000 lvl=info msg="Smartcard socket not found, disabling" err="stat /run/pcscd/pcscd.comm: no such file or directory"
t=2024-10-16T19:05:42+0000 lvl=info msg="Using pebble as db engine"
t=2024-10-16T19:05:42+0000 lvl=debug msg="FS scan times" list=17.822µs set=315ns diff=776ns
t=2024-10-16T19:05:42+0000 lvl=debug msg="Sanitizing Go's GC trigger" percent=20
t=2024-10-16T19:05:42+0000 lvl=info msg="Enabling recording of key preimages since archive mode is used"
t=2024-10-16T19:05:42+0000 lvl=warn msg="Disabled transaction unindexing for archive node"
t=2024-10-16T19:05:42+0000 lvl=warn msg="Forcing hash state-scheme for archive mode"
t=2024-10-16T19:05:42+0000 lvl=info msg="Set global gas cap" cap=50000000
t=2024-10-16T19:05:42+0000 lvl=info msg="Initializing the KZG library" backend=gokzg
t=2024-10-16T19:05:42+0000 lvl=info msg="Allocated trie memory caches" clean="2.40 GiB" dirty="0.00 B"
t=2024-10-16T19:05:42+0000 lvl=info msg="Using pebble as the backing database"
t=2024-10-16T19:05:42+0000 lvl=info msg="Allocated cache and file handles" database=/db/geth/chaindata cache="4.00 GiB" handles=524288
t=2024-10-16T19:05:42+0000 lvl=debug msg="Chain freezer table opened" database=/db/geth/chaindata/ancient/chain table=headers items=0 size="0.00 B"
t=2024-10-16T19:05:42+0000 lvl=debug msg="Chain freezer table opened" database=/db/geth/chaindata/ancient/chain table=hashes items=0 size="0.00 B"
t=2024-10-16T19:05:42+0000 lvl=debug msg="Chain freezer table opened" database=/db/geth/chaindata/ancient/chain table=bodies items=0 size="0.00 B"
t=2024-10-16T19:05:42+0000 lvl=debug msg="Chain freezer table opened" database=/db/geth/chaindata/ancient/chain table=receipts items=0 size="0.00 B"
t=2024-10-16T19:05:42+0000 lvl=debug msg="Chain freezer table opened" database=/db/geth/chaindata/ancient/chain table=diffs items=0 size="0.00 B"
t=2024-10-16T19:05:42+0000 lvl=info msg="Opened ancient database" database=/db/geth/chaindata/ancient/chain readonly=false
t=2024-10-16T19:05:42+0000 lvl=info msg="State scheme set by user" scheme=hash
t=2024-10-16T19:05:42+0000 lvl=debug msg="Current full block not old enough to freeze" err="freezing threshold is not available"
Fatal: Failed to register the Ethereum service: database contains incompatible genesis (have 2b955e7dd8df0ba8a50ee3e5c79065be9a370b1e52398694001067d40f7d6731, new 7ca38a1916c42007829c55e69d3e9a73265554b586a499015373241b8a3fa48b)
Fatal: Failed to register the Ethereum service: database contains incompatible genesis (have 2b955e7dd8df0ba8a50ee3e5c79065be9a370b1e52398694001067d40f7d6731, new 7ca38a1916c42007829c55e69d3e9a73265554b586a499015373241b8a3fa48b)

Steps to reproduce the behaviour

make geth
mkdir tmp        
./build/bin/geth --datadir=./temp --op-network=op-mainnet
./build/bin/geth --datadir=./temp --op-network=op-mainnet

INFO [10-18|11:59:58.521] Starting geth on an OP network...        network=op-mainnet
INFO [10-18|11:59:58.521] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [10-18|11:59:58.521] Maximum peer count                       ETH=50 total=50
INFO [10-18|11:59:58.524] Set global gas cap                       cap=50,000,000
INFO [10-18|11:59:58.525] Initializing the KZG library             backend=gokzg
INFO [10-18|11:59:58.541] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [10-18|11:59:58.541] Defaulting to pebble as the backing database
INFO [10-18|11:59:58.541] Allocated cache and file handles         database=/Users/jacob/workspace/ethereum-optimism/op-geth/temp/geth/chaindata cache=2.00GiB handles=5120
INFO [10-18|11:59:58.631] Opened ancient database                  database=/Users/jacob/workspace/ethereum-optimism/op-geth/temp/geth/chaindata/ancient/chain readonly=false
INFO [10-18|11:59:58.631] State schema set to default              scheme=hash
INFO [10-18|11:59:58.636] Writing custom genesis block
INFO [10-18|11:59:58.680] 
INFO [10-18|11:59:58.680] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [10-18|11:59:58.680] Chain ID:  10 (OP-Mainnet)
...
INFO [10-18|11:59:58.681] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [10-18|11:59:58.681] 
#### Below is the unexpected log
INFO [10-18|11:59:58.681] Loaded most recent local block           number=0 hash=7ca38a..3fa48b td=1 age=55y7mo4d
INFO [10-18|11:59:58.681] Genesis state is missing, wait state sync
WARN [10-18|11:59:58.681] Failed to load snapshot                  err="missing or corrupted snapshot"
########## End 
INFO [10-18|11:59:58.689] Rebuilding state snapshot
INFO [10-18|11:59:58.693] Initialized transaction indexer          range="last 2350000 blocks"
INFO [10-18|11:59:58.693] Initialising Ethereum protocol           network=10         dbversion=<nil>
INFO [10-18|11:59:58.693] Resuming state snapshot generation       root=eddb4c..a568eb accounts=0 slots=0 storage=0.00B dangling=0 elapsed=3.983ms
INFO [10-18|11:59:58.693] Trie missing, state snapshotting paused  root=eddb4c..a568eb accounts=0 slots=0 storage=0.00B dangling=0 elapsed=4.089ms
INFO [10-18|11:59:58.697] Entered PoS stage
INFO [10-18|11:59:58.697] Enabled snap sync                        head=0 hash=7ca38a..3fa48b
INFO [10-18|11:59:58.698] Chain post-merge, sync via beacon client
INFO [10-18|11:59:58.698] Gasprice oracle is ignoring threshold set threshold=2
WARN [10-18|11:59:58.705] Engine API enabled                       protocol=eth
INFO [10-18|11:59:58.705] Starting peer-to-peer node               instance=Geth/v1.101315.2-stable-7c281983/darwin-arm64/go1.22.8
INFO [10-18|11:59:58.740] New local node record                    seq=1,729,270,798,740 id=19644f57c253c7d4 ip=127.0.0.1 udp=30303 tcp=30303
INFO [10-18|11:59:58.741] Started P2P networking                   self=enode://a0289b00c66742231eb6859e7a50e6f26973b506a5bb39eaed18f846edaf2a11e2893115e33fd6809e7628bb7ade5325ece167e6ae4108990655be0eb44bbcd2@127.0.0.1:30303
INFO [10-18|11:59:58.741] IPC endpoint opened                      url=/Users/jacob/workspace/ethereum-optimism/op-geth/temp/geth.ipc
INFO [10-18|11:59:58.741] Generated JWT secret                     path=/Users/jacob/workspace/ethereum-optimism/op-geth/temp/geth/jwtsecret
INFO [10-18|11:59:58.742] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [10-18|11:59:58.742] HTTP server started                      endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost
INFO [10-18|12:00:08.925] Looking for peers                        peercount=2 tried=246 static=0
INFO [10-18|12:00:18.989] Looking for peers                        peercount=1 tried=85  static=0
INFO [10-18|12:00:29.123] Looking for peers                        peercount=1 tried=72  static=0
protolambda commented 3 days ago

0x2b955e7dd8df0ba8a50ee3e5c79065be9a370b1e52398694001067d40f7d6731 is not a valid genesis hash. It prints it (without 0x, loaded from the canonical-hash data, which is part of the freezer-DB).

The correct hash for block 0 of op-mainnet is 0x7ca38a1916c42007829c55e69d3e9a73265554b586a499015373241b8a3fa48b, which it correctly produced from the genesis, which it loaded from the config and block header RLP.

So this means that the canonical block-hash store is corrupted in some way. And the 0x2b hash doesn't seem to be a real block on a known network like L1 ethereum, op mainnet, op-sepolia, or base mainnet. So I wonder where that's coming from.

I tested it here: https://gist.github.com/protolambda/032b2a289b3c222cced7c8952a5b6959

And tried to modify the header/config to try and reproduce the modification needed for the 0x2b hash, but no luck.

protolambda commented 3 days ago

Output of above test:

    genesis_optimism_test.go:98: uncles root: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
    genesis_optimism_test.go:99: receipts root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
    genesis_optimism_test.go:100: txs root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
    genesis_optimism_test.go:101: state root: 0xeddb4c1786789419153a27c4c80ff44a2226b6eda04f7e22ce5bae892ea568eb
    genesis_optimism_test.go:105: got hash: 0x7ca38a1916c42007829c55e69d3e9a73265554b586a499015373241b8a3fa48b
    genesis_optimism_test.go:106: expected hash: 0x7ca38a1916c42007829c55e69d3e9a73265554b586a499015373241b8a3fa48b

That eddb does look like the correct state for genesis. The syncing op-geth node should not be generating a snapshot for block 0 however. It should not try snapshot, and/or fail in some way, until it has snap-synced data from another node, which allows it to connect the header-chain, but go straight to a later full state. And thus never bother with a pre-bedrock state snapshot.

protolambda commented 3 days ago

Below is the unexpected log

The logs below it are expected. If starting from just the superchain-registry, it won't have a state to work with, and needs to snap-sync first. It won't be able to produce a state snapshot for the genesis it started from, as no accounts state was included in it.

protolambda commented 3 days ago

Reproduced the unknown hash! It's the same op-mainnet block-header, but with the state-root set to 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 (the root of an empty MPT tree).

So along the way it failed to handle the no-allocs, and assumed an empty state instead, and then created a header and wrote it to the DB. I'll review the initialization code path further, to see how that can happen.

protolambda commented 3 days ago

Reproduced it, and found the issue: it's running Commit() on the genesis spec, which doesn't use the same state-root computation path as ToBlock() anymore.

And here are the logs after the fix:

..... --ws --ws.port=8946 --ws.addr=0.0.0.0 --ws.origins=* --ws.api=eth,net,debug,admin,txpool --http --http.port=8945 --http.addr=0.0.0.0 --http.vhosts=* --http.corsdomain=* --http.api=eth,net,debug,admin,txpool --authrpc.addr=localhost --authrpc.jwtsecret=/home/proto/projects/devnet-deploy/keys/op-goerli-sync/engine_jwt_secret.txt --authrpc.port=8951 --authrpc.vhosts=* --datadir=/home/proto/projects/op-mainnet-temp/gethdata --verbosity=4 --rollup.disabletxpoolgossip=true --rollup.sequencerhttp=https://mainnet-sequencer.optimism.io --nodiscover --maxpeers=1 --verbosity=4 --log.vmodule=snap=5,p2p=5 --metrics --metrics.addr=0.0.0.0 --metrics.port=6970 --op-network=op-mainnet --syncmode=snap --port=4900
API server listening at: 127.0.0.1:39607
INFO [10-18|20:28:14.905] Starting geth on an OP network...        network=op-mainnet
INFO [10-18|20:28:14.905] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [10-18|20:28:14.905] Enabling metrics collection
INFO [10-18|20:28:14.905] Enabling stand-alone metrics HTTP endpoint address=0.0.0.0:6970
INFO [10-18|20:28:14.905] Starting metrics server                  addr=http://0.0.0.0:6970/debug/metrics
INFO [10-18|20:28:14.906] Maximum peer count                       ETH=1 total=1
INFO [10-18|20:28:14.908] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
DEBUG[10-18|20:28:14.909] FS scan times                            list="15.399µs" set=461ns diff=921ns
DEBUG[10-18|20:28:14.911] Sanitizing Go's GC trigger               percent=25
INFO [10-18|20:28:14.913] Set global gas cap                       cap=50,000,000
INFO [10-18|20:28:14.914] Initializing the KZG library             backend=gokzg
INFO [10-18|20:28:15.011] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [10-18|20:28:15.011] Defaulting to pebble as the backing database
INFO [10-18|20:28:15.011] Allocated cache and file handles         database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata cache=2.00GiB handles=5,000,000
DEBUG[10-18|20:28:15.078] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/chain table=headers items=0 size=0.00B
DEBUG[10-18|20:28:15.090] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/chain table=hashes items=0 size=0.00B
DEBUG[10-18|20:28:15.103] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/chain table=bodies items=0 size=0.00B
DEBUG[10-18|20:28:15.115] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/chain table=receipts items=0 size=0.00B
DEBUG[10-18|20:28:15.128] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/chain table=diffs items=0 size=0.00B
INFO [10-18|20:28:15.128] Opened ancient database                  database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/chain readonly=false
INFO [10-18|20:28:15.129] State schema set to default              scheme=path
ERROR[10-18|20:28:15.129] Head block is not reachable
DEBUG[10-18|20:28:15.129] Current full block not old enough to freeze err="freezing threshold is not available"
WARN [10-18|20:28:15.135] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
DEBUG[10-18|20:28:15.148] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/state table=history.meta items=0 size=0.00B
DEBUG[10-18|20:28:15.161] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/state table=account.index items=0 size=0.00B
DEBUG[10-18|20:28:15.173] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/state table=storage.index items=0 size=0.00B
DEBUG[10-18|20:28:15.186] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/state table=account.data items=0 size=0.00B
DEBUG[10-18|20:28:15.198] Chain freezer table opened               database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/state table=storage.data items=0 size=0.00B
INFO [10-18|20:28:15.198] Opened ancient database                  database=/home/proto/projects/op-mainnet-temp/gethdata/geth/chaindata/ancient/state readonly=false
INFO [10-18|20:28:15.198] Writing custom genesis block
INFO [10-18|20:28:23.624] 
INFO [10-18|20:28:23.624] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [10-18|20:28:23.624] Chain ID:  10 (OP Mainnet)
INFO [10-18|20:28:23.624] Consensus: Optimism
INFO [10-18|20:28:23.624] 
INFO [10-18|20:28:23.624] Pre-Merge hard forks (block based):
INFO [10-18|20:28:23.624]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [10-18|20:28:23.624]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [10-18|20:28:23.624]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [10-18|20:28:23.624]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [10-18|20:28:23.624]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [10-18|20:28:23.624]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [10-18|20:28:23.624]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [10-18|20:28:23.624]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [10-18|20:28:23.624]  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [10-18|20:28:23.624]  - Berlin:                      #3950000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [10-18|20:28:23.624]  - London:                      #105235063 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [10-18|20:28:23.624]  - Arrow Glacier:               #105235063 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [10-18|20:28:23.624]  - Gray Glacier:                #105235063 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [10-18|20:28:23.624] 
INFO [10-18|20:28:23.624] Merge configured:
INFO [10-18|20:28:23.624]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [10-18|20:28:23.624]  - Network known to be merged: true
INFO [10-18|20:28:23.624]  - Total terminal difficulty:  0
INFO [10-18|20:28:23.624]  - Merge netsplit block:       #105235063
INFO [10-18|20:28:23.624] 
INFO [10-18|20:28:23.624] Post-Merge hard forks (timestamp based):
INFO [10-18|20:28:23.624]  - Shanghai:                    @1704992401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [10-18|20:28:23.624]  - Cancun:                      @1710374401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [10-18|20:28:23.624]  - Regolith:                    @0         
INFO [10-18|20:28:23.624]  - Canyon:                      @1704992401
INFO [10-18|20:28:23.624]  - Ecotone:                     @1710374401
INFO [10-18|20:28:23.624]  - Fjord:                       @1720627201
INFO [10-18|20:28:23.624]  - Granite:                     @1726070401
INFO [10-18|20:28:23.624] 
INFO [10-18|20:28:23.624] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [10-18|20:28:23.624] 
INFO [10-18|20:28:23.625] Loaded most recent local block           number=0 hash=7ca38a..3fa48b td=1 age=55y7mo4d
INFO [10-18|20:28:23.625] Genesis state is missing, wait state sync
WARN [10-18|20:28:23.625] Failed to load snapshot                  err="missing or corrupted snapshot"
INFO [10-18|20:28:23.637] Rebuilding state snapshot
DEBUG[10-18|20:28:23.637] Journalled generator progress            progress=empty
DEBUG[10-18|20:28:23.643] Start snapshot generation                root=eddb4c..a568eb
INFO [10-18|20:28:23.643] Initialized transaction indexer          range="last 2350000 blocks"
INFO [10-18|20:28:23.643] Initialising Ethereum protocol           network=10         dbversion=<nil>
INFO [10-18|20:28:23.643] Resuming state snapshot generation       root=eddb4c..a568eb accounts=0 slots=0 storage=0.00B dangling=0 elapsed=6.483ms
DEBUG[10-18|20:28:23.644] Regenerated local transaction journal    transactions=0 accounts=0
INFO [10-18|20:28:23.644] Trie missing, state snapshotting paused  root=eddb4c..a568eb accounts=0 slots=0 storage=0.00B dangling=0 elapsed=6.884ms
INFO [10-18|20:28:23.644] Enabled snap sync                        head=0 hash=7ca38a..3fa48b
INFO [10-18|20:28:23.644] Gasprice oracle is ignoring threshold set threshold=2
WARN [10-18|20:28:23.652] Engine API enabled                       protocol=eth
INFO [10-18|20:28:23.652] Starting peer-to-peer node               instance=Geth/vuntagged-ed3077df-20241018/linux-amd64/go1.23.2
DEBUG[10-18|20:28:23.678] TCP listener up                          addr=[::]:4900
DEBUG[10-18|20:28:23.679] IPCs registered                          namespaces=admin,debug,web3,eth,txpool,miner,net,engine
INFO [10-18|20:28:23.679] IPC endpoint opened                      url=/home/proto/projects/op-mainnet-temp/gethdata/geth.ipc
DEBUG[10-18|20:28:23.679] Allowed origin(s) for WS RPC interface [*]
INFO [10-18|20:28:23.680] Loaded JWT secret file                   path=/home/proto/projects/devnet-deploy/keys/op-goerli-sync/engine_jwt_secret.txt crc32=0x18867ff7
INFO [10-18|20:28:23.680] New local node record                    seq=1,729,276,103,678 id=8a248c0817ab1690 ip=127.0.0.1 udp=0 tcp=4900
INFO [10-18|20:28:23.680] Started P2P networking                   self="enode://7443a90b9d1d92666d423d5988c451e93426ddeccd4f5cf35bc554f2ce6a9884407d7ed0fa69a423fdde80bb691be1de0d7841d20f69cb6199a4518c897930b0@127.0.0.1:4900?discport=0"
DEBUG[10-18|20:28:23.680] Allowed origin(s) for WS RPC interface [localhost]
INFO [10-18|20:28:23.680] HTTP server started                      endpoint=[::]:8945 auth=false prefix= cors=* vhosts=*
INFO [10-18|20:28:23.680] WebSocket enabled                        url=ws://[::]:8946
INFO [10-18|20:28:23.681] WebSocket enabled                        url=ws://127.0.0.1:8951
INFO [10-18|20:28:23.681] HTTP server started                      endpoint=127.0.0.1:8951 auth=true  prefix= cors=localhost vhosts=*
DEBUG[10-18|20:28:25.894] External IP changed                      ip=92.251.117.139 interface="UPNP IGDv1-IP1"
TRACE[10-18|20:28:25.894] Attempting port mapping                  proto=TCP extport=4900 intport=4900 interface="UPNP IGDv1-IP1"
INFO [10-18|20:28:26.349] NAT mapped port                          proto=TCP extport=4900 intport=4900 interface="UPNP IGDv1-IP1"
INFO [10-18|20:28:46.771] Got interrupt, shutting down...
DEBUG[10-18|20:28:46.771] RPC server shutting down
INFO [10-18|20:28:46.771] HTTP server stopped                      endpoint=[::]:8945
DEBUG[10-18|20:28:46.771] RPC server shutting down
INFO [10-18|20:28:46.771] HTTP server stopped                      endpoint=[::]:8946
DEBUG[10-18|20:28:46.771] RPC server shutting down
DEBUG[10-18|20:28:46.771] RPC server shutting down
INFO [10-18|20:28:46.771] HTTP server stopped                      endpoint=127.0.0.1:8951
DEBUG[10-18|20:28:46.771] RPC server shutting down
INFO [10-18|20:28:46.771] IPC endpoint closed                      url=/home/proto/projects/op-mainnet-temp/gethdata/geth.ipc
DEBUG[10-18|20:28:46.771] RPC server shutting down
INFO [10-18|20:28:46.771] Ethereum protocol stopped
DEBUG[10-18|20:28:46.771] RPC connection read error                err=EOF
INFO [10-18|20:28:46.771] Transaction pool stopped
INFO [10-18|20:28:46.771] Journalling in-progress snapshot         root=eddb4c..a568eb accounts=0 slots=0 storage=0.00B dangling=0 elapsed=23.134s
DEBUG[10-18|20:28:46.771] Journalled generator progress            progress=empty
DEBUG[10-18|20:28:46.778] Journalled disk layer                    root=eddb4c..a568eb
INFO [10-18|20:28:46.784] Failed to journal in-memory trie nodes   err="triedb layer [0xeddb4c1786789419153a27c4c80ff44a2226b6eda04f7e22ce5bae892ea568eb] missing"
INFO [10-18|20:28:46.789] Blockchain stopped
DEBUG[10-18|20:28:46.806] Read error                               err="accept tcp [::]:4900: use of closed network connection"
TRACE[10-18|20:28:46.806] P2P networking is spinning down
DEBUG[10-18|20:28:46.806] Deleting port mapping                    proto=TCP extport=4900 intport=4900 interface="UPNP IGDv1-IP1"

Debugger finished with the exit code 0