search

mantlenetworkio / mantle-v2

MIT License
24 stars 17 forks source link

Mantle Sepolia Stops Syncing - "Safe block not in canonical chain" #177

Closed andreclaro closed 4 months ago

andreclaro commented 4 months ago

Summary

Mantle sepolia stops processing new blocks with warnings:

Logs

op-geth logs:

Jul 11 09:02:38 t-mantle-sepolia-02 geth[378456]: INFO [07-11|09:02:38.128] Looking for peers                        peercount=2 tried=339 static=0
Jul 11 09:02:41 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:41.749] Safe block not in canonical chain
Jul 11 09:02:41 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:41.749] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39725 duration="291.253µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"
Jul 11 09:02:45 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:45.562] Safe block not in canonical chain
Jul 11 09:02:45 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:45.562] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39729 duration="291.483µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"
Jul 11 09:02:48 t-mantle-sepolia-02 geth[378456]: INFO [07-11|09:02:48.181] Looking for peers                        peercount=2 tried=404 static=0
Jul 11 09:02:49 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:49.621] Safe block not in canonical chain
Jul 11 09:02:49 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:49.621] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39733 duration="406.948µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"
Jul 11 09:02:53 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:53.553] Safe block not in canonical chain
Jul 11 09:02:53 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:53.554] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39737 duration="915.132µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"
Jul 11 09:02:55 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:55.660] Safe block not in canonical chain
Jul 11 09:02:55 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:55.660] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39741 duration="269.662µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"
Jul 11 09:02:58 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:58.107] Safe block not in canonical chain
Jul 11 09:02:58 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:02:58.107] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39745 duration="287.993µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"
Jul 11 09:02:58 t-mantle-sepolia-02 geth[378456]: INFO [07-11|09:02:58.203] Looking for peers                        peercount=2 tried=342 static=0
Jul 11 09:03:01 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:03:01.573] Safe block not in canonical chain
Jul 11 09:03:01 t-mantle-sepolia-02 geth[378456]: WARN [07-11|09:03:01.573] Served engine_forkchoiceUpdatedV1        conn=127.0.0.1:45482 reqid=39749 duration="272.833µs"  err="Invalid forkchoice state" errdata="{Error:safe block not in canonical chain}"

op-node logs:

Jul 11 09:03:07 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:07+0000 lvl=warn msg="Failed to share forkchoice-updated signal" state="&{HeadBlockHash:0x7cba2637a0465e0f3b397565a0afd6b0309a2b941d7dc43ba14e8e13b1761660 SafeBlockHash:0xd259b7bcfaa611336d48f7a797b04a9b82675279768f7e5576f515102b142e6e FinalizedBlockHash:0xf9f716a940d7f82e45f475353fbaefb38d006cf2a7f745854ef8ce681c9c87b2}" attr=<nil> err="Invalid forkchoice state"
Jul 11 09:03:07 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:07+0000 lvl=warn msg="Derivation pipeline is reset"           err="engine stage failed: reset: forkchoice update was inconsistent with engine, need reset to resolve: Invalid forkchoice state"
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0xb056dae5bb2373865c5b451d7d9c54ea103856dff3383a2ed81f1f073fcd5205:9571958 peer=16Uiu2HAmKXpCW586g24LvFytdgexC6rTkzVrJr6C2JEoho8t5vgx
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xb056dae5bb2373865c5b451d7d9c54ea103856dff3383a2ed81f1f073fcd5205:9571958
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="connected to peer"                      peer=16Uiu2HAmHC7FqenCYDaJr8H6yRMAt5e3NZRtf9uUFuwZz2FddP5M addr=/ip4/65.21.136.102/tcp/54644
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="disconnected from peer"                 peer=16Uiu2HAmHC7FqenCYDaJr8H6yRMAt5e3NZRtf9uUFuwZz2FddP5M addr=/ip4/65.21.136.102/tcp/54644
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="connected to peer"                      peer=16Uiu2HAm5KEMvvPuCtqs85HZW2JN5QpTWv6AEz8iMi1dULSw8Rzo addr=/ip4/83.150.2.27/tcp/37771
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Loaded current L2 heads"                unsafe=0x7cba2637a0465e0f3b397565a0afd6b0309a2b941d7dc43ba14e8e13b1761660:9555604 safe=0xbcf68bcbb400f2333efdcf3efcfa2ec957743783e07d9b059dc9e3d59024a10c:9525402 finalized=0xf9f716a940d7f82e45f475353fbaefb38d006cf2a7f745854ef8ce681c9c87b2:9525399 unsafe_origin=0x222c4ebaa9bab9fc909d4f8dbc51f967f5fc692c8a4578526609e1491cacf11c:6286263 safe_origin=0xe025a5fdb5821c681df156822a2260dd0bb39b6f3bc75472014550571e2946ec:6281664
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Walking back L1Block by number"         curr=0x222c4ebaa9bab9fc909d4f8dbc51f967f5fc692c8a4578526609e1491cacf11c:6286263 next=0x222c4ebaa9bab9fc909d4f8dbc51f967f5fc692c8a4578526609e1491cacf11c:6286263 l2block=0x7cba2637a0465e0f3b397565a0afd6b0309a2b941d7dc43ba14e8e13b1761660:9555604
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Found highest L2 block with canonical L1 origin. Skip further sanity check and jump to the safe head"
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Walking back L1Block by number"         curr=0xe025a5fdb5821c681df156822a2260dd0bb39b6f3bc75472014550571e2946ec:6281664 next=0xe025a5fdb5821c681df156822a2260dd0bb39b6f3bc75472014550571e2946ec:6281664 l2block=0xbcf68bcbb400f2333efdcf3efcfa2ec957743783e07d9b059dc9e3d59024a10c:9525402
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Sync progress"                          reason="reset derivation work"                l2_finalized=0xf9f716a940d7f82e45f475353fbaefb38d006cf2a7f745854ef8ce681c9c87b2:9525399 l2_safe=0xd259b7bcfaa611336d48f7a797b04a9b82675279768f7e5576f515102b142e6e:9525401 l2_unsafe=0x7cba2637a0465e0f3b397565a0afd6b0309a2b941d7dc43ba14e8e13b1761660:9555604 l2_engineSyncTarget=0x7cba2637a0465e0f3b397565a0afd6b0309a2b941d7dc43ba14e8e13b1761660:9555604 l2_time=1,720,655,881 l1_derived=0x5542e3f55c519eb3a4c8cd2fbdb9a13022ab3f947d00b8264dd48c43ee4e6d13:6281363
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="completed reset of derivation pipeline" origin=0x5542e3f55c519eb3a4c8cd2fbdb9a13022ab3f947d00b8264dd48c43ee4e6d13:6281363
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Derived by Eigenda da"                  EigenDaUpgradeHeight=9,525,400 safeL2Ref=0xd259b7bcfaa611336d48f7a797b04a9b82675279768f7e5576f515102b142e6e:9525401 l1InBoxBlock=0x5542e3f55c519eb3a4c8cd2fbdb9a13022ab3f947d00b8264dd48c43ee4e6d13:6281363
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="get data from eigenda"                  size=0    blobHashes=[]
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="Reset of L1Retrieval done"              origin=0x5542e3f55c519eb3a4c8cd2fbdb9a13022ab3f947d00b8264dd48c43ee4e6d13:6281363
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=warn msg="Failed to share forkchoice-updated signal" state="&{HeadBlockHash:0x7cba2637a0465e0f3b397565a0afd6b0309a2b941d7dc43ba14e8e13b1761660 SafeBlockHash:0xd259b7bcfaa611336d48f7a797b04a9b82675279768f7e5576f515102b142e6e FinalizedBlockHash:0xf9f716a940d7f82e45f475353fbaefb38d006cf2a7f745854ef8ce681c9c87b2}" attr=<nil> err="Invalid forkchoice state"
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=warn msg="Derivation pipeline is reset"           err="engine stage failed: reset: forkchoice update was inconsistent with engine, need reset to resolve: Invalid forkchoice state"
Jul 11 09:03:09 t-mantle-sepolia-02 op-node[378507]: t=2024-07-11T09:03:09+0000 lvl=info msg="disconnected from peer"                 peer=16Uiu2HAm5KEMvvPuCtqs85HZW2JN5QpTWv6AEz8iMi1dULSw8Rzo addr=/ip4/83.150.2.27/tcp/37771

CLI

op-geth:

/usr/local/bin/geth \
    --datadir=/var/lib/mantle/data/execution \
    --networkid=5003 \
    --nat=extip:0.0.0.0 \
    --syncmode=full \
    --gcmode=archive \
    --rollup.sequencerhttp=https://rpc.sepolia.mantle.xyz \
    --rpc.allow-unprotected-txs \
    --http \
    --http.addr=0.0.0.0 \
    --http.port=8545 \
    --http.corsdomain="*" \
    --http.vhosts="*" \
    --http.api=eth,web3 \
    --ws \
    --ws.addr=0.0.0.0 \
    --ws.port=8546 \
    --ws.origins="*" \
    --ws.api=eth,web3 \
# Engine API is meant only for communication between the Execution client and
# the Consensus client, both of which live in the same node.
    --authrpc.addr=127.0.0.1 \
    --authrpc.vhosts="*" \
    --authrpc.port=8551 \
    --authrpc.jwtsecret=/var/lib/mantle/jwt.hex \
    --metrics \
    --metrics.addr=0.0.0.0 \
    --metrics.port=9001 \
    --snapshot=false \
    --verbosity=3

op-node:

/usr/local/bin/op-node \
    --l1=http://t-eth-sepolia:8545 \
    --l2=http://127.0.0.1:8551 \
    --l2.jwt-secret=/var/lib/mantle/jwt.hex \
    --l2.backup-unsafe-sync-rpc=https://rpc.sepolia.mantle.xyz \
    --l2.engine-sync=true \
    --l2.skip-sync-start-check=true \
    --rollup.config=/var/lib/mantle/data/consensus/rollup.json \
    --rpc.addr=0.0.0.0 \
    --rpc.port=9545 \
# Read the hex-encoded 32-byte private key for the peer ID from this txt file.
# Created if not already exists. Important to persist to keep the same network identity after restarting, maintaining the previous advertised identity.
    --p2p.priv.path=/var/lib/mantle/data/consensus/priv.key \
# The port to listen on for p2p connections.
# UDP port defaults to be the same as TCP port, but specifying the port is a good idea.
# Both UDP and TCP ports will be advertised to other nodes, so they should be open to the public.
    --p2p.listen.tcp=30304 \
    --p2p.listen.udp=30304 \
    --p2p.peerstore.path=/var/lib/mantle/data/consensus/peerstore \
    --p2p.discovery.path=/var/lib/mantle/data/consensus/discovery \
    --p2p.static=/ip4/18.139.118.7/tcp/9003/p2p/16Uiu2HAkywYkvLRUH2MXbD6tSoT3jSMAzTWTp1aDijxpXQXxG6VM \
    --p2p.listen.ip=0.0.0.0 \
    --da.indexer-enable=true \
    --da.indexer-socket=da-indexer-api.sepolia.mantle.xyz:80 \
# Enable metrics.
    --metrics.enabled=true \
# Metrics listen on all interfaces by default, but specifying the address is a good idea.
    --metrics.addr=0.0.0.0 \
    --metrics.port=9002

Versions

andreclaro commented 4 months ago

Restarting the services solved the issue but IMO the service should be able to recover without intervention...

andreclaro commented 4 months ago

Node got now stuck on block "9562664" with the same error... restart mitigates the issue... but there is a problem here that needs to be investigated..

andreclaro commented 4 months ago

It happened again at block 9567724...

andreclaro commented 4 months ago

To be clear, we are using "v1.0.1" tags from: 

# op-node --version
op-node version v0.0.0-44178945-1720062434

# geth version
Geth
Version: 0.1.0-unstable
Git Commit: 80eb718e9810d98366f58151fe72a546ca8b346b
Upstream Version: 1.11.5-stable
Architecture: amd64
Go Version: go1.20.14
Operating System: linux
pandainzoo commented 4 months ago

thanks for reply, try to modify the following config for op-node, then restart op-geth & op-node, see if the problem can be solved

andreclaro commented 4 months ago

I have updated op-node with the following flags:

    --l2.engine-sync=false \
    --l2.skip-sync-start-check=false \
    --p2p.sync.req-resp=false \

Now I'm seeing logs like:

INFO [07-11|11:06:35.028] Imported new potential chain segment     number=9,527,207 hash=6da5c5..7f717b blocks=1 txs=1 mgas=0.064     elapsed="534.434µs"  mgasps=119.732    age=1d2h55m dirty=0.00B

age=1d2h55m... So node is importing blocks from yesterday ... is this expected?

andreclaro commented 4 months ago

in our previous mantle sepolia configuration we were passing the following flags to deal with these issue:

    # this is needed on testnet to solve the op-node error:
    # 'Retrieval frames from mantleDa error'
    --da.indexer-socket=false \
    --da.retriever-socket=retriever.sepolia.mantle.xyz:8000 \
    #  this is needed on testnet to solve the op-node error:
    # 'Query subgraph fail: unsupported protocol scheme'
    --da.graph-provider=https://graphnode.sepolia.mantle.xyz/subgraphs/name/datalayr \
andreclaro commented 4 months ago

Node is still 13 hours behind and very slowly catching up 😢 

Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.640] Imported new potential chain segment     number=9,553,786 hash=25e33c..b87456 blocks=1 txs=1 mgas=0.055     elapsed="641.332µs"  mgasps=86.172     age=13h11m40s dirty=0.00B
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.640] Chain head was updated                   number=9,553,786 hash=25e33c..b87456 root=2a99dc..a5d9e6 elapsed="94.705µs"   age=13h11m40s
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.642] Starting work on payload                 id=0x57b6208df42dc61b
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.644] Imported new potential chain segment     number=9,553,787 hash=0d41e0..409437 blocks=1 txs=1 mgas=0.064     elapsed="514.095µs"  mgasps=124.446    age=13h11m38s dirty=0.00B
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.645] Chain head was updated                   number=9,553,787 hash=0d41e0..409437 root=c0804e..91c90d elapsed="86.954µs"   age=13h11m38s
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.647] Starting work on payload                 id=0x2cab37ec5e72c42f
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.649] Imported new potential chain segment     number=9,553,788 hash=0cf855..724c5e blocks=1 txs=1 mgas=0.047     elapsed="517.405µs"  mgasps=90.600     age=13h11m36s dirty=0.00B
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.650] Chain head was updated                   number=9,553,788 hash=0cf855..724c5e root=cfdba7..c7d283 elapsed="82.214µs"   age=13h11m36s
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.652] Starting work on payload                 id=0x1f6714731cd8a4de
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.654] Imported new potential chain segment     number=9,553,789 hash=db5034..0a87d8 blocks=1 txs=1 mgas=0.047     elapsed="523.086µs"  mgasps=89.616     age=13h11m34s dirty=0.00B
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.655] Chain head was updated                   number=9,553,789 hash=db5034..0a87d8 root=99dd03..01308e elapsed="83.134µs"   age=13h11m34s
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.657] Starting work on payload                 id=0x2d2ef110eb7124c3
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.659] Imported new potential chain segment     number=9,553,790 hash=aec50f..7db95e blocks=1 txs=1 mgas=0.047     elapsed="526.536µs"  mgasps=89.029     age=13h11m32s dirty=0.00B
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.660] Chain head was updated                   number=9,553,790 hash=aec50f..7db95e root=b3c6c7..ed10c9 elapsed="78.224µs"   age=13h11m32s
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.662] Starting work on payload                 id=0x510fa554f848cf85
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.664] Imported new potential chain segment     number=9,553,791 hash=44854c..7103fa blocks=1 txs=1 mgas=0.047     elapsed="522.895µs"  mgasps=89.649     age=13h11m30s dirty=0.00B
Jul 11 12:09:05 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:05.664] Chain head was updated                   number=9,553,791 hash=44854c..7103fa root=6fe55a..cd0fcd elapsed="81.793µs"   age=13h11m30s
Jul 11 12:09:09 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:09.345] Looking for peers                        peercount=0 tried=273 static=0
Jul 11 12:09:17 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:17.626] Starting work on payload                 id=0x6e1e684afe085eb3
Jul 11 12:09:17 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:17.628] Imported new potential chain segment     number=9,553,792 hash=d0635c..3b78d9 blocks=1 txs=1 mgas=0.055     elapsed="583.179µs"  mgasps=94.765     age=13h11m40s dirty=0.00B
Jul 11 12:09:17 t-mantle-sepolia-02 geth[556004]: INFO [07-11|12:09:17.629] Chain head was updated                   number=9,553,792 hash=d0635c..3b78d9 root=84d657..3ea419 elapsed="137.496µs"  age=13h11m40s
pandainzoo commented 4 months ago

this configuration is forced to derive from L1, so the speed is slow, if the node catch up latest height, you can turn on these configuration again

in our previous mantle sepolia configuration we were passing the following flags to deal with these issue:

    # this is needed on testnet to solve the op-node error:
    # 'Retrieval frames from mantleDa error'
    --da.indexer-socket=false \
    --da.retriever-socket=retriever.sepolia.mantle.xyz:8000 \
    #  this is needed on testnet to solve the op-node error:
    # 'Query subgraph fail: unsupported protocol scheme'
    --da.graph-provider=https://graphnode.sepolia.mantle.xyz/subgraphs/name/datalayr \

we have hard fork at height 9525400 that switch DA layer from mantleDA to EigenDA on sepolia testnet, so can't use historical configuration, now should use da-indexer services to sync data refers to https://github.com/mantlenetworkio/networks/blob/main/docker-compose-sepolia-upgrade-da-indexer.yml#L106

andreclaro commented 4 months ago

@pandainzoo, Now op-geth is not processing new blocks...

and op-node is getting an error:

Jul 11 12:33:21 t-mantle-sepolia-02 op-node[579386]: t=2024-07-11T12:33:21+0000 lvl=warn msg="Derivation process temporary error"  attempts=110 err="stage 0 failed resetting: temp: failed to find the L2 Heads to start from: cannot reorg finalized block: finalized 0x6f7377e755fae74baec7e6d7a16e05f4a13f74aacbf40f7eb91913b15a2cfeb2:9553497, got: 0x0b28fe494aee3c71346dd52ade6bd0815091eb4bff352220313eddea48180d31:9553497"
pandainzoo commented 4 months ago

cannot reorg finalized block

can you show the full configuration of op-geth & op-node

andreclaro commented 4 months ago

op-geth:

Jul 11 12:38:00 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:38:00.875] Looking for peers                        peercount=0 tried=221 static=0
Jul 11 12:38:10 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:38:10.875] Looking for peers                        peercount=0 tried=408 static=0
Jul 11 12:38:20 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:38:20.888] Looking for peers                        peercount=0 tried=253 static=0
Jul 11 12:38:30 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:38:30.954] Looking for peers                        peercount=0 tried=334 static=0
Jul 11 12:38:41 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:38:41.168] Looking for peers                        peercount=0 tried=379 static=0
Jul 11 12:38:51 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:38:51.185] Looking for peers                        peercount=0 tried=301 static=0
Jul 11 12:39:01 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:39:01.243] Looking for peers                        peercount=0 tried=204 static=0
Jul 11 12:39:11 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:39:11.302] Looking for peers                        peercount=0 tried=354 static=0
Jul 11 12:39:21 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:39:21.443] Looking for peers                        peercount=0 tried=263 static=0
Jul 11 12:39:31 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:39:31.444] Looking for peers                        peercount=0 tried=357 static=0
Jul 11 12:39:41 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:39:41.473] Looking for peers                        peercount=0 tried=350 static=0
Jul 11 12:39:51 t-mantle-sepolia-02 geth[580118]: INFO [07-11|12:39:51.822] Looking for peers                        peercount=0 tried=300 static=0

op-node:

Jul 11 12:40:12 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:12+0000 lvl=info msg="attempting connection"               peer=16Uiu2HAmLkgHt5ZdS7UViqhX5S5BYPPFMiARX3A1VCb5Zr14aW9B
Jul 11 12:40:13 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:13+0000 lvl=info msg="connected to peer"                   peer=16Uiu2HAm84D6sGGAZguWpvkKnBE6DnGoDyynQJPLV5jxJGcHvu3A addr=/ip4/193.164.249.99/tcp/45668
Jul 11 12:40:13 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:13+0000 lvl=info msg="Received signed execution payload from p2p" id=0x7e73189138659c7118361bed4d2b1fac35fb367ac053620c2e0a012fbce62013:9578470 peer=16Uiu2HAmKXpCW586g24LvFytdgexC6rTkzVrJr6C2JEoho8t5vgx
Jul 11 12:40:13 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:13+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x7e73189138659c7118361bed4d2b1fac35fb367ac053620c2e0a012fbce62013:9578470
Jul 11 12:40:13 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:13+0000 lvl=info msg="disconnected from peer"              peer=16Uiu2HAm84D6sGGAZguWpvkKnBE6DnGoDyynQJPLV5jxJGcHvu3A addr=/ip4/193.164.249.99/tcp/45668
Jul 11 12:40:15 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:15+0000 lvl=info msg="Received signed execution payload from p2p" id=0xa5d099ddec87f171a79488ac90a88bc70970c2030fd59c9c294151db728b486a:9578471 peer=16Uiu2HAmKXpCW586g24LvFytdgexC6rTkzVrJr6C2JEoho8t5vgx
Jul 11 12:40:15 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:15+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xa5d099ddec87f171a79488ac90a88bc70970c2030fd59c9c294151db728b486a:9578471
Jul 11 12:40:16 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:16+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0xbb8fedffd46f6fabebfbd1a36a8fec8221cb8961175eac7d8bf4317fb2beada8:6289737 new_l1_head_parent=0x5153b78c610a5d381cf7f8b89910c7430ea2af69e2ed25ac5cc62e79decd72d4 new_l1_head=0x1e33692534802e6d1ac04e263b2d1882e711c2ee5843061879302d09285c9096:6289739
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Loaded current L2 heads"             unsafe=0x8c55f6463241d8a6976492db43f0e242d36d8b29212578fc88107c16500f19aa:9553912 safe=0x6f7377e755fae74baec7e6d7a16e05f4a13f74aacbf40f7eb91913b15a2cfeb2:9553497 finalized=0x6f7377e755fae74baec7e6d7a16e05f4a13f74aacbf40f7eb91913b15a2cfeb2:9553497 unsafe_origin=0x50e73686d178cc3cb0011f4a335e94eae67937bb4ae7eb276d4355733bedbb82:6286008 safe_origin=0xd1e4dd44c980f4083c9fc3d591b4c948fdbba326e803240a69d0a7a53b6eeddf:6285944
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by number"      curr=0x50e73686d178cc3cb0011f4a335e94eae67937bb4ae7eb276d4355733bedbb82:6286008 next=0x50e73686d178cc3cb0011f4a335e94eae67937bb4ae7eb276d4355733bedbb82:6286008 l2block=0x8c55f6463241d8a6976492db43f0e242d36d8b29212578fc88107c16500f19aa:9553912
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x50e73686d178cc3cb0011f4a335e94eae67937bb4ae7eb276d4355733bedbb82:6286008 next=0xb8b3eb874845ecc154dfbcc87481de4d1f665a5036378bf3c12db53073184146:6286007 l2block=0xd5d63fbeff13a0c9ce45e376d523ba1a6b70be52b97bf5fc4fe68121004c58e5:9553909
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xb8b3eb874845ecc154dfbcc87481de4d1f665a5036378bf3c12db53073184146:6286007 next=0xe242f69925b167a5bceaa61e5c95b12fea530a1a6bff63f7ef4c14e3a7b0ba18:6286006 l2block=0xfd5b3a7e6fa9b06466d4dbac74cb8d9f62902e336374a1b566533af396cb3699:9553903
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xe242f69925b167a5bceaa61e5c95b12fea530a1a6bff63f7ef4c14e3a7b0ba18:6286006 next=0x2e968746601a540f10ce58a9c1bd8eb222287ec0ba2beb24435bcac9b8702b7c:6286005 l2block=0xe3cf0f7f4b5600c9e0b457fd6688f07028c6d436df6814c2beebdd4e135f5894:9553897
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x2e968746601a540f10ce58a9c1bd8eb222287ec0ba2beb24435bcac9b8702b7c:6286005 next=0x490722044a90f6c9ac3aadc44b74377323f347f367042000fe223e4f1a66fd84:6286004 l2block=0xcf25f6b5428a85d8a86ad38178c507e74a22752e8c6304587ce522cb356a53ec:9553891
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x490722044a90f6c9ac3aadc44b74377323f347f367042000fe223e4f1a66fd84:6286004 next=0xa6d3686cf17caa671073d3e8c755d332371e6e5b50e9cb2c4ed9c0177ac04f07:6286003 l2block=0x3d55a324659d8255912bff27540b2c8921340b61f63803cd63dfcf08383613e5:9553885
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xa6d3686cf17caa671073d3e8c755d332371e6e5b50e9cb2c4ed9c0177ac04f07:6286003 next=0x6d23213439257ddc1e37dc191391761ac8ee6e281e1d8edd9b35ff3b4195399f:6286002 l2block=0x72c6713e58e5a3b04502eb583ec7eead2308a1469168fe82e078cf632e211209:9553879
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x6d23213439257ddc1e37dc191391761ac8ee6e281e1d8edd9b35ff3b4195399f:6286002 next=0xd13e5837377580c4ee7499147f7e48ac9bb550c825c49790c891b6fe23303483:6286001 l2block=0xf038e0a1bba76a31964bb14d98437d904f7016b2f615cc0eefc08b0273f079b7:9553867
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xd13e5837377580c4ee7499147f7e48ac9bb550c825c49790c891b6fe23303483:6286001 next=0xb0bc61e7a75751b7984587432d1c8b44b71e29cac7253f0f84f5e763ebe7ee88:6286000 l2block=0x189bb3f14d4a90d4e34f25b4628eca30215d361d12b05d9e4ac7706a0ff2cb01:9553861
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xb0bc61e7a75751b7984587432d1c8b44b71e29cac7253f0f84f5e763ebe7ee88:6286000 next=0xf76eed744fa3959bccbbaff0b3278c19a120f6efbb5c621a9d48ff84b6934281:6285999 l2block=0xbacc5c4e344f1c76e9c546dbf2d2cf4314e95197745de4033e006566146b5841:9553855
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xf76eed744fa3959bccbbaff0b3278c19a120f6efbb5c621a9d48ff84b6934281:6285999 next=0xfb4ad759ff2aab2ec45b02dbd1e38bf4ac5905ec89c428511a5e27d786f86c9f:6285998 l2block=0x3bd4eb87d8db11b06285935be40f0e981503bb7564816e16773536de4f98010c:9553849
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xfb4ad759ff2aab2ec45b02dbd1e38bf4ac5905ec89c428511a5e27d786f86c9f:6285998 next=0x7300b1a5ddaf2ee559ba98c559f0c0cd9956d04059bd7adc2dd69c1b87b6e4c4:6285997 l2block=0x82d3b3b94bc9fc104e4faf6004e321196c65c0d0e7558e7d5080f468fe4fd52a:9553843
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x7300b1a5ddaf2ee559ba98c559f0c0cd9956d04059bd7adc2dd69c1b87b6e4c4:6285997 next=0x1e32d2bdbca33d302cd3ba280f3b7d1d6f09fd69ac224ddfb0ec290690288e56:6285996 l2block=0xbb6b8e1592854fe3b12374d10f53c22cf17eb0c22eb726e0f24bf523e22f2331:9553837
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x1e32d2bdbca33d302cd3ba280f3b7d1d6f09fd69ac224ddfb0ec290690288e56:6285996 next=0x8d63382a64087b947b4a68228580870f95e043f677d2f0a93303c67c5d4a2b1f:6285995 l2block=0x830ad26d59a3c567179ae8cd3c94f8ebb12ea3f43e2a70d7892cae36aea4a5da:9553831
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x8d63382a64087b947b4a68228580870f95e043f677d2f0a93303c67c5d4a2b1f:6285995 next=0x56231e7f95de4c8934418826d57f96e163ea84040e2f42f9e197007b5919aef9:6285994 l2block=0x39230464b2088106d9a9deb7e9897d96404e2dbe06815f5c265929ad46cff215:9553825
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x56231e7f95de4c8934418826d57f96e163ea84040e2f42f9e197007b5919aef9:6285994 next=0x0e0f790f9b95f3e6cc8d5a27a3ee241f300797af65dc3a9006a88c9854a0eff5:6285993 l2block=0xb93723e9f1461083d92455f414be0ea7128c0deff3bf021e6a37c881a1a8c654:9553819
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x0e0f790f9b95f3e6cc8d5a27a3ee241f300797af65dc3a9006a88c9854a0eff5:6285993 next=0xaad0f35db94e7de45c5b89d0b13934cf5d40d8da3808199b42a53fecfe285c74:6285992 l2block=0x193b0f407bac9fd4f1ec3ce0f2443fab1503c6146b95a29d8420ce9e9861d695:9553813
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xaad0f35db94e7de45c5b89d0b13934cf5d40d8da3808199b42a53fecfe285c74:6285992 next=0x4ebdc98dea3f4eaff9676b3a2ed5cc3021311784798cdc4b8b9cf463a64e8a96:6285991 l2block=0x16ba84631c1242082a92415568000deb1d129541e0e3af4a5ae1250504cd1842:9553807
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x4ebdc98dea3f4eaff9676b3a2ed5cc3021311784798cdc4b8b9cf463a64e8a96:6285991 next=0x97760a113f0211d5ff768fd65004c0151da45f6c273aa5efb34e925e4cdb0031:6285990 l2block=0x9ef13cd2922b466fd55e3c99378f955d3730f1ee6e183b87d19c84a2d742b8a9:9553801
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x97760a113f0211d5ff768fd65004c0151da45f6c273aa5efb34e925e4cdb0031:6285990 next=0x3bf6121a5434fdc7c3c3ec1a70ac40591a8a8c0551dafc874258ac590192b3ee:6285989 l2block=0x84712f96abb83c6c339e31074d3ebf8d6008c0d1a5f9fae1f4f82c7b7d006775:9553795
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x3bf6121a5434fdc7c3c3ec1a70ac40591a8a8c0551dafc874258ac590192b3ee:6285989 next=0x7afb00788010657cdf55fcda5a0c647acdf993e230f9b512e0421300ef1a8cd7:6285988 l2block=0x391d2bc27b7dff85183cb6015aeb5e74ecff324beb5b9c068122bbdf86e797ed:9553789
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x7afb00788010657cdf55fcda5a0c647acdf993e230f9b512e0421300ef1a8cd7:6285988 next=0x3e2cfc6c7c9c95ffb30489cd3e51542d0ebcc09b44e94f08eeee9875835dce58:6285987 l2block=0xcbdc66c52d05d433749f9e6b890121d73f1fa50d0d3ffa970eb4e070a923fe3b:9553783
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x3e2cfc6c7c9c95ffb30489cd3e51542d0ebcc09b44e94f08eeee9875835dce58:6285987 next=0x51b7778a1851c47186193b6bf0e089173ec6804d7a223c761d01fea9eee8c3fe:6285986 l2block=0x267319c09f910b50d85059702ecff6cb7ee8b9a5095de1c13208f4abbe705358:9553777
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x51b7778a1851c47186193b6bf0e089173ec6804d7a223c761d01fea9eee8c3fe:6285986 next=0x8b0ff4fb6447afa6c5dee61035bf9066232d52d91f84ce0e2234ea65fc3405f9:6285985 l2block=0x90c6706180807d10544068aedaccf55b38c52b028f6b10fe2ef19848b238b8ea:9553771
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x8b0ff4fb6447afa6c5dee61035bf9066232d52d91f84ce0e2234ea65fc3405f9:6285985 next=0x771fb36569d70fc408ab4691eed69f12b53deda8913fd566553ebb48065d1a40:6285984 l2block=0xc0219f0f87166d60a02b287d2eafab6132006ecfcae0acc09d1e74c6a5845b43:9553765
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x771fb36569d70fc408ab4691eed69f12b53deda8913fd566553ebb48065d1a40:6285984 next=0x8e1052844607b41b19c05a4ed8d492e0e9db3bade3c7c914c0f1ea2fee75500f:6285983 l2block=0x279cba59d73bedd162e099288ac140c27914af939d39eb504d5f6f7854f83912:9553759
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x8e1052844607b41b19c05a4ed8d492e0e9db3bade3c7c914c0f1ea2fee75500f:6285983 next=0xc40224e02c33458e757c34b37817dc2e57bc4ee651dc0f83ce17b117cb7b46b4:6285982 l2block=0x68e072570e735e85086ee42af61ece6cea2eb4fee559089ceccb4e06a85e918c:9553753
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xc40224e02c33458e757c34b37817dc2e57bc4ee651dc0f83ce17b117cb7b46b4:6285982 next=0x703c458ec3856db93902c6e760cce5590d6c3dc7ed7b89e6e847a8224cf5c914:6285981 l2block=0x3541fd47977ddb225996fa1684254b9fedb159cf98cb82f6ee9ea36f91b7d739:9553747
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x703c458ec3856db93902c6e760cce5590d6c3dc7ed7b89e6e847a8224cf5c914:6285981 next=0xc9d10779a77c1adfd3f39f1e0679e0a64b2c347287277a081ef6032f9f7c6f90:6285980 l2block=0x8836f2df21f779dbac4bd9155b9df9f41f900f4c5ec4f7436c773cb96733eacc:9553741
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xc9d10779a77c1adfd3f39f1e0679e0a64b2c347287277a081ef6032f9f7c6f90:6285980 next=0x4b99b2547deb0b21436df36086aba72ca366c749fe4e6e4a8812b62ec1a871b3:6285979 l2block=0x18e5d78d6b3a8251e78c396d9582b24bfc6e0155123e813fa6f62881210fc7f5:9553735
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x4b99b2547deb0b21436df36086aba72ca366c749fe4e6e4a8812b62ec1a871b3:6285979 next=0x93eb3283e8576123fd5ed9277e221492432dbb52f01164573461b74b2c7f0ced:6285978 l2block=0x08d0d78ab1455a4a24bb4364b30150347d0f2ec43bf3e542ff269e4b152fa8c1:9553729
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x93eb3283e8576123fd5ed9277e221492432dbb52f01164573461b74b2c7f0ced:6285978 next=0x826023f7f2ff59f2e12fcabceb761f7a309d497223b98adb51a0be3a3930a52e:6285977 l2block=0x12c143505191f89ce88a40525bf36ed3fd923455c2f71c253d97dfdeb2cac0b7:9553723
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x826023f7f2ff59f2e12fcabceb761f7a309d497223b98adb51a0be3a3930a52e:6285977 next=0x396a6f377e6835b64551ceabdb0a47f861f1e93c3c4a1ef89e6ab2c9e73412b5:6285976 l2block=0xf7248252f8ba96759bb18a08f0350947530a3587189921422403105cfd2d013f:9553717
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x396a6f377e6835b64551ceabdb0a47f861f1e93c3c4a1ef89e6ab2c9e73412b5:6285976 next=0x488e5dc44b1386b49334ba9dbfa9c64089a2b1081f12bf125603a448b3da5767:6285975 l2block=0xbad15f545f071eac8099fc847069d22f3eb3a86703a975d682e383da7ad9974d:9553711
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x488e5dc44b1386b49334ba9dbfa9c64089a2b1081f12bf125603a448b3da5767:6285975 next=0xe97f291c77fbbb77939b6b55d83bd02dde991a4487cade4078a4d8f3aa27604c:6285974 l2block=0xaee4250922d469448122ef2da130adf3281d5f78c981a910de465d57678763fa:9553705
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xe97f291c77fbbb77939b6b55d83bd02dde991a4487cade4078a4d8f3aa27604c:6285974 next=0xffa9d4d65cebced356a8ddb11882c48397393353671c2b846acf55e3b1b8cea4:6285973 l2block=0x6b7a8a06a560c811d47b05114a3004dd79fd370b05d570668e6cab0b66b86513:9553699
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xffa9d4d65cebced356a8ddb11882c48397393353671c2b846acf55e3b1b8cea4:6285973 next=0x8e2c8b73b555a79f8169072f2d6a487fa91c8daed5cf56eb5931e66bc8e68986:6285972 l2block=0xce10049e9a0fedd84b57926a67dc63ffc304e3ac1aeab0cffc13015a0ac75d91:9553693
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x8e2c8b73b555a79f8169072f2d6a487fa91c8daed5cf56eb5931e66bc8e68986:6285972 next=0x16818b6e8e1963f57fd84b77b4a912a16aaaccbf9feb8f6e893228c8a92fbf06:6285971 l2block=0x39cc230ae2578906f63e0240b47b9a892328a2b83463d74bc0c8fac6b5393ed4:9553687
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x16818b6e8e1963f57fd84b77b4a912a16aaaccbf9feb8f6e893228c8a92fbf06:6285971 next=0xca4103790192e4b6563bfaa136d33c504482bd4b0be5300d914c5e526adff0c0:6285970 l2block=0x844ece158176cb721209d40e4dd153c2afd17b7a97abe3eb70fd9779d0bf1009:9553681
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xca4103790192e4b6563bfaa136d33c504482bd4b0be5300d914c5e526adff0c0:6285970 next=0x133a85d5e38ecdcd016ce943f233f7e90e3bec09a1ea46831bfb50500fbb1cb6:6285969 l2block=0x147cc2f43d95bed31365dd309ba26856c3d791ef7835e51aaa7571c5b7ec610c:9553675
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x133a85d5e38ecdcd016ce943f233f7e90e3bec09a1ea46831bfb50500fbb1cb6:6285969 next=0x239d9f3683c27a6a959d6d7a67783ad9aa505ce2961faee7f6e0e8848dd17a6a:6285968 l2block=0x3b01e4ab0a8f1edeb1468efc693e2b79a5e04bee4bec3bb67b27ff23cce0ab3b:9553669
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x239d9f3683c27a6a959d6d7a67783ad9aa505ce2961faee7f6e0e8848dd17a6a:6285968 next=0xd86fc92f6cfe0d1921811c8ac765b69d62e1aad5be42dbac154131df8a3c7479:6285967 l2block=0x0e1dbf8d4d3d86db4429f8f02461190c5804e262517ed820c72114950d1ab134:9553663
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xd86fc92f6cfe0d1921811c8ac765b69d62e1aad5be42dbac154131df8a3c7479:6285967 next=0xaa355271f4acbe2dc6fd2d4e4f3255c9d0e6de71ee4114c1d81dedbcdcbac631:6285966 l2block=0xf5555cfd959f3a999994aeeae21166a65593939fd71d960e0da06a756384e771:9553657
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xaa355271f4acbe2dc6fd2d4e4f3255c9d0e6de71ee4114c1d81dedbcdcbac631:6285966 next=0x057d45b6eaa6aeddb8b9b15b67c996d37b2d831e32c84a8604f7d4d71378adb5:6285965 l2block=0x86fbbc3ba4ec9bfa32f2a3f6f3f4c40dfebdb7c5f66068ca5b26934203cd9132:9553651
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x057d45b6eaa6aeddb8b9b15b67c996d37b2d831e32c84a8604f7d4d71378adb5:6285965 next=0x45e9074219beda5b722d6fd1892e4d3d0d7eab0fb5cd0d1f1423cf91c4f2bca8:6285964 l2block=0x51992225672378fd847213e1569f8ea2d1dc8fde48f0b5d0011b9d728d95a322:9553645
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x45e9074219beda5b722d6fd1892e4d3d0d7eab0fb5cd0d1f1423cf91c4f2bca8:6285964 next=0xf1c6314a3d44316cc30c48a879943627e35232394c50d73ce181621bf68d7b9a:6285963 l2block=0x12261e15cedebf1e82ac24be4a16c245809abcb619c56070cadc8aeb8af2c578:9553639
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xf1c6314a3d44316cc30c48a879943627e35232394c50d73ce181621bf68d7b9a:6285963 next=0x337a3e1f794801ef28e421acdbd6a2cd204c80e2fd836e15ac1425de4025e358:6285962 l2block=0x827c4f5f36d596fe0dbdca33c67989cf9a80f219e9a493fdf2c5d74531de3fef:9553633
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x337a3e1f794801ef28e421acdbd6a2cd204c80e2fd836e15ac1425de4025e358:6285962 next=0xcaef4b6364a36fb9f2ad9d2bac56c33980cf2fc6ba8c337ed6758a7b3cf374d4:6285961 l2block=0xc816101cdd035c3876df9d944fa377aedaf324eba674d6d5861c7912afa7aca8:9553627
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xcaef4b6364a36fb9f2ad9d2bac56c33980cf2fc6ba8c337ed6758a7b3cf374d4:6285961 next=0x41d17d50fbd11f746d5725bf211cbb3a9f5f544622557e4ea45317430db1a775:6285960 l2block=0x2f89a56f23752349e67c3f30d341279ed723588f465f23ae2c6e3170484ed34e:9553621
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x41d17d50fbd11f746d5725bf211cbb3a9f5f544622557e4ea45317430db1a775:6285960 next=0x7ba8f007f8cf89646a0e9aef2a1eac5771ea10e8ec545a66949a330bf7b3f52c:6285959 l2block=0x715d5f28b0fc2b986b293239a6bc3f2707276964faf9ea5c6714415660d0f2a7:9553615
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x7ba8f007f8cf89646a0e9aef2a1eac5771ea10e8ec545a66949a330bf7b3f52c:6285959 next=0x5b6705eccf520af7a2da957f98ff086c40cfa7fd24481c0cc12cd3573aa0971f:6285958 l2block=0xd378373944a09ea78c42f87bbd1a908cb324b62aa06fb4362d4b646dc5fb7a9f:9553609
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x5b6705eccf520af7a2da957f98ff086c40cfa7fd24481c0cc12cd3573aa0971f:6285958 next=0x1c07a3c0287d4b133111bf249a585d090f9c3975e5a1b5c11cfdbaa6c19be7c7:6285957 l2block=0x1d51d63c07b7c62ca52392890a268096b8a7eebe51184b5bf024d306e08d43c1:9553603
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x1c07a3c0287d4b133111bf249a585d090f9c3975e5a1b5c11cfdbaa6c19be7c7:6285957 next=0x635ccff7ef33ca17106adbc5244d2538bb6cac16543acaa0d78c72fe42f21223:6285956 l2block=0x2602657a8ef1ef60a3947c0cc25c7821f06e25eded9ac42301a204fe383a03f7:9553597
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x635ccff7ef33ca17106adbc5244d2538bb6cac16543acaa0d78c72fe42f21223:6285956 next=0xe7f57a6839abe8eefeceba30f9cc3e3d2e0a5ea2265f7719a6e80fd9d5b66978:6285955 l2block=0x4f72c7c114ee16e8fb9c17f22bf7019a40a9332daccd271ce1b8c4c4dbc23527:9553585
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xe7f57a6839abe8eefeceba30f9cc3e3d2e0a5ea2265f7719a6e80fd9d5b66978:6285955 next=0xf0cae66dfb1072aa702e0a3f728ffae23ce16e1fdd676f5268a75a59c6a59ba2:6285954 l2block=0x979d3c7d16f4d49362041060afccf80ce0440c586d96659cdf65acf9bb08b399:9553579
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xf0cae66dfb1072aa702e0a3f728ffae23ce16e1fdd676f5268a75a59c6a59ba2:6285954 next=0xb7b5a4fa55472cd3f57b9b7724214d0a9f323d8bce1916f2039cb3c27646e4ea:6285953 l2block=0x9a57010d2038cd59b685f257fbad9519fb39e4a5daa7cbc8e085886d06954986:9553573
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xb7b5a4fa55472cd3f57b9b7724214d0a9f323d8bce1916f2039cb3c27646e4ea:6285953 next=0x7fb7ac42d93dcd15c17faf97772b37bbdbebe8aa4ece824adf049f2151fcf256:6285952 l2block=0x02f8ffb221906f5a0b58527fcac52407724a0dfb59dc11825c3878cf22c9d692:9553567
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x7fb7ac42d93dcd15c17faf97772b37bbdbebe8aa4ece824adf049f2151fcf256:6285952 next=0x944dcc591179cc1fb3c7a85133c71f41aafb636587b87b5420d7898698b9820a:6285951 l2block=0x4ceee810e9534d6af0ee9ba24f5176833222a6ee4f6f6936b0a6d1787477f147:9553561
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x944dcc591179cc1fb3c7a85133c71f41aafb636587b87b5420d7898698b9820a:6285951 next=0xee4691066375ed035ed2f41a687c75beda14a4bf0c8239b39ad5cecb794b9372:6285950 l2block=0x28abb8b7769972509f31e8340f88e30736075ebda76f6e5ea8a5a9d436ee295c:9553555
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xee4691066375ed035ed2f41a687c75beda14a4bf0c8239b39ad5cecb794b9372:6285950 next=0x25109e045d9e9a18c9e0ba1685f9a61de22c3be914064b6e34e84c5a1718fa81:6285949 l2block=0x1897f882036721e839a2b5b2755232702223bbbd100dfda54556626dd06f5470:9553549
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x25109e045d9e9a18c9e0ba1685f9a61de22c3be914064b6e34e84c5a1718fa81:6285949 next=0x783262f6b24e216b6edd31f9c72f4d253a148ce4e8f19019455720e74e75d1c6:6285948 l2block=0x1fe1c4b3a712cd5fde5953af24fd53ba02b1164c8b21704f05256a37fc158820:9553543
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x783262f6b24e216b6edd31f9c72f4d253a148ce4e8f19019455720e74e75d1c6:6285948 next=0x42726633760b5346e1064fbdcc941b0e8721044a38b34b31b8ac9bc4c66f7c0c:6285947 l2block=0x0fa3b8e0f8d8de21090be6aa39f492717461261c2b9f567d61d425ddd3259496:9553537
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x42726633760b5346e1064fbdcc941b0e8721044a38b34b31b8ac9bc4c66f7c0c:6285947 next=0x41287f21c4f91991a62b67047f8383f364828f5cb79103c4d708b87b34049521:6285946 l2block=0x4f60908218a2954283443565d9208889504bad64105565ebd81287b5ef94edde:9553531
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x41287f21c4f91991a62b67047f8383f364828f5cb79103c4d708b87b34049521:6285946 next=0xd56dabae3dd585808df2ba9bb13ba8e27e8ea0d553bad9a723f1c9fcd6e6a8b4:6285945 l2block=0x63131f16988c43c02d45d3e18a50f7ad59bd1439612450ff00dc1021f37d1657:9553525
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xd56dabae3dd585808df2ba9bb13ba8e27e8ea0d553bad9a723f1c9fcd6e6a8b4:6285945 next=0xd1e4dd44c980f4083c9fc3d591b4c948fdbba326e803240a69d0a7a53b6eeddf:6285944 l2block=0xaa4808182b02ee2dc864e97e5293eb3a2243c811668485dbacfa25f19948201d:9553519
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0xd1e4dd44c980f4083c9fc3d591b4c948fdbba326e803240a69d0a7a53b6eeddf:6285944 next=0x1008c42832ef600c0d98ac2511c1b69b6d3bf4c6fd5f2e96253df2edb7c63b40:6285943 l2block=0xcad645ca44a71421ea0205538c8ec5a6d57f6fc01f7c2a31c340779f6fb7697d:9553513
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x1008c42832ef600c0d98ac2511c1b69b6d3bf4c6fd5f2e96253df2edb7c63b40:6285943 next=0x64e42b11d7738d287c1d633f6a4a70cb865f87bfc616ccfb521cee93fc18d969:6285942 l2block=0xa50a35c41b48e91c5d9f3a0e07e2815094e497a5a6131fb40bbe8522330ed5b1:9553507
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x64e42b11d7738d287c1d633f6a4a70cb865f87bfc616ccfb521cee93fc18d969:6285942 next=0x13715b03a0f3e21fa6d4ed02c8cb7f64a2cba3c566060bdc484b17650df7974d:6285941 l2block=0x53ef800481aef90748facf6f48771bde09efcbdbfa59eaadaf22aa5bcaef9738:9553501
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=warn msg="Derivation process temporary error"  attempts=23 err="stage 0 failed resetting: temp: failed to find the L2 Heads to start from: cannot reorg finalized block: finalized 0x6f7377e755fae74baec7e6d7a16e05f4a13f74aacbf40f7eb91913b15a2cfeb2:9553497, got: 0x0b28fe494aee3c71346dd52ade6bd0815091eb4bff352220313eddea48180d31:9553497"
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Received signed execution payload from p2p" id=0xd90879b0360b098b1e1c5df30bc8efdc2f408512cb44b50ee2d503a53e9b4a1d:9578472 peer=16Uiu2HAkzQm9cS8kGvHk3stFeqnKABpVBgTstZGh3qzgomqoQvho
Jul 11 12:40:17 t-mantle-sepolia-02 op-node[580141]: t=2024-07-11T12:40:17+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xd90879b0360b098b1e1c5df30bc8efdc2f408512cb44b50ee2d503a53e9b4a1d:9578472
andreclaro commented 4 months ago

Do you have a recent snapshot so we can use to recover our node? 🤔

BlueShells commented 4 months ago

@andreclaro please follow this guide https://github.com/mantlenetworkio/networks/blob/main/run-node-sepolia.md

andreclaro commented 4 months ago

@BlueShells , this is a very old snapshot:

tarball="20240623-sepolia-chaindata.tar.zst"

don't you have a near one after the upgrade?

BlueShells commented 4 months ago

@andreclaro try this one https://s3.ap-southeast-1.amazonaws.com/snapshot.sepolia.mantle.xyz/20240707-sepolia-chaindata.tar.zst , we will update the snapshto to the doc too

andreclaro commented 4 months ago

@BlueShells , is that one after the upgrade?

andreclaro commented 4 months ago

@BlueShells @pandainzoo , can you provide a more recent snapshot after the upgrade please? thanks 🙏🏽

andreclaro commented 4 months ago

is this one https://s3.ap-southeast-1.amazonaws.com/snapshot.sepolia.mantle.xyz/20240710-sepolia-chaindata.tar.zst after the upgrade? if no, can you provide a most recent one please? thanks

andreclaro commented 4 months ago

I used the snapshot from 2024-07-10 and the node is now recovering:

INFO [07-11|14:09:59.100] Imported new potential chain segment     number=9,527,594 hash=2e83d6..8b6879 blocks=1 txs=2 mgas=1129.822  elapsed=2.106ms     mgasps=536460.825   age=1d5h45m dirty=0.00B
andreclaro commented 4 months ago

Node is now 13 hours behind... very slowly syncing and seems it won't be able to catch up soon:

op-geth logs:

Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.977] Chain head was updated                   number=9,559,042 hash=2436ce..4bbb63 root=1f738c..2f5a62 elapsed="77.354µs"  age=13h12m39s
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.979] Starting work on payload                 id=0xd9ced0f6964f3b93
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.981] Imported new potential chain segment     number=9,559,043 hash=0aead3..e3019c blocks=1 txs=1 mgas=0.064 elapsed="414.78µs"  mgasps=154.243 age=13h12m37s dirty=0.00B
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.982] Chain head was updated                   number=9,559,043 hash=0aead3..e3019c root=258239..ccae75 elapsed="72.323µs"  age=13h12m37s
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.984] Starting work on payload                 id=0x12a2fb1d170466cd
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.985] Imported new potential chain segment     number=9,559,044 hash=8165de..8098b1 blocks=1 txs=1 mgas=0.047 elapsed="436.502µs" mgasps=107.392 age=13h12m35s dirty=0.00B
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.986] Chain head was updated                   number=9,559,044 hash=8165de..8098b1 root=c6bbfb..29a433 elapsed="73.963µs"  age=13h12m35s
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.988] Starting work on payload                 id=0x117d8d95ea1632b5
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.990] Imported new potential chain segment     number=9,559,045 hash=4ecda8..e1ecde blocks=1 txs=1 mgas=0.047 elapsed="420.54µs"  mgasps=111.469 age=13h12m33s dirty=0.00B
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.991] Chain head was updated                   number=9,559,045 hash=4ecda8..e1ecde root=e69d6a..13843f elapsed="72.184µs"  age=13h12m33s
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.993] Starting work on payload                 id=0x809c0c090f663a13
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.994] Imported new potential chain segment     number=9,559,046 hash=9f4569..3f5415 blocks=1 txs=1 mgas=0.047 elapsed="408.83µs"  mgasps=114.661 age=13h12m31s dirty=0.00B
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.995] Chain head was updated                   number=9,559,046 hash=9f4569..3f5415 root=36693d..3f9168 elapsed="77.573µs"  age=13h12m31s
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.997] Starting work on payload                 id=0x024c80d382f32568
Jul 11 15:05:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:16.999] Imported new potential chain segment     number=9,559,047 hash=c1e1f7..a70fe5 blocks=1 txs=1 mgas=0.047 elapsed="457.252µs" mgasps=102.519 age=13h12m29s dirty=0.00B
Jul 11 15:05:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:17.000] Chain head was updated                   number=9,559,047 hash=c1e1f7..a70fe5 root=e48bf3..d9c903 elapsed="72.414µs"  age=13h12m30s
Jul 11 15:05:21 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:21.556] Looking for peers                        peercount=0 tried=295 static=0
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.586] Starting work on payload                 id=0x20076638f9a3a5b5
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.588] Imported new potential chain segment     number=9,559,048 hash=9d1b87..041649 blocks=1 txs=1 mgas=0.055 elapsed="543.296µs" mgasps=101.722 age=13h12m42s dirty=0.00B
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.588] Chain head was updated                   number=9,559,048 hash=9d1b87..041649 root=6d0505..a62c0b elapsed="77.434µs"  age=13h12m42s
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.591] Starting work on payload                 id=0x8c1d6e3bb2a27293
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.592] Imported new potential chain segment     number=9,559,049 hash=9f923e..0f6a67 blocks=1 txs=1 mgas=0.064 elapsed="471.264µs" mgasps=135.756 age=13h12m40s dirty=0.00B
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.593] Chain head was updated                   number=9,559,049 hash=9f923e..0f6a67 root=be92ca..a0a8c1 elapsed="73.123µs"  age=13h12m40s
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.595] Starting work on payload                 id=0xe2ad8915b1870edc
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.597] Imported new potential chain segment     number=9,559,050 hash=a6aaaf..d84ab1 blocks=1 txs=1 mgas=0.047 elapsed="487.344µs" mgasps=96.189  age=13h12m38s dirty=0.00B
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.598] Chain head was updated                   number=9,559,050 hash=a6aaaf..d84ab1 root=23d618..654f99 elapsed="69.063µs"  age=13h12m38s
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.600] Starting work on payload                 id=0x576bfb9d53ec9d05
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.601] Imported new potential chain segment     number=9,559,051 hash=343fb3..7fd16c blocks=1 txs=1 mgas=0.047 elapsed="451.123µs" mgasps=103.912 age=13h12m36s dirty=0.00B
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.602] Chain head was updated                   number=9,559,051 hash=343fb3..7fd16c root=caa76c..a4b70d elapsed="68.364µs"  age=13h12m36s
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.604] Starting work on payload                 id=0xe5636b3bb460d69c
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.606] Imported new potential chain segment     number=9,559,052 hash=8aa04d..4ae7fb blocks=1 txs=1 mgas=0.047 elapsed="429.111µs" mgasps=109.242 age=13h12m34s dirty=0.00B
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.607] Chain head was updated                   number=9,559,052 hash=8aa04d..4ae7fb root=2e044a..6909b9 elapsed="86.184µs"  age=13h12m34s
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.609] Starting work on payload                 id=0x695feeb79c95998c
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.611] Imported new potential chain segment     number=9,559,053 hash=2170bc..8a5de9 blocks=1 txs=1 mgas=0.047 elapsed="444.302µs" mgasps=105.507 age=13h12m32s dirty=0.00B
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.611] Chain head was updated                   number=9,559,053 hash=2170bc..8a5de9 root=0ff2c1..15fd72 elapsed="70.533µs"  age=13h12m32s
Jul 11 15:05:31 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:31.711] Looking for peers                        peercount=0 tried=207 static=0
Jul 11 15:05:41 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:41.766] Looking for peers                        peercount=0 tried=303 static=0
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.589] Starting work on payload                 id=0x97362259b3fa61cf
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.591] Imported new potential chain segment     number=9,559,054 hash=c352fe..5aa272 blocks=1 txs=1 mgas=0.055 elapsed="551.947µs" mgasps=100.127 age=13h12m42s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.592] Chain head was updated                   number=9,559,054 hash=c352fe..5aa272 root=641907..54a781 elapsed="78.454µs"  age=13h12m42s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.594] Starting work on payload                 id=0x64113ad35af8fece
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.596] Imported new potential chain segment     number=9,559,055 hash=3c67f6..0771f5 blocks=1 txs=1 mgas=0.064 elapsed="406.951µs" mgasps=157.211 age=13h12m40s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.596] Chain head was updated                   number=9,559,055 hash=3c67f6..0771f5 root=903214..fd03d2 elapsed="97.645µs"  age=13h12m40s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.598] Starting work on payload                 id=0x29b774ea25b4dc19
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.600] Imported new potential chain segment     number=9,559,056 hash=e83288..4dfa8f blocks=1 txs=1 mgas=0.047 elapsed="422.66µs"  mgasps=110.909 age=13h12m38s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.601] Chain head was updated                   number=9,559,056 hash=e83288..4dfa8f root=6e4cef..818c9f elapsed="71.513µs"  age=13h12m38s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.603] Starting work on payload                 id=0x551374c7236e84b1
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.605] Imported new potential chain segment     number=9,559,057 hash=163659..31a6ed blocks=1 txs=1 mgas=0.047 elapsed="455.142µs" mgasps=102.994 age=13h12m36s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.605] Chain head was updated                   number=9,559,057 hash=163659..31a6ed root=d8e283..5c0863 elapsed="67.023µs"  age=13h12m36s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.607] Starting work on payload                 id=0x6493c1f217718e01
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.609] Imported new potential chain segment     number=9,559,058 hash=c2392e..ac629f blocks=1 txs=1 mgas=0.047 elapsed="398.769µs" mgasps=117.554 age=13h12m34s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.610] Chain head was updated                   number=9,559,058 hash=c2392e..ac629f root=ab5d0d..70d7ad elapsed="63.453µs"  age=13h12m34s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.612] Starting work on payload                 id=0x67d9c80eebdcbc9b
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.613] Imported new potential chain segment     number=9,559,059 hash=98616e..686fe1 blocks=1 txs=1 mgas=0.047 elapsed="475.884µs" mgasps=98.505  age=13h12m32s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.614] Chain head was updated                   number=9,559,059 hash=98616e..686fe1 root=1bd227..602fb6 elapsed="71.974µs"  age=13h12m32s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.616] Starting work on payload                 id=0x6682d84938b283b9
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.618] Imported new potential chain segment     number=9,559,060 hash=b42d55..cb8667 blocks=1 txs=1 mgas=0.047 elapsed="451.082µs" mgasps=103.921 age=13h12m30s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.619] Chain head was updated                   number=9,559,060 hash=b42d55..cb8667 root=5e1bda..01a1bc elapsed="73.404µs"  age=13h12m30s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.621] Starting work on payload                 id=0x3cb9bdce50fefb38
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.622] Imported new potential chain segment     number=9,559,061 hash=53040c..491459 blocks=1 txs=1 mgas=0.047 elapsed="409.36µs"  mgasps=114.513 age=13h12m28s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.623] Chain head was updated                   number=9,559,061 hash=53040c..491459 root=42cfe6..ac8cba elapsed="69.494µs"  age=13h12m28s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.625] Starting work on payload                 id=0xb444a097b0b318b0
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.627] Imported new potential chain segment     number=9,559,062 hash=0997b5..d688ae blocks=1 txs=1 mgas=0.047 elapsed="450.913µs" mgasps=103.960 age=13h12m26s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.627] Chain head was updated                   number=9,559,062 hash=0997b5..d688ae root=948576..cc0ef7 elapsed="68.163µs"  age=13h12m26s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.629] Starting work on payload                 id=0x5efff6d5ede00a35
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.631] Imported new potential chain segment     number=9,559,063 hash=c042c9..7d2431 blocks=1 txs=1 mgas=0.047 elapsed="482.523µs" mgasps=97.150  age=13h12m24s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.631] Chain head was updated                   number=9,559,063 hash=c042c9..7d2431 root=f4aa65..9573ed elapsed="72.633µs"  age=13h12m24s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.633] Starting work on payload                 id=0x1bef507d6b0c2be9
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.635] Imported new potential chain segment     number=9,559,064 hash=ae7ab5..120a50 blocks=1 txs=1 mgas=0.047 elapsed="441.381µs" mgasps=106.205 age=13h12m22s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.636] Chain head was updated                   number=9,559,064 hash=ae7ab5..120a50 root=0df0de..2fba37 elapsed="70.293µs"  age=13h12m22s
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.638] Starting work on payload                 id=0x36cc6aaee770b281
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.640] Imported new potential chain segment     number=9,559,065 hash=52c6aa..32e141 blocks=1 txs=1 mgas=0.047 elapsed="419.22µs"  mgasps=111.820 age=13h12m20s dirty=0.00B
Jul 11 15:05:43 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:43.640] Chain head was updated                   number=9,559,065 hash=52c6aa..32e141 root=771aea..2941c4 elapsed="73.544µs"  age=13h12m20s
Jul 11 15:05:52 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:52.110] Looking for peers                        peercount=0 tried=107 static=0
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.587] Starting work on payload                 id=0x6fa9b65ab7cee6fd
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.590] Imported new potential chain segment     number=9,559,066 hash=5ca96d..b6000a blocks=1 txs=1 mgas=0.055 elapsed="554.097µs" mgasps=99.739  age=13h12m28s dirty=0.00B
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.591] Chain head was updated                   number=9,559,066 hash=5ca96d..b6000a root=6e76c8..9ff1bb elapsed="85.164µs"  age=13h12m28s
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.593] Starting work on payload                 id=0x086bf6440207dea5
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.595] Imported new potential chain segment     number=9,559,067 hash=f432f6..2a0af5 blocks=1 txs=1 mgas=0.064 elapsed="480.173µs" mgasps=133.237 age=13h12m26s dirty=0.00B
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.596] Chain head was updated                   number=9,559,067 hash=f432f6..2a0af5 root=39b2a2..422955 elapsed="74.193µs"  age=13h12m26s
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.598] Starting work on payload                 id=0xa20ba4039c4d30e1
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.600] Imported new potential chain segment     number=9,559,068 hash=cae011..440a01 blocks=1 txs=1 mgas=0.047 elapsed="435.481µs" mgasps=107.644 age=13h12m24s dirty=0.00B
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.600] Chain head was updated                   number=9,559,068 hash=cae011..440a01 root=233300..995775 elapsed="67.223µs"  age=13h12m24s
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.602] Starting work on payload                 id=0x3c70d9bf45759cf6
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.604] Imported new potential chain segment     number=9,559,069 hash=aee3fd..18d880 blocks=1 txs=1 mgas=0.047 elapsed="449.122µs" mgasps=104.375 age=13h12m22s dirty=0.00B
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.605] Chain head was updated                   number=9,559,069 hash=aee3fd..18d880 root=b447c3..bfb9d5 elapsed="71.663µs"  age=13h12m22s
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.607] Starting work on payload                 id=0xc976d0646b7f7c3d
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.609] Imported new potential chain segment     number=9,559,070 hash=112e24..63b75e blocks=1 txs=1 mgas=0.047 elapsed="465.742µs" mgasps=100.650 age=13h12m20s dirty=0.00B
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.609] Chain head was updated                   number=9,559,070 hash=112e24..63b75e root=a8e9c6..cd3fae elapsed="77.893µs"  age=13h12m20s
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.611] Starting work on payload                 id=0xa4f01a21675d94eb
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.613] Imported new potential chain segment     number=9,559,071 hash=d440ed..d71de1 blocks=1 txs=1 mgas=0.047 elapsed="392.889µs" mgasps=119.314 age=13h12m18s dirty=0.00B
Jul 11 15:05:53 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:05:53.613] Chain head was updated                   number=9,559,071 hash=d440ed..d71de1 root=421478..00bb37 elapsed="64.573µs"  age=13h12m18s
Jul 11 15:06:02 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:02.600] Looking for peers                        peercount=0 tried=286 static=0
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.579] Starting work on payload                 id=0xd5527418cbab0f43
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.581] Imported new potential chain segment     number=9,559,072 hash=20600f..a6025b blocks=1 txs=1 mgas=0.055 elapsed="568.848µs" mgasps=97.152  age=13h12m30s dirty=0.00B
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.582] Chain head was updated                   number=9,559,072 hash=20600f..a6025b root=c870c4..b42c7e elapsed="90.614µs"  age=13h12m30s
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.584] Starting work on payload                 id=0x7055a59be67fa1d3
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.587] Imported new potential chain segment     number=9,559,073 hash=ce5cea..b788ed blocks=1 txs=1 mgas=0.064 elapsed="521.705µs" mgasps=122.631 age=13h12m28s dirty=0.00B
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.588] Chain head was updated                   number=9,559,073 hash=ce5cea..b788ed root=93522b..ccb7c0 elapsed="71.333µs"  age=13h12m28s
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.590] Starting work on payload                 id=0xbc74cec9df4121b2
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.592] Imported new potential chain segment     number=9,559,074 hash=5861ee..a2af02 blocks=1 txs=1 mgas=0.047 elapsed="457.403µs" mgasps=102.485 age=13h12m26s dirty=0.00B
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.593] Chain head was updated                   number=9,559,074 hash=5861ee..a2af02 root=be4f78..5e7be1 elapsed="72.773µs"  age=13h12m26s
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.595] Starting work on payload                 id=0xb49560870882304e
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.597] Imported new potential chain segment     number=9,559,075 hash=44288f..3fa9df blocks=1 txs=1 mgas=0.047 elapsed="478.964µs" mgasps=97.872  age=13h12m24s dirty=0.00B
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.598] Chain head was updated                   number=9,559,075 hash=44288f..3fa9df root=d55cc5..0a9e54 elapsed="78.604µs"  age=13h12m24s
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.600] Starting work on payload                 id=0x649a57345d7b043a
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.602] Imported new potential chain segment     number=9,559,076 hash=c1e1eb..2dc8f2 blocks=1 txs=1 mgas=0.047 elapsed="466.072µs" mgasps=100.579 age=13h12m22s dirty=0.00B
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.602] Chain head was updated                   number=9,559,076 hash=c1e1eb..2dc8f2 root=4d857e..a31d74 elapsed="70.614µs"  age=13h12m22s
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.605] Starting work on payload                 id=0x1eaf585d779276a2
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.606] Imported new potential chain segment     number=9,559,077 hash=93e530..3e26df blocks=1 txs=1 mgas=0.047 elapsed="429.601µs" mgasps=109.118 age=13h12m20s dirty=0.00B
Jul 11 15:06:07 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:07.607] Chain head was updated                   number=9,559,077 hash=93e530..3e26df root=e33108..dfdf34 elapsed="77.133µs"  age=13h12m20s
Jul 11 15:06:12 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:12.607] Looking for peers                        peercount=0 tried=130 static=0
Jul 11 15:06:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:16.992] Starting work on payload                 id=0x616988a60a723ed3
Jul 11 15:06:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:16.994] Imported new potential chain segment     number=9,559,078 hash=441999..b3138d blocks=1 txs=1 mgas=0.055 elapsed="527.315µs" mgasps=104.805 age=13h12m27s dirty=0.00B
Jul 11 15:06:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:16.995] Chain head was updated                   number=9,559,078 hash=441999..b3138d root=3967df..29539f elapsed="74.663µs"  age=13h12m27s
Jul 11 15:06:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:16.997] Starting work on payload                 id=0x151548304c0852bf
Jul 11 15:06:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:16.998] Imported new potential chain segment     number=9,559,079 hash=e586e1..44cbe9 blocks=1 txs=1 mgas=0.064 elapsed="424.141µs" mgasps=150.839 age=13h12m25s dirty=0.00B
Jul 11 15:06:16 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:16.999] Chain head was updated                   number=9,559,079 hash=e586e1..44cbe9 root=37ae11..bc7ade elapsed="66.963µs"  age=13h12m25s
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.001] Starting work on payload                 id=0x48be60f8395340e8
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.003] Imported new potential chain segment     number=9,559,080 hash=4b9e7b..9d47a5 blocks=1 txs=1 mgas=0.047 elapsed="458.162µs" mgasps=102.315 age=13h12m24s dirty=0.00B
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.003] Chain head was updated                   number=9,559,080 hash=4b9e7b..9d47a5 root=db0977..b2b97f elapsed="69.763µs"  age=13h12m24s
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.005] Starting work on payload                 id=0xa8fa5e7afd9403b5
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.007] Imported new potential chain segment     number=9,559,081 hash=c145e7..94b5d6 blocks=1 txs=1 mgas=0.047 elapsed="423.57µs"  mgasps=110.671 age=13h12m22s dirty=0.00B
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.007] Chain head was updated                   number=9,559,081 hash=c145e7..94b5d6 root=cbcfc0..0e18e7 elapsed="67.353µs"  age=13h12m22s
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.009] Starting work on payload                 id=0x617198f4fd27e96e
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.011] Imported new potential chain segment     number=9,559,082 hash=1f8774..28db0a blocks=1 txs=1 mgas=0.047 elapsed="441.891µs" mgasps=106.083 age=13h12m20s dirty=0.00B
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.012] Chain head was updated                   number=9,559,082 hash=1f8774..28db0a root=7e31eb..43c9f0 elapsed="71.794µs"  age=13h12m20s
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.014] Starting work on payload                 id=0x6ff1fdb40b2586bc
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.015] Imported new potential chain segment     number=9,559,083 hash=db5946..561f58 blocks=1 txs=1 mgas=0.047 elapsed="397.999µs" mgasps=117.782 age=13h12m18s dirty=0.00B
Jul 11 15:06:17 t-mantle-sepolia-02 geth[602104]: INFO [07-11|15:06:17.016] Chain head was updated                   number=9,559,083 hash=db5946..561f58 root=763d16..b3cc2c elapsed="65.733µs"  age=13h12m18s

Block speed:

[2024-07-11 15:05:12] Latest Block Number: 9559041
[2024-07-11 15:06:12] Latest Block Number: 9559077
[2024-07-11 15:07:12] Latest Block Number: 9559107
[2024-07-11 15:08:12] Latest Block Number: 9559131

image

I don't see any bottleneck on the instance (no cpu, memory nor disk access limitation)... I will provide an update tomorrow morning (UTC timezone). Thanks

andreclaro commented 4 months ago

@BlueShells and @pandainzoo , it seems the sepolia node is forking...

curl -s http://localhost:8545 -X POST -H "Content-Type: application/json" --data '{"method":"eth_getBlockByNumber","params":["latest",false],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":{"baseFeePerGas":"0x1312d00","difficulty":"0x0","extraData":"0x","gasLimit":"0xe8d4a51000","gasUsed":"0xb729","hash":"0x69b4f2ddf75674a145173266730dcb698aa71c0f3a70f9292749901eae0f6f5a","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x4200000000000000000000000000000000000011","mixHash":"0xdfc8719408e7b542a6550ab72b68108718e6a57fa27f5804462240535e9bfcc9","nonce":"0x0000000000000000","number":"0x92516d","parentHash":"0xfc03dfb4d961cfbee468069f7253fc0431747d5dcd5cc7dbcd8f860dcdb71f03","receiptsRoot":"0x8f468282575b3a1ccef67b268423ac1c52e9b72ca3bda3cb187069dd45a8443b","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x36b","stateRoot":"0xd2fa511bbf311cf773f84a1dd3f325f188f10a3c39319c2609fd960b39392649","timestamp":"0x669025bb","totalDifficulty":"0x0","transactions":["0x3bfb7409cc34dbaa18bba3a2585394b41e805b94e7dc906faa761646e8c2cbba"],"transactionsRoot":"0xc4241cb8d2d8e3349f422ec394064c7e7637582ee81d18f7f9706a95cf266d89","uncles":[]}}

the hash of the block 9589101 (0x92516d) is be 0x06917b6e1596e489513185ca6d42185c3bb0f29143baad8590eab8dcfae246c2 as per the explorer: https://explorer.sepolia.mantle.xyz/block/0x06917b6e1596e489513185ca6d42185c3bb0f29143baad8590eab8dcfae246c2

andreclaro commented 4 months ago

trying again to sync-up from the latest snapshot (20240710-sepolia-chaindata.tar.zst). I have reviewed the op-node settings and update a few parameters. Here is the op-node config:

/usr/local/bin/op-node \
    --l1=http://t-eth-sepolia:8545 \
    --l2=http://127.0.0.1:8551 \
    --l2.jwt-secret=/var/lib/mantle/jwt.hex \
    --l2.backup-unsafe-sync-rpc=https://rpc.sepolia.mantle.xyz \
    --l2.engine-sync=true \
    --l2.skip-sync-start-check=true \
    --sequencer.enabled=false \
    --rollup.config=/var/lib/mantle/data/consensus/rollup.json \
    --rpc.addr=0.0.0.0 \
    --rpc.port=9545 \
    --p2p.priv.path=/var/lib/mantle/data/consensus/priv.key \
    --p2p.listen.tcp=30304 \
    --p2p.listen.udp=30304 \
    --p2p.peerstore.path=/var/lib/mantle/data/consensus/peerstore \
    --p2p.discovery.path=/var/lib/mantle/data/consensus/discovery \
    --p2p.static=/ip4/18.139.118.7/tcp/9003/p2p/16Uiu2HAkywYkvLRUH2MXbD6tSoT3jSMAzTWTp1aDijxpXQXxG6VM \
    --p2p.listen.ip=0.0.0.0 \
    --p2p.scoring.peers=light \
    --p2p.ban.peers=true \
    --p2p.sync.req-resp=true \
    --da.indexer-enable=true \
    --da.indexer-socket=da-indexer-api.sepolia.mantle.xyz:80 \
    --metrics.enabled=true \
    --metrics.addr=0.0.0.0 \
    --metrics.port=9002
pandainzoo commented 4 months ago

can you show the rollup.json of op-node

andreclaro commented 4 months ago

Ah! Forgot to mention that I have also updated the rollup.json to the latest release: https://github.com/mantlenetworkio/networks/blob/main/sepolia/rollup.json

andreclaro commented 4 months ago

Node is syncing without problems (warning, error) or forking... I will provide an update soon. Thanks

andreclaro commented 4 months ago

Node has now recovered... The issue was due to not updating the rollup.json to the latest version.

We can close this issue.