search

ethereum-optimism / developers

This repository is to serve as a place where builders on every level of the OP Stack can come to collaborate.
Creative Commons Zero v1.0 Universal
71 stars 44 forks source link

Optimistically queueing unsafe L2 execution payload #464

Open yuanwj210 opened 5 months ago

yuanwj210 commented 5 months ago

version:op-node-v1.7.3 system:ubuntu20.04

After the update to version 1.7.3, the op-geth is stuck, the op-node shows the following log, and the wait for L1 is very long


t=2024-04-16T01:54:07+0000 lvl=warn msg="failed to serve p2p sync request" serve=payloads_by_number peer=16Uiu2HAku-1938 remote=/ip4/173.231.40.186/tcp/9222 req=118801905 err="peer requested unknown block by number: not found"
t=2024-04-16T01:54:07+0000 lvl=info msg="Received signed execution payload from p2p" id=0x93096ac52bc9b3aeacd2b87e7cdeb10b3c5011c6a13207f23eedb391648dd688:118816835 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:07+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x93096ac52bc9b3aeacd2b87e7cdeb10b3c5011c6a13207f23eedb391648dd688:118816835
t=2024-04-16T01:54:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0x67501cf757210241e214083d2eb32bb3278ba080a657816dd4057764e62c639f:118816836 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:09+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x67501cf757210241e214083d2eb32bb3278ba080a657816dd4057764e62c639f:118816836
t=2024-04-16T01:54:11+0000 lvl=warn msg="failed to serve p2p sync request" serve=payloads_by_number peer=16Uiu2HAku-1938 remote=/ip4/173.231.40.186/tcp/9222 req=118801861 err="peer requested unknown block by number: not found"
t=2024-04-16T01:54:11+0000 lvl=info msg="Received signed execution payload from p2p" id=0xacb719ca8dd6e89fb6873d9d1939114fdcb5f3af20942b140c4fed0ff1d37d95:118816837 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:11+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xacb719ca8dd6e89fb6873d9d1939114fdcb5f3af20942b140c4fed0ff1d37d95:118816837
t=2024-04-16T01:54:13+0000 lvl=info msg="Received signed execution payload from p2p" id=0xaf2748a592d22fc5d4cae777e634c6b110cd6f3a925b39633b0ab05516e75e1e:118816838 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:13+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xaf2748a592d22fc5d4cae777e634c6b110cd6f3a925b39633b0ab05516e75e1e:118816838
yuanwj210 commented 5 months ago

@shazow @mslipper @fgimenez

jwelch-qn commented 5 months ago

I am also seeing this at times. Block height will struggle to stay at tip for a while and the logs are full of

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xdc78579b115e5a967ae549af90ab97b1c22a2d8079a85af3e3640fa0205868b9:118944857

and 

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=error msg="Payload execution failed" block_hash=0x688b7bebbdacc9a26d5fd2b55152ea49f627a730d27980ae20904a359200685f err="context deadline exceeded"

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"

Running version Geth/v1.101311.0-stable-e9a306ba

wangjiangw commented 5 months ago

we have also found this issue, which makes the block syncing latency become bigger ?

jwelch-qn commented 5 months ago

Yes, nodes start to struggle to stay at blockchain height, they generally resolve on their own, but this appears to be a recent behavior.

wangjiangw commented 5 months ago

jwelch-qn

yes, it will recover at some time in the future, and this happens just in recent monthes