search

paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
4.03k stars 1.24k forks source link

Consensus client keeps timeout while requesting reth. #4123

Closed mjm114514 closed 1 year ago

mjm114514 commented 1 year ago

Describe the bug

Consensus client keeps reporting timeout while connecting reth.

Steps to reproduce

  1. Start reth with 
    RUST_LOG=info reth node \
   --authrpc.jwtsecret /mnt/pool0/mainnet/ee-secret/jwtsecret \
   --authrpc.addr 127.0.0.1 \
   --authrpc.port 8551 \
   --datadir /mnt/pool0/mainnet/reth \
   --metrics 0.0.0.0:9001 \
   --ipcpath /mnt/pool0/mainnet/reth/reth.ipc \
  --rpc-max-tracing-requests 50 \
  --rpc-max-connections 200 \
  2. Start lighthouse with 
    lighthouse beacon_node \
  --network mainnet \
  --port 30200 \
  --datadir /mnt/pool-ssd/mainnet/lighthouse \
  --http \
  --http-port 5052 \
  --execution-endpoint http://localhost:8551 \
  --execution-jwt /mnt/pool0/mainnet/ee-secret/jwtsecret \
 --purge-db \
 --disable-deposit-contract-sync \
 --checkpoint-sync-url https://sync-mainnet.beaconcha.in \
 --metrics-address 0.0.0.0 \
 --metrics-port 8008
  3. After reth synced till recent blocks, consensus client keeps reporting timeout, and reth keeps reporting no CL connected.

Node logs

From reth side.

eth-docker-reth-execution-1  | 2023-08-09T05:54:59.173572Z  INFO make_canonical{block_hash=0xde8d7660b3fdfe0c57614ad4bc67b099cf20393f906fb7550ad2cff9fa8c624d}: blockchain_tree: Committing new canonical chain: [(17875312, 0xde8d7660b3fdfe0c57614ad4bc67b099cf20393f906fb7550ad2cff9fa8c624d)]
eth-docker-reth-execution-1  | 2023-08-09T05:55:00.933814Z  INFO net: Session established remote_addr=65.108.198.235:30303 client_version=erigon/v2.48.1-stable-674b77f0/linux-amd64/go1.20.6 peer_id=0x431b7a2bc017ecbb44f9d596f9585d4cfe16f8e641513635a1d912404faa811010a83ed155ec3717174480bb600e7ac3d7c1a45f08bca00341366f3de9172545 total_active=21
eth-docker-reth-execution-1  | 2023-08-09T05:55:03.935165Z  INFO net::session: Received disconnect command for session reason=Some(DisconnectRequested) remote_peer_id=0x602f3b4258dcdb5da73231739955cadda5b3e4f471f15bbf8eb01a8e63728300e6862a443ecd31d5835c7a83ededa45249c7a9ca002e4754d6ada28c30017929
eth-docker-reth-execution-1  | 2023-08-09T05:55:06.488731Z  INFO net::session: Received disconnect command for session reason=Some(DisconnectRequested) remote_peer_id=0xb2a62bd3005ba3205a829b4ad68e695af91d172ce1ced85a20acdb4ef4d1f8e4ea3729f266d5e965b27c49f5947b1c5570f96c602ad0b5d39e50a599ad4d5522
eth-docker-reth-execution-1  | 2023-08-09T05:55:07.174305Z  WARN connection{remote_addr=172.25.0.10:57776 conn_id=111}: jsonrpsee_server::server: HTTP serve connection failed hyper::Error(IncompleteMessage)
eth-docker-reth-execution-1  | 2023-08-09T05:55:07.274686Z  WARN connection{remote_addr=172.25.0.10:57786 conn_id=112}: jsonrpsee_server::server: HTTP serve connection failed hyper::Error(IncompleteMessage)
eth-docker-reth-execution-1  | 2023-08-09T05:55:07.905189Z  INFO reth::cli: Status connected_peers=19 latest_block=17875312
eth-docker-reth-execution-1  | 2023-08-09T05:55:11.045292Z  WARN connection{remote_addr=172.25.0.10:38222 conn_id=113}: jsonrpsee_server::server: HTTP serve connection failed hyper::Error(IncompleteMessage)

From lighthouse side

eth-docker-reth-consensus-1  | Aug 09 05:56:08.919 WARN Error whilst processing payload status  error: Api { error: HttpClient(url: http://execution:8551/, kind: timeout, detail: 
operation timed out) }, service: exec                                                                                                                                              eth-docker-reth-consensus-1  | Aug 09 05:56:08.919 CRIT Failed to update execution head         error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: HttpClient(url: htt
p://execution:8551/, kind: timeout, detail: operation timed out) })), service: beacon                                                                                              
eth-docker-reth-consensus-1  | Aug 09 05:56:16.921 WARN Execution engine call failed            error: HttpClient(url: http://execution:8551/, kind: timeout, detail: operation timed out), service: exec                                                                                                                                                             
eth-docker-reth-consensus-1  | Aug 09 05:56:16.921 WARN Error whilst processing payload status  error: Api { error: HttpClient(url: http://execution:8551/, kind: timeout, detail: 
operation timed out) }, service: exec                                                                                                                                              
eth-docker-reth-consensus-1  | Aug 09 05:56:16.921 CRIT Failed to update execution head         error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: HttpClient(url: htt
p://execution:8551/, kind: timeout, detail: operation timed out) })), service: beacon                                                                                              
eth-docker-reth-consensus-1  | Aug 09 05:56:17.000 INFO Syncing                                 est_time: 2 mins, speed: 0.44 slots/sec, distance: 67 slots (13 mins), peers: 87, s
ervice: slot_notifier                                                                                                                                                              
eth-docker-reth-consensus-1  | Aug 09 05:56:22.609 INFO Sync state updated                      new_state: Syncing Head Chain, old_state: Syncing Finalized Chain, service: sync   
eth-docker-reth-consensus-1  | Aug 09 05:56:29.001 INFO Syncing                                 est_time: 0 mins, speed: 0.92 slots/sec, distance: 45 slots (9 mins), peers: 82, se
rvice: slot_notifier                                                                                                                                                               
eth-docker-reth-consensus-1  | Aug 09 05:56:30.885 WARN Execution engine call failed            error: HttpClient(url: http://execution:8551/, kind: timeout, detail: operation tim
ed out), service: exec                                                                                                                                                             
eth-docker-reth-consensus-1  | Aug 09 05:56:30.885 WARN Error whilst processing payload status  error: Api { error: HttpClient(url: http://execution:8551/, kind: timeout, detail: 
operation timed out) }, service: exec                                                                                                                                              
eth-docker-reth-consensus-1  | Aug 09 05:56:30.885 CRIT Failed to update execution head         error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: HttpClient(url: htt
p://execution:8551/, kind: timeout, detail: operation timed out) })), service: beacon                                                                                              eth-docker-reth-consensus-1  | Aug 09 05:56:41.000 INFO Syncing                                 est_time: 0 mins, speed: 1.46 slots/sec, distance: 20 slots (4 mins), peers: 87, se
rvice: slot_notifier                                                                                                                                                               
eth-docker-reth-consensus-1  | Aug 09 05:56:41.781 INFO New RPC block received                  hash: 0x97a1…5c60, slot: 7061362                                                   
eth-docker-reth-consensus-1  | Aug 09 05:56:53.001 INFO Syncing                                 est_time: 0 mins, speed: 1.08 slots/sec, distance: 18 slots (3 mins), peers: 80, se
rvice: slot_notifier

Platform(s)

Ubuntu x64 Cpu: 2xE5-2678V3 Mem: 64g Disk: 2x raid0 nvme drive.

What version/commit are you on?

reth Version: 0.1.0-alpha.4 Commit SHA: 3f63a088 Build Timestamp: 2023-08-04T05:03:29.743633163Z Build Features: default,jemalloc Build Profile: release

What database version are you on?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

mjm114514 commented 1 year ago

Got some detailed log execution

2023-08-09T12:23:30.415872Z TRACE rpc::engine: Serving engine_newPayloadV2                                                                                                                                                                                                                                                                                  [10/4538]
2023-08-09T12:23:33.441544Z TRACE rpc::engine: Serving engine_exchangeTransitionConfigurationV1                                                                                                                                                                                                                                                                      
2023-08-09T12:23:33.849857Z TRACE rpc::engine: Serving engine_newPayloadV2                                                                                                                                                                                                                                                                                           
2023-08-09T12:23:33.972195Z  INFO try_insert_validated_block{block=(17877260, 0xed24fe9b785f35ee5ab42642044378d21ea22353b66fe7dbd8bff62ccc5330f5)}: blockchain_tree: return=Ok(Valid)                                                                                                                                                                                
2023-08-09T12:23:33.972315Z  WARN make_canonical{block_hash=0x1ddbdf2e70946aecade21fd061ec997086c87b7d1b214be23b3859cfa95a7320}: blockchain_tree: Block hash not found in block indices block_hash=0x1ddbdf2e70946aecade21fd061ec997086c87b7d1b214be23b3859cfa95a7320                                                                                                
2023-08-09T12:23:34.215396Z TRACE rpc::engine: Serving engine_newPayloadV2                                                                                                                                                                                                                                                                                           
2023-08-09T12:23:35.438121Z TRACE rpc::engine: Serving engine_forkchoiceUpdatedV2                                                                                                                                                                                                                                                                                    
2023-08-09T12:23:38.290985Z  INFO reth::cli: Status connected_peers=23 latest_block=17877019                                                                                                                                                                                                                                                                         
2023-08-09T12:23:40.153000Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877262, 0x31c81a2ce05779e0d8261ca05ca949e67752b5b8935da1b6955f7c847e8f820d)}: blockchain_tree: return=Ok(Valid)                              
2023-08-09T12:23:43.563440Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877263, 0x9386143b59ecd1477180bffbf723fbef28e44168821f956566cbd3ce0455deab)}: blockchain_tree: return=Ok(Valid)                              
2023-08-09T12:23:47.047007Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877264, 0x2f8c2029da4e2828720038d6cb704d0a5a5592ba554d25d47d6f6ceaba409344)}: blockchain_tree: return=Ok(Valid)                              
2023-08-09T12:23:50.070097Z  INFO net: Session established remote_addr=150.136.20.117:30303 client_version=Nethermind/v1.19.2+42ba08b7/linux-x64/dotnet7.0.5 peer_id=0xc8510d4a2c3dd7ec00ebad4b9d2add9490cb5186f2f8f1068706a2eb16e18328417fe377eeefb75f9d94ccdcaa095f0795e57d03283b85f1d544ff0593b6cf3d total_active=23
2023-08-09T12:23:50.168476Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877265, 0xed6bdfb02068a3c560c83ee4938d265ac8eb0ad5f0926414c1601d46b1022f4b)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:23:53.539799Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877266, 0x0f0ef52b019847445b85e13ea0543e83cbacfd66dc2abd0480d4ed6d42b0a1b7)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:23:53.553809Z  INFO net: Session established remote_addr=89.31.47.252:38956 client_version=erigon/v2.43.1/linux-amd64/go1.20.1 peer_id=0x7206939a95a94d5ea67725e4015f3ec6125f225689e1cf0b3cd6739e91344990820f84e58aec84b484f5f6a059aa45c5a074996fe42f1f510f11ac2d315a7e88 total_active=24
2023-08-09T12:23:55.383347Z  INFO net: Session established remote_addr=68.84.55.89:30303 client_version=besu/v23.4.0/linux-x86_64/openjdk-java-17 peer_id=0xf163930451bed951c94faf4215c36975d0efd0e1badbad4d52c6712e785f4f10d931695312fc2db196411717b139165a3db006f6bf1f0f645b73906abb9826cd total_active=25
2023-08-09T12:23:55.968349Z TRACE rpc::engine: Serving engine_newPayloadV2
2023-08-09T12:23:58.010900Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877267, 0x2c967dc7b7c9f2ce98096a8b0db99e9164a8a0c1a7b151f58519021081e9ced9)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:00.364816Z  INFO net: Session established remote_addr=178.196.11.54:30303 client_version=besu/v23.4.4/linux-x86_64/openjdk-java-17 peer_id=0xa1efd5d4362377ac86e352682cb28f19a7d32bcfb2efa4dd01199dc29a9ec44fe9d547247f715b35a0f1b08ef1787f504d2bae0a175eaf2b73acd1eddd352aee total_active=25
2023-08-09T12:24:01.578525Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877268, 0xdb074aec94ebf2aaeb0273db1e5cf102ea2a2f50d1365785d99626f4c2e2f21a)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:05.135347Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877269, 0xeafaa7c5fd2b99314e9f3b7895f324a0faaa77309d3fe2e573820747ff952907)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:08.290715Z  INFO reth::cli: Status connected_peers=25 latest_block=17877019
2023-08-09T12:24:09.031668Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877270, 0xef722b3f126c8f1b3709faef2c40df215a0d22faca36cda90fb404fae2761df2)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:09.182433Z TRACE rpc::engine: Serving engine_newPayloadV2
2023-08-09T12:24:12.465937Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877271, 0xbfafd3a614795ade3708235562acd3148a1fb350ada29c0bca81ca3505601a70)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:15.692353Z  INFO net: Session established remote_addr=34.198.161.50:32952 client_version=Geth/v1.12.0-stable-e501b3b0/linux-amd64/go1.20.3 peer_id=0x4fa207af80cf251a929399eea5190ec70cef4d9c8e0b398344441d0fd61ea8e45732944388e00a9d58904ff2251deaae6b876b73a61cb0a1dbba9ca3f81e67cb total_active=25
2023-08-09T12:24:16.355062Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877272, 0xa8bf22466d01eb1f4a9d23bdcd25e716b9683ac04d87d56087e0695600abed90)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:17.981579Z  INFO net: Session established remote_addr=97.87.138.152:30303 client_version=besu/v23.4.4/linux-x86_64/openjdk-java-17 peer_id=0xa758637a490663ddc348ab9ddefb9f9b8ec556d18dac0d3bff4c90eaaf1256a58e262c64a386343a7d4b9f0cd9261acf2f25d7c76a72ba50d744437a97cc4ec8 total_active=26
2023-08-09T12:24:21.807882Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877273, 0x6fb29c22d16282ffb67e2a7224f23dd6893ed5f54c84c7154a26f9b697d4175f)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:22.808820Z  INFO net: Session established remote_addr=5.161.195.102:30313 client_version= peer_id=0x93ed01e41cc6872faa9873394d7a2cd274bc2a122c01722c4015b279e2836b31172a141efc415aadab72cc37b264feb601ee79ce246c38dd96e14a2b377d487f total_active=26
2023-08-09T12:24:25.130826Z TRACE rpc::engine: Serving engine_newPayloadV2
2023-08-09T12:24:26.245091Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877274, 0xd3e8d8fd13f2a1613b61c89bcfcd54a173357a754648651dfa6d54e3864c6768)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:30.043138Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877275, 0xec42a6eeec2ac01dd7d167691ae9c606b6f636790a0f100f9a99e3b886599fac)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:32.564295Z  INFO net: Session established remote_addr=13.37.220.1:30315 client_version= peer_id=0x21c3fbcbd40ad7d0bb739704a78df2f9c613171440409b49f07ae76d3fc8897a94ea44bfbaf8e8c994ee21056000af3d8b0e260d1405cc9268bf4abf26028f58 total_active=27
2023-08-09T12:24:33.443412Z TRACE rpc::engine: Serving engine_exchangeTransitionConfigurationV1
2023-08-09T12:24:33.859373Z  INFO try_insert_validated_block{block=(17877261, 0x8e5365a918a2100d538126a8b27508c6e79d6a8943478de0cf35f69740c4b01b)}:try_insert_block_into_side_chain:try_insert_validated_block{block=(17877276, 0xcb08fbbf1e653af5584671043db581dc4e5569284e9948b4fe4d544db67f4f72)}: blockchain_tree: return=Ok(Valid)
2023-08-09T12:24:37.756586Z  INFO net: Session established remote_addr=35.224.170.50:1024 client_version=Geth/v1.11.5-stable-a38f4108/linux-amd64/go1.20.2 peer_id=0xb1d64fa9f3e0b80cd28e824e9f10b00d525d834cd22f838725323dcc8a0217947c598191da770fbbdf203d444624aeab6fc6883a26a57093ce57aa13f09caa3a total_active=28
2023-08-09T12:24:37.799881Z  INFO net: Session established remote_addr=194.163.177.216:30303 client_version=Nethermind/v1.20.1+9f39c0c7/linux-x64/dotnet7.0.9 peer_id=0xd15dd899dbe2c1e707fc71f1f3214c8c356ffadbde9fd89a2ad59367dbea8807dc3fcb972eccbd2b553e0d83f08b72480ddd97e3b096a4380b4cdaff152dd355 total_active=29
2023-08-09T12:24:38.005809Z TRACE rpc::engine: Serving engine_newPayloadV2
2023-08-09T12:24:38.290053Z  INFO reth::cli: Status connected_peers=29 latest_block=17877019

consensus

Aug 09 12:23:43.439 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec                     
Aug 09 12:23:43.439 CRIT Failed to update execution head         error: ExecutionForkChoiceUpdateFailed(EngineError(Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: Timed
Out }) })), service: beacon                                                                                                                                                                                                                                                                                                                                          
Aug 09 12:23:53.000 INFO Downloading historical blocks           est_time: 1 week 0 days, speed: 10.67 slots/sec, distance: 7055073 slots (139 weeks 6 days), service: slot_notifier                                                                                                                                                                                 
Aug 09 12:23:53.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier                                                                            
Aug 09 12:23:53.000 INFO Synced                                  slot: 7063317, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 77, service: slot_notifier                                                                                                                      
Aug 09 12:24:03.968 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Aug 09 12:24:03.968 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Aug 09 12:24:05.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:24:05.000 INFO Synced                                  slot: 7063318, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 76, service: slot_notifier
Aug 09 12:24:17.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:24:17.000 INFO Synced                                  slot: 7063319, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 77, service: slot_notifier
Aug 09 12:24:17.182 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Aug 09 12:24:17.182 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Aug 09 12:24:29.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:24:29.000 INFO Synced                                  slot: 7063320, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 77, service: slot_notifier
Aug 09 12:24:33.129 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Aug 09 12:24:33.129 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Aug 09 12:24:41.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:24:41.000 INFO Synced                                  slot: 7063321, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 77, service: slot_notifier
Aug 09 12:24:46.005 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Aug 09 12:24:46.005 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Aug 09 12:24:53.001 INFO Downloading historical blocks           est_time: 1 week 3 days, speed: 8.00 slots/sec, distance: 7054497 slots (139 weeks 6 days), service: slot_notifier
Aug 09 12:24:53.001 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:24:53.001 INFO Synced                                  slot: 7063322, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 79, service: slot_notifier
Aug 09 12:24:57.971 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Aug 09 12:24:57.971 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Aug 09 12:25:05.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:25:05.000 INFO Synced                                  slot: 7063323, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 78, service: slot_notifier
Aug 09 12:25:10.326 WARN Execution engine call failed            error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Aug 09 12:25:10.326 WARN Error whilst processing payload status  error: Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }) }, service: exec
Aug 09 12:25:17.000 WARN Head is optimistic                      execution_block_hash: 0x38d0e66587763a7d85ebb3e82cc6de0410fe9f84b49ebdbfc1299dd2e3a181fa, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Aug 09 12:25:17.001 INFO Synced                                  slot: 7063324, block:    …  empty, epoch: 220728, finalized_epoch: 220726, finalized_root: 0xd313…914c, exec_hash: 0x38d0…81fa (unverified), peers: 79, service: slot_notifier

It looks like stuck on newPayloadV2 engine API.

mjm114514 commented 1 year ago

Never mind, it is an IO issue, when I change from ZFS to ext4, everything works fine....

quickchase commented 6 months ago

Maybe Related #7233