search

aptos-labs / aptos-ait3

6 stars 9 forks source link

Fullnode doesn't sync #62

Open CrNeppos opened 2 years ago

CrNeppos commented 2 years ago

My VFN is stuck at zero, restart doesn't help

Fullnode:

# HELP aptos_state_sync_version The versions processed by the storage synchronizer operations
# TYPE aptos_state_sync_version gauge
aptos_state_sync_version{type="applied_transaction_outputs"} 0
aptos_state_sync_version{type="executed_transactions"} 0
aptos_state_sync_version{type="synced"} 0
aptos_state_sync_version{type="synced_epoch"} 1

Fullnode logs:

client bd706701: client's self-reported peer id and pubkey-derived peer id don't match: self-reported: bd7067011ca4bb3702404c3270329ebf4f3e0d82c4e83d31ce739d447c721fcb, derived: 53c9c7ae5d1d9cc643413beacca13c62deedcd840db34fd93bb0bee8ec11bb3a","network_address":"/ip4/54.158.198.7/tcp/21994","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"}}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:04.008831Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:05.158410Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:07.008293Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:07.158449Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:09.158553Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:10.008353Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:11.158620Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:582"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:12.373552Z","message":"[full_node,Public,0257d89a] Dialing peer 07d429d2 at /ip4/34.64.49.220/tcp/6182/noise-ik/0x44fd1324c66371b4788af0b901c9eb8088781acb29e6b8b9c791d5d9838fbe1f/handshake/0","data":{"network_address":"/ip4/34.64.49.220/tcp/6182/noise-ik/0x44fd1324c66371b4788af0b901c9eb8088781acb29e6b8b9c791d5d9838fbe1f/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"07d429d25c23497040431c6ddfde65dba6dbb8702876fb1f060f00bd435420a9"}}
{"level":"INFO","source":{"package":"network","file":"network/src/peer_manager/mod.rs:238"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:12.373589Z","message":"Current connected peers","data":{"network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"peers":"[]"}}
{"level":"ERROR","source":{"package":"network","file":"network/src/peer_manager/transport.rs:258"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:12.898608Z","message":"[full_node,Public,0257d89a] Outbound connection failed for peer 07d429d2 at /ip4/34.64.49.220/tcp/6182/noise-ik/0x44fd1324c66371b4788af0b901c9eb8088781acb29e6b8b9c791d5d9838fbe1f/handshake/0: Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","data":{"error":"Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","network_address":"/ip4/34.64.49.220/tcp/6182/noise-ik/0x44fd1324c66371b4788af0b901c9eb8088781acb29e6b8b9c791d5d9838fbe1f/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"07d429d25c23497040431c6ddfde65dba6dbb8702876fb1f060f00bd435420a9"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:869"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:12.898661Z","message":"[full_node,Public,0257d89a] Failed to connect to peer: 07d429d2 at address: /ip4/34.64.49.220/tcp/6182/noise-ik/0x44fd1324c66371b4788af0b901c9eb8088781acb29e6b8b9c791d5d9838fbe1f/handshake/0; error: Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","data":{"error":"Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","network_address":"/ip4/34.64.49.220/tcp/6182/noise-ik/0x44fd1324c66371b4788af0b901c9eb8088781acb29e6b8b9c791d5d9838fbe1f/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"07d429d25c23497040431c6ddfde65dba6dbb8702876fb1f060f00bd435420a9"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:13.008909Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:13.158833Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:15.158846Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:16.008102Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:17.158765Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:19.008900Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:19.158737Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:21.158917Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"ERROR","source":{"package":"network","file":"network/src/peer_manager/transport.rs:258"},"thread_name":"network-vfn","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:21.429651Z","message":"[full_node,vfn,66fa17ac] Outbound connection failed for peer 00000000 at /ip4/159.69.24.237/tcp/6181/noise-ik/0xf0274c2774519281a8332d0bb9d8101bd58bc7bb154b38039bc9096ce04e1237/handshake/0: Transport error: timeout elapsed","data":{"error":"Transport error: timeout elapsed","network_address":"/ip4/159.69.24.237/tcp/6181/noise-ik/0xf0274c2774519281a8332d0bb9d8101bd58bc7bb154b38039bc9096ce04e1237/handshake/0","network_context":{"role":"full_node","network_id":"vfn","peer_id":"66fa17ace89578178c497cda0709e9514d919daa0445c04b39cbf39b99747935"},"remote_peer":"00000000000000000000000000000000d58bc7bb154b38039bc9096ce04e1237"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:869"},"thread_name":"network-vfn","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:21.429703Z","message":"[full_node,vfn,66fa17ac] Failed to connect to peer: 00000000 at address: /ip4/159.69.24.237/tcp/6181/noise-ik/0xf0274c2774519281a8332d0bb9d8101bd58bc7bb154b38039bc9096ce04e1237/handshake/0; error: Transport error: timeout elapsed","data":{"error":"Transport error: timeout elapsed","network_address":"/ip4/159.69.24.237/tcp/6181/noise-ik/0xf0274c2774519281a8332d0bb9d8101bd58bc7bb154b38039bc9096ce04e1237/handshake/0","network_context":{"role":"full_node","network_id":"vfn","peer_id":"66fa17ace89578178c497cda0709e9514d919daa0445c04b39cbf39b99747935"},"remote_peer":"00000000000000000000000000000000d58bc7bb154b38039bc9096ce04e1237"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:22.008961Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"network","file":"network/src/peer_manager/transport.rs:309"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:22.051428Z","message":"[full_node,Public,0257d89a] Inbound connection from /ip4/94.130.17.248/tcp/38794 failed to upgrade after 0.001 secs: noise server: client e6d895bb: client's self-reported peer id and pubkey-derived peer id don't match: self-reported: e6d895bbef224ca6e24208cecc174784e39cf626def01374a8f9833e79183dd2, derived: c8734b1aafdb108d32e0e0043540b0bbcb326da3b7e531561e6673c387ba2926","data":{"error":"noise server: client e6d895bb: client's self-reported peer id and pubkey-derived peer id don't match: self-reported: e6d895bbef224ca6e24208cecc174784e39cf626def01374a8f9833e79183dd2, derived: c8734b1aafdb108d32e0e0043540b0bbcb326da3b7e531561e6673c387ba2926","network_address":"/ip4/94.130.17.248/tcp/38794","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"}}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:23.158317Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:582"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:23.413983Z","message":"[full_node,Public,0257d89a] Dialing peer f326fd30 at /ip4/34.151.231.150/tcp/6182/noise-ik/0x2f807b7a337a1f42190555fd8b67346701c9ef93142146c0143a14f4792d2a2f/handshake/0","data":{"network_address":"/ip4/34.151.231.150/tcp/6182/noise-ik/0x2f807b7a337a1f42190555fd8b67346701c9ef93142146c0143a14f4792d2a2f/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"f326fd3007939a18e96d9cbb68da732e06a160759ae12b1fe3d4bc0ef7066a81"}}
{"level":"ERROR","source":{"package":"network","file":"network/src/peer_manager/transport.rs:258"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:23.820664Z","message":"[full_node,Public,0257d89a] Outbound connection failed for peer f326fd30 at /ip4/34.151.231.150/tcp/6182/noise-ik/0x2f807b7a337a1f42190555fd8b67346701c9ef93142146c0143a14f4792d2a2f/handshake/0: Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","data":{"error":"Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","network_address":"/ip4/34.151.231.150/tcp/6182/noise-ik/0x2f807b7a337a1f42190555fd8b67346701c9ef93142146c0143a14f4792d2a2f/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"f326fd3007939a18e96d9cbb68da732e06a160759ae12b1fe3d4bc0ef7066a81"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:869"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:23.820720Z","message":"[full_node,Public,0257d89a] Failed to connect to peer: f326fd30 at address: /ip4/34.151.231.150/tcp/6182/noise-ik/0x2f807b7a337a1f42190555fd8b67346701c9ef93142146c0143a14f4792d2a2f/handshake/0; error: Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","data":{"error":"Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","network_address":"/ip4/34.151.231.150/tcp/6182/noise-ik/0x2f807b7a337a1f42190555fd8b67346701c9ef93142146c0143a14f4792d2a2f/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"f326fd3007939a18e96d9cbb68da732e06a160759ae12b1fe3d4bc0ef7066a81"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:25.008351Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:25.158445Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:27.158335Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:28.008988Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:29.158150Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:31.008601Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:31.158276Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:626"},"thread_name":"network-vfn","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:31.357493Z","message":"Current eligible peers","data":{"discovered_peers":"DiscoveredPeerSet({00000000000000000000000000000000d58bc7bb154b38039bc9096ce04e1237: DiscoveredPeer { role: validator, addrs: [[], [], [/ip4/159.69.24.237/tcp/6181/noise-ik/0xf0274c2774519281a8332d0bb9d8101bd58bc7bb154b38039bc9096ce04e1237/handshake/0]], keys: [{}, {}, {x25519::PublicKey(f0274c2774519281a8332d0bb9d8101bd58bc7bb154b38039bc9096ce04e1237)}], last_dial_time: SystemTime { tv_sec: 1662670586, tv_nsec: 357403788 } }})","network_context":{"role":"full_node","network_id":"vfn","peer_id":"66fa17ace89578178c497cda0709e9514d919daa0445c04b39cbf39b99747935"}}}
{"level":"INFO","source":{"package":"aptos_mempool","file":"mempool/src/shared_mempool/coordinator.rs:342"},"thread_name":"shared-mem","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:31.388424Z","data":{"event":"live","name":"g_c_runtime"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:582"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:32.421153Z","message":"[full_node,Public,0257d89a] Dialing peer ff79fc4b at /ip4/95.217.88.91/tcp/6182/noise-ik/0xbc6dbdfa4be873536e7394284074996805ad174d92f54f6a0b88c8d6f5131d65/handshake/0","data":{"network_address":"/ip4/95.217.88.91/tcp/6182/noise-ik/0xbc6dbdfa4be873536e7394284074996805ad174d92f54f6a0b88c8d6f5131d65/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"ff79fc4bf0b39c4b8a9810b57d311527700639b661198f4d8db77d7c6d64f0c9"}}
{"level":"ERROR","source":{"package":"network","file":"network/src/peer_manager/transport.rs:258"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:32.469942Z","message":"[full_node,Public,0257d89a] Outbound connection failed for peer ff79fc4b at /ip4/95.217.88.91/tcp/6182/noise-ik/0xbc6dbdfa4be873536e7394284074996805ad174d92f54f6a0b88c8d6f5131d65/handshake/0: Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","data":{"error":"Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","network_address":"/ip4/95.217.88.91/tcp/6182/noise-ik/0xbc6dbdfa4be873536e7394284074996805ad174d92f54f6a0b88c8d6f5131d65/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"ff79fc4bf0b39c4b8a9810b57d311527700639b661198f4d8db77d7c6d64f0c9"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:869"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:32.469991Z","message":"[full_node,Public,0257d89a] Failed to connect to peer: ff79fc4b at address: /ip4/95.217.88.91/tcp/6182/noise-ik/0xbc6dbdfa4be873536e7394284074996805ad174d92f54f6a0b88c8d6f5131d65/handshake/0; error: Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","data":{"error":"Transport error: noise client: error reading server handshake response message, server probably rejected our handshake message: unexpected end of file","network_address":"/ip4/95.217.88.91/tcp/6182/noise-ik/0xbc6dbdfa4be873536e7394284074996805ad174d92f54f6a0b88c8d6f5131d65/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"ff79fc4bf0b39c4b8a9810b57d311527700639b661198f4d8db77d7c6d64f0c9"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:33.158260Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:582"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:33.401747Z","message":"[full_node,Public,0257d89a] Dialing peer bfeac9b2 at /ip4/65.109.21.207/tcp/6182/noise-ik/0xa9909fab9fe47d2f9409e95555f4cb1634f2964e214a3e91b60f3a7d96824e66/handshake/0","data":{"network_address":"/ip4/65.109.21.207/tcp/6182/noise-ik/0xa9909fab9fe47d2f9409e95555f4cb1634f2964e214a3e91b60f3a7d96824e66/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"bfeac9b2da698dee2d9b882322d20dc7508bb6cf2df4165c8938ffd748e11337"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:34.008375Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:35.158318Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:37.008985Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:37.158214Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:39.158245Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"data_streaming_service","file":"state-sync/state-sync-v2/data-streaming-service/src/streaming_service.rs:246"},"thread_name":"data-streaming-service","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:40.008249Z","data":{"message":"Latest global data summary is empty.","name":"refresh_global_data"}}
{"level":"WARN","source":{"package":"aptos_data_client","file":"state-sync/aptos-data-client/src/aptosnet/mod.rs:704"},"thread_name":"aptos-data-client","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:41.158194Z","data":{"error":"DataIsUnavailable(\"No connected AptosNet peers!\")","event":"peer_polling_error","message":"Unable to fetch peers to poll!","name":"storage_summary_request"}}
{"level":"INFO","source":{"package":"network","file":"network/src/connectivity_manager/mod.rs:582"},"thread_name":"network-Public","hostname":"dd06797f8234","timestamp":"2022-09-08T20:56:42.377057Z","message":"[full_node,Public,0257d89a] Dialing peer 8d7766a6 at /dns/ait3-fullnode.polkachu.com/tcp/6182/noise-ik/0x1059b842b505b72eaf6cf3e898536c7fab73855c26b64725eb0ddc3fdeea8227/handshake/0","data":{"network_address":"/dns/ait3-fullnode.polkachu.com/tcp/6182/noise-ik/0x1059b842b505b72eaf6cf3e898536c7fab73855c26b64725eb0ddc3fdeea8227/handshake/0","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"},"remote_peer":"8d7766a61a01a33199db8412362805e9550d7dcdce6ab74fecb2ddf9899f74d0"}}
holms2 commented 2 years ago

this issue has been discussed many times!))) solution is here: https://github.com/aptos-labs/aptos-ait3/issues/60#issuecomment-1240544855

CrNeppos commented 2 years ago

this issue has been discussed many times!))) solution is here: #60 (comment)

Despite being discussed so many times, typical solution doesn't help me to overcome the problem

JoshLind commented 2 years ago

It appears your node is unable to connect to any peers in the network Unable to fetch peers to poll!.

  1. What is your owner peer id?
  2. Can you provide more logs?

At the top of the logs, I see:

client's self-reported peer id and pubkey-derived peer id don't match: self-reported: bd7067011ca4bb3702404c3270329ebf4f3e0d82c4e83d31ce739d447c721fcb, derived: 53c9c7ae5d1d9cc643413beacca13c62deedcd840db34fd93bb0bee8ec11bb3a","network_address":"/ip4/54.158.198.7/tcp/21994","network_context":{"role":"full_node","network_id":"Public","peer_id":"0257d89a9b63cc904654f132f8f7876530c1fc9f7b133588768ad5e71b958a13"}}}

which suggests to me there is a key mismatch in your deployment. Did you try and rotate your keys/addresses at all?