search

cosmos / gaia

Cosmos Hub
https://hub.cosmos.network
Apache License 2.0
465 stars 679 forks source link

[Bug]: After upgrading to v19.1.0 Node is throwing peer issue #3352

Open mayank-daga opened 1 week ago

mayank-daga commented 1 week ago

Is there an existing issue for this?

What happened?

Hi I am getting following logs after upgrading to v19.1.0

6:57AM INF starting node with ABCI CometBFT in-process module=server 6:57AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service" 6:57AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service" 6:57AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service" 6:57AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service" 6:57AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service" 6:57AM INF service start impl=EventBus module=events msg="Starting EventBus service" 6:57AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service" 7:03AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service" 7:03AM INF ABCI Handshake App Info hash=96977406B9BCC19F78614EAFB8E9FE6368A780668E51C586FADBCE1EA2CD58E7 height=22004611 module=consensus protocol-version=0 software-version=HEAD-4106e7a673da18b518fd81231a8e8b99bbf0fd0d 7:03AM INF ABCI Replay Blocks appHeight=22004611 module=consensus stateHeight=22004611 storeHeight=22004611 7:03AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=96977406B9BCC19F78614EAFB8E9FE6368A780668E51C586FADBCE1EA2CD58E7 appHeight=22004611 module=consensus 7:03AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.11 7:03AM INF This node is not a validator addr=46D05282E6580EBAD176AA6B5F287FC60F39977C module=consensus pubKey=PubKeyEd25519{9D82E4177952344DFB862689F4FCA8A7FF8E4015526C80AE5C959DB86FEE20EF} 7:03AM INF P2P Node ID ID=92ac45d8a7d21907382b832c57dc1201738b237c file=/gaia/.gaia/config/node_key.json module=p2p 7:03AM INF Adding persistent peers addrs=["4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090","d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456","f527c58fb1c30eca8c4226edd7495c9bc2ec69b5@46.4.80.48:14956","f956e0b6368d9712f173a8459679fd4413a3a3d1@192.69.220.226:26656"] module=p2p 7:03AM INF Adding unconditional peer ids ids=[] module=p2p 7:03AM INF Add our address to book addr=92ac45d8a7d21907382b832c57dc1201738b237c@0.0.0.0:26656 book=/gaia/.gaia/config/addrbook.json module=p2p 7:03AM INF service start impl=Node module=server msg="Starting Node service" 7:03AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service" 7:03AM INF serve module=rpc-server msg="Starting RPC HTTP server on [::]:26657" 7:03AM INF service start impl=PEX module=pex msg="Starting PEX service" 7:03AM INF service start book=/gaia/.gaia/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service" 7:03AM INF service start impl=Mempool module=mempool msg="Starting Mempool service" 7:03AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service" 7:03AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service" 7:03AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service" 7:03AM INF Reactor module=consensus waitSync=true 7:03AM INF service start impl=Evidence module=evidence msg="Starting Evidence service" 7:03AM INF service start impl=StateSync module=statesync msg="Starting StateSync service" 7:03AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10 7:03AM INF Saving AddrBook to file book=/gaia/.gaia/config/addrbook.json module=p2p size=1241 7:03AM INF service start impl="Peer{MConn{51.210.118.45:26656} a9380db8758f9b78cbe4253ccdacc992cf80f6f6 out}" module=p2p msg="Starting Peer service" peer=a9380db8758f9b78cbe4253ccdacc992cf80f6f6@51.210.118.45:26656 7:03AM INF service start impl=MConn{51.210.118.45:26656} module=p2p msg="Starting MConnection service" peer=a9380db8758f9b78cbe4253ccdacc992cf80f6f6@51.210.118.45:26656 7:03AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{51.210.118.45:26656} module=p2p peer=a9380db8758f9b78cbe4253ccdacc992cf80f6f6@51.210.118.45:26656 7:03AM INF service stop impl=MConn{51.210.118.45:26656} module=p2p msg="Stopping MConnection service" peer=a9380db8758f9b78cbe4253ccdacc992cf80f6f6@51.210.118.45:26656 7:03AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{51.210.118.45:26656} a9380db8758f9b78cbe4253ccdacc992cf80f6f6 out}" 7:03AM INF service stop impl="Peer{MConn{51.210.118.45:26656} a9380db8758f9b78cbe4253ccdacc992cf80f6f6 out}" module=p2p msg="Stopping Peer service" peer=a9380db8758f9b78cbe4253ccdacc992cf80f6f6@51.210.118.45:26656 7:03AM INF service start impl="Peer{MConn{176.9.101.44:26656} 505f4467926cdad29932c44dc5ea7a5da6982f48 out}" module=p2p msg="Starting Peer service" peer=505f4467926cdad29932c44dc5ea7a5da6982f48@176.9.101.44:26656 7:03AM INF service start impl=MConn{176.9.101.44:26656} module=p2p msg="Starting MConnection service" peer=505f4467926cdad29932c44dc5ea7a5da6982f48@176.9.101.44:26656 7:03AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{176.9.101.44:26656} module=p2p peer=505f4467926cdad29932c44dc5ea7a5da6982f48@176.9.101.44:26656 7:03AM INF service stop impl=MConn{176.9.101.44:26656} module=p2p msg="Stopping MConnection service" peer=505f4467926cdad29932c44dc5ea7a5da6982f48@176.9.101.44:26656 7:03AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{176.9.101.44:26656} 505f4467926cdad29932c44dc5ea7a5da6982f48 out}" 7:03AM INF service stop impl="Peer{MConn{176.9.101.44:26656} 505f4467926cdad29932c44dc5ea7a5da6982f48 out}" module=p2p msg="Stopping Peer service" peer=505f4467926cdad29932c44dc5ea7a5da6982f48@176.9.101.44:26656 7:03AM INF service start impl="Peer{MConn{37.27.56.238:5000} e6d729935d77865bf79fc051b2a9d1a263f1fcdc out}" module=p2p msg="Starting Peer service" peer=e6d729935d77865bf79fc051b2a9d1a263f1fcdc@37.27.56.238:5000 7:03AM INF service start impl=MConn{37.27.56.238:5000} module=p2p msg="Starting MConnection service" peer=e6d729935d77865bf79fc051b2a9d1a263f1fcdc@37.27.56.238:5000 7:03AM INF service start impl="Peer{MConn{18.143.53.64:26656} c5ccf0aaf81e5e523a56bd676822c9f73abcb833 out}" module=p2p msg="Starting Peer service" peer=c5ccf0aaf81e5e523a56bd676822c9f73abcb833@18.143.53.64:26656 7:03AM INF service start impl=MConn{18.143.53.64:26656} module=p2p msg="Starting MConnection service" peer=c5ccf0aaf81e5e523a56bd676822c9f73abcb833@18.143.53.64:26656 7:03AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{18.143.53.64:26656} module=p2p peer=c5ccf0aaf81e5e523a56bd676822c9f73abcb833@18.143.53.64:26656 7:03AM INF service stop impl=MConn{18.143.53.64:26656} module=p2p msg="Stopping MConnection service" peer=c5ccf0aaf81e5e523a56bd676822c9f73abcb833@18.143.53.64:26656 7:03AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{18.143.53.64:26656} c5ccf0aaf81e5e523a56bd676822c9f73abcb833 out}" 7:03AM INF service stop impl="Peer{MConn{18.143.53.64:26656} c5ccf0aaf81e5e523a56bd676822c9f73abcb833 out}" module=p2p msg="Stopping Peer service" peer=c5ccf0aaf81e5e523a56bd676822c9f73abcb833@18.143.53.64:26656 7:03AM INF service start impl="Peer{MConn{46.4.80.48:14956} f527c58fb1c30eca8c4226edd7495c9bc2ec69b5 out}" module=p2p msg="Starting Peer service" peer=f527c58fb1c30eca8c4226edd7495c9bc2ec69b5@46.4.80.48:14956 7:03AM INF service start impl=MConn{46.4.80.48:14956} module=p2p msg="Starting MConnection service" peer=f527c58fb1c30eca8c4226edd7495c9bc2ec69b5@46.4.80.48:14956 7:03AM INF starting gRPC server... address=0.0.0.0:9090 module=grpc-server 7:03AM INF starting API server... address=tcp://0.0.0.0:1317 module=api-server 7:03AM INF serve module=api-server msg="Starting RPC HTTP server on [::]:1317" 7:03AM ERR Error dialing peer err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p 7:03AM INF Reconnecting to peer addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 module=p2p 7:03AM ERR Error dialing peer err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p 7:03AM INF Reconnecting to peer addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 module=p2p 7:03AM INF service start impl="Peer{MConn{192.69.220.226:26656} f956e0b6368d9712f173a8459679fd4413a3a3d1 out}" module=p2p msg="Starting Peer service" peer=f956e0b6368d9712f173a8459679fd4413a3a3d1@192.69.220.226:26656 7:03AM INF service start impl=MConn{192.69.220.226:26656} module=p2p msg="Starting MConnection service" peer=f956e0b6368d9712f173a8459679fd4413a3a3d1@192.69.220.226:26656 7:03AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=0 7:03AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=0 7:03AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=1 7:03AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=1 7:03AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=2 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=2 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=3 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=3 7:04AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=3 numToDial=7 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=4 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=4 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=5 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=5 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=6 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=6 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=7 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=7 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=8 7:04AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=3 numToDial=7 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=8 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=9 7:04AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=9 7:04AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=10 7:05AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=11 7:05AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=10 7:05AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=12 7:05AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=11 7:05AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=3 numToDial=7 7:05AM INF service start impl="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}" module=p2p msg="Starting Peer service" peer=cfd785a4224c7940e9a10f6c1ab24c343e923bec@164.68.107.188:26656 7:05AM INF service start impl=MConn{164.68.107.188:26656} module=p2p msg="Starting MConnection service" peer=cfd785a4224c7940e9a10f6c1ab24c343e923bec@164.68.107.188:26656 7:05AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{164.68.107.188:26656} module=p2p peer=cfd785a4224c7940e9a10f6c1ab24c343e923bec@164.68.107.188:26656 7:05AM INF service stop impl=MConn{164.68.107.188:26656} module=p2p msg="Stopping MConnection service" peer=cfd785a4224c7940e9a10f6c1ab24c343e923bec@164.68.107.188:26656 7:05AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}" 7:05AM INF service stop impl="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}" module=p2p msg="Stopping Peer service" peer=cfd785a4224c7940e9a10f6c1ab24c343e923bec@164.68.107.188:26656 7:05AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=13 7:05AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=12 7:05AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=14 7:05AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=13 7:05AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=15 7:05AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=14 7:05AM INF Error reconnecting to peer. Trying again addr=d9bfa29e0cf9c4ce0cc9c26d98e5d97228f93b0b@37.27.61.38:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=16 7:05AM INF Saving AddrBook to file book=/gaia/.gaia/config/addrbook.json module=p2p size=1241 7:05AM INF Ensure peers module=pex numDialing=1 numInPeers=0 numOutPeers=3 numToDial=6 7:05AM INF service start impl="Peer{MConn{74.118.140.203:26656} 66248ea4d70c4b8fd6c9ba848895a0332a3a345e out}" module=p2p msg="Starting Peer service" peer=66248ea4d70c4b8fd6c9ba848895a0332a3a345e@74.118.140.203:26656 7:05AM INF service start impl=MConn{74.118.140.203:26656} module=p2p msg="Starting MConnection service" peer=66248ea4d70c4b8fd6c9ba848895a0332a3a345e@74.118.140.203:26656 7:05AM INF Error reconnecting to peer. Trying again addr=4a038579e4af256118f3dc7925747c8b8f5321b8@142.93.235.166:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=15 7:05AM ERR Error in validation err="wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=blocksync 7:05AM ERR Stopping peer for error err="reactor validation error: wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=p2p peer="Peer{MConn{74.118.140.203:26656} 66248ea4d70c4b8fd6c9ba848895a0332a3a345e out}" 7:05AM INF service stop impl=MConn{74.118.140.203:26656} module=p2p msg="Stopping MConnection service" peer=66248ea4d70c4b8fd6c9ba848895a0332a3a345e@74.118.140.203:26656 7:05AM INF service stop impl="Peer{MConn{74.118.140.203:26656} 66248ea4d70c4b8fd6c9ba848895a0332a3a345e out}" module=p2p msg="Stopping Peer service" peer=66248ea4d70c4b8fd6c9ba848895a0332a3a345e@74.118.140.203:26656

It synced some blocks and started giving this issue

Gaia Version

v19.1.0

How to reproduce?

No response

mayank-daga commented 1 week ago

HI @mpoke can you help here

mayank-daga commented 2 days ago

HI @mpoke just following up

mayank-daga commented 7 hours ago

Hi just following if anyone can help