search

paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.91k stars 1.14k forks source link

Peers not connecting to each other #11906

Open birozuru opened 2 days ago

birozuru commented 2 days ago

Describe the bug

We are experiencing an issue with our customized Reth binary where nodes fail to connect to each other as pairs in a distributed network setup. Interestingly, when the nodes are set up on a single VM, they connect successfully. All firewall rules and networking rules checkout. When the peers are added manually they still return an empty array of peers on query.

cc: @0xBEEFCAF3

Steps to reproduce

Expected Behavior

Nodes should successfully connect to each other as pairs in the distributed network

Actual Behavior

Nodes fail to establish connections with each other when distributed across different machines

Node logs

2024-10-20T13:00:58.655949Z DEBUG tower::buffer::worker: service.ready=true processing request
2024-10-20T13:00:58.656229Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(65), flags: (0x4: END_HEADERS) }
2024-10-20T13:00:58.656306Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(65) }
2024-10-20T13:00:58.656337Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(65), flags: (0x1: END_STREAM) }
2024-10-20T13:00:58.656932Z DEBUG Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(65), flags: (0x5: END_HEADERS | END_STREAM) }

The above log line shows that the federation peers are acknowledged by the node but still returns an empty peer list.

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 1.0.5-dev Commit SHA: da1ef10e8d2754b255da2a7e2906aa1c8dded2e2

What database version are you on?

null

Which chain / network are you on?

botanix_testnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

mattsse commented 22 hours ago

this isn't much to go on really

When the peers are added manually they still return an empty array of peers on query. is this done via rpc or via cli

can you check whether that ip/enode is reachable? you can use the devp2p rlpx ping command for example

if this is a dialing issue, could you perhaps share some logs from the net target?

birozuru commented 7 hours ago

It's done over rpc

this isn't much to go on really

When the peers are added manually they still return an empty array of peers on query. is this done via rpc or via cli

Now i'm able to get a response from pinging the node after killing and restarting the reth process but still no peer connection

The command devp2p rlpx ping enode_url returned a connection timed out response initially

can you check whether that ip/enode is reachable? you can use the devp2p rlpx ping command for example

Response from ping {Version:5 Name:reth/v1.0.5-da1ef10e/x86_64-unknown-linux-gnu Caps:[eth/68 eth/67 eth/66 frost/0] ListenPort:30303 ID:[189 194 114 178 68 247 23 96 79 255 230 89 210 217 130 5 209 230 118 79 223 69 61 22 49 244 44 45 180 216 215 16 96 96 132 218 129 73 93 85 103 59 252 3 139 223 65 227 244 193 125 9 200 117 160 188 193 234 128 146 25 227 72 38] Rest:[]}

Would share them now

if this is a dialing issue, could you perhaps share some logs from the net target?

birozuru commented 4 hours ago

More verbose logs on this @mattsse

2024-10-22T14:08:21.102532Z TRACE txpool: awaiting new block or reorg state=InSync 2024-10-22T14:08:21.102975Z TRACE discv4: opened UDP socket local_addr=0.0.0.0:30303 2024-10-22T14:08:21.104204Z DEBUG discv4: pinging boot node record=NodeRecord { address: 65.108.70.101, udp_port: 30303, tcp_port: 30303, id: 0x2b252ab6a1d0f971d9722cb839a42cb81db019ba44c08754628ab4a823487071b5695317c8ccd085219c3a03af063495b2f1da8d18218da2d6a82981b45e6ffc } 2024-10-22T14:08:21.104263Z TRACE discv4: sending ping ping=Ping { from: NodeEndpoint { address: 0.0.0.0, udp_port: 30303, tcp_port: 30303 }, to: NodeEndpoint { address: 65.108.70.101, udp_port: 30303, tcp_port: 30303 }, expire: 1729606121, enr_sq: Some(1) } 2024-10-22T14:08:21.104328Z TRACE discv4: sending packet type=Ping to=65.108.70.101:30303 hash=0xda270e929f0615530783673b77273e3aca0e5eded34a4e8f83f8d42743cfb99b 2024-10-22T14:08:21.104346Z DEBUG discv4: pinging boot node record=NodeRecord { address: 18.138.108.67, udp_port: 30303, tcp_port: 30303, id: 0xd860a01f9722d78051619d1e2351aba3f43f943f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666 } 2024-10-22T14:08:21.104363Z TRACE discv4: sending ping ping=Ping { from: NodeEndpoint { address: 0.0.0.0, udp_port: 30303, tcp_port: 30303 }, to: NodeEndpoint { address: 18.138.108.67, udp_port: 30303, tcp_port: 30303 }, expire: 1729606121, enr_sq: Some(1) } 2024-10-22T14:08:21.104411Z TRACE discv4: sending packet type=Ping to=18.138.108.67:30303 hash=0x0842e75893b2302b6ced8de313ffefa2520db5aa9307ebc75362feac28573e5a 2024-10-22T14:08:21.104418Z DEBUG discv4: pinging boot node record=NodeRecord { address: 3.209.45.79, udp_port: 30303, tcp_port: 30303, id: 0x22a8232c3abc76a16ae9d6c3b164f98775fe226f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de } 2024-10-22T14:08:21.104425Z TRACE discv4: sending ping ping=Ping { from: NodeEndpoint { address: 0.0.0.0, udp_port: 30303, tcp_port: 30303 }, to: NodeEndpoint { address: 3.209.45.79, udp_port: 30303, tcp_port: 30303 }, expire: 1729606121, enr_sq: Some(1) } 2024-10-22T14:08:21.104480Z TRACE discv4: sending packet type=Ping to=3.209.45.79:30303 hash=0x575918c667ad0e8d9ca9e244f5dcda3044d10f5bb0a46703ddaf83f06db9a5a6 2024-10-22T14:08:21.104486Z DEBUG discv4: pinging boot node record=NodeRecord { address: 157.90.35.166, udp_port: 30303, tcp_port: 30303, id: 0x4aeb4ab6c14b23e2c4cfdce879c04b0748a20d8e9b59e25ded2a08143e265c6c25936e74cbc8e641e3312ca288673d91f2f93f8e277de3cfa444ecdaaf982052 } 2024-10-22T14:08:21.104494Z TRACE discv4: sending ping ping=Ping { from: NodeEndpoint { address: 0.0.0.0, udp_port: 30303, tcp_port: 30303 }, to: NodeEndpoint { address: 157.90.35.166, udp_port: 30303, tcp_port: 30303 }, expire: 1729606121, enr_sq: Some(1) } 2024-10-22T14:08:21.104549Z TRACE discv4: sending packet type=Ping to=157.90.35.166:30303 hash=0xe4b1dc1c6afde9c9be077a04309258fd4cb12899b1e51dea67303faf050f010c 2024-10-22T14:08:21.104655Z TRACE discv4: sent payload to=65.108.70.101:30303 size=130 2024-10-22T14:08:21.104678Z TRACE discv4: sent payload to=18.138.108.67:30303 size=130 2024-10-22T14:08:21.104696Z TRACE discv4: sent payload to=3.209.45.79:30303 size=130 2024-10-22T14:08:21.104713Z TRACE discv4: sent payload to=157.90.35.166:30303 size=130 2024-10-22T14:08:21.104759Z TRACE discv4: Starting lookup target=0xbdc272b244f717604fffe659d2d98205d1e6764fdf453d1631f42c2db4d8d710606084da81495d55673bfc038bdf41e3f4c17d09c875a0bcc1ea809219e34826 2024-10-22T14:08:21.104863Z DEBUG discv4: pinging boot node record=NodeRecord { address: 65.108.70.101, udp_port: 30303, tcp_port: 30303, id: 0x2b252ab6a1d0f971d9722cb839a42cb81db019ba44c08754628ab4a823487071b5695317c8ccd085219c3a03af063495b2f1da8d18218da2d6a82981b45e6ffc } 2024-10-22T14:08:21.104878Z DEBUG discv4: pinging boot node record=NodeRecord { address: 18.138.108.67, udp_port: 30303, tcp_port: 30303, id: 0xd860a01f9722d78051619d1e2351aba3f43f943f6f00718d1b9baa4101932a1f5011f16bb2b1bb35db20d6fe28fa0bf09636d26a87d31de9ec6203eeedb1f666 } 2024-10-22T14:08:21.104886Z DEBUG discv4: pinging boot node record=NodeRecord { address: 3.209.45.79, udp_port: 30303, tcp_port: 30303, id: 0x22a8232c3abc76a16ae9d6c3b164f98775fe226f0917b0ca871128a74a8e9630b458460865bab457221f1d448dd9791d24c4e5d88786180ac185df813a68d4de } 2024-10-22T14:08:21.104903Z DEBUG discv4: pinging boot node record=NodeRecord { address: 157.90.35.166, udp_port: 30303, tcp_port: 30303, id: 0x4aeb4ab6c14b23e2c4cfdce879c04b0748a20d8e9b59e25ded2a08143e265c6c25936e74cbc8e641e3312ca288673d91f2f93f8e277de3cfa444ecdaaf982052 } 2024-10-22T14:08:21.105205Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", ipinfo.io) 2024-10-22T14:08:21.105258Z DEBUG reqwest::connect: starting new connection: http://ipinfo.io/ 2024-10-22T14:08:21.105291Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("ipinfo.io"), port=None 2024-10-22T14:08:21.105421Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", icanhazip.com) 2024-10-22T14:08:21.105687Z DEBUG reqwest::connect: starting new connection: http://icanhazip.com/ 2024-10-22T14:08:21.105750Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("icanhazip.com"), port=None 2024-10-22T14:08:21.105665Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="ipinfo.io" 2024-10-22T14:08:21.106182Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="icanhazip.com" 2024-10-22T14:08:21.106373Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", ifconfig.me) 2024-10-22T14:08:21.106462Z DEBUG reqwest::connect: starting new connection: http://ifconfig.me/ 2024-10-22T14:08:21.106554Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("ifconfig.me"), port=None 2024-10-22T14:08:21.106953Z TRACE net::peers: schedule outbound connection peer_id=0x6cd057c4aa263a6c1f0c9f11045030d23e075d497ea3f359db2f8a02fcc5d52d2bb81b2dc55e72d53bd47e3f5ac7ec10974983cba1d549e5bc61ae8ab047beb4 addr=PeerAddr { tcp: 34.79.189.111:30303, udp: Some(34.79.189.111:30303) } 2024-10-22T14:08:21.107073Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="ifconfig.me" 2024-10-22T14:08:21.107346Z DEBUG reth::cli: Spawning payload builder service 2024-10-22T14:08:21.107487Z TRACE net::peers: schedule outbound connection peer_id=0x9bef292b80427d355cecb89eda8a50a7d2196a93d73dade5a0c4a07cd334815d50a117189201f0ad9096d36cd690ae34e79a42d9e71c972e55048dabdc8f9651 addr=PeerAddr { tcp: 34.35.52.165:30303, udp: Some(34.35.52.165:30303) } 2024-10-22T14:08:21.107686Z TRACE net: Starting outbound connection. remote_addr=34.79.189.111:30303 peer_id=0x6cd057c4aa263a6c1f0c9f11045030d23e075d497ea3f359db2f8a02fcc5d52d2bb81b2dc55e72d53bd47e3f5ac7ec10974983cba1d549e5bc61ae8ab047beb4 2024-10-22T14:08:21.107721Z TRACE net: Starting outbound connection. remote_addr=34.35.52.165:30303 peer_id=0x9bef292b80427d355cecb89eda8a50a7d2196a93d73dade5a0c4a07cd334815d50a117189201f0ad9096d36cd690ae34e79a42d9e71c972e55048dabdc8f9651 2024-10-22T14:08:21.107533Z TRACE blockchain_tree: Registered subscriber for canonical state 2024-10-22T14:08:21.107954Z DEBUG reth::cli: Spawned payload builder service 2024-10-22T14:08:21.107989Z INFO reth_authority_consensus::builder: Aggregate public key: None 2024-10-22T14:08:21.108225Z TRACE tonic::transport::service::reconnect: poll_ready; idle 2024-10-22T14:08:21.108254Z TRACE tonic::transport::service::reconnect: poll_ready; connecting 2024-10-22T14:08:21.108299Z TRACE hyper::client::connect::http: Http::connect; scheme=Some("http"), host=Some("127.0.0.1"), port=Some(Port(8080)) 2024-10-22T14:08:21.108329Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8080 2024-10-22T14:08:21.108499Z TRACE tonic::transport::service::reconnect: poll_ready; not ready 2024-10-22T14:08:21.108529Z TRACE tonic::transport::service::reconnect: poll_ready; connecting 2024-10-22T14:08:21.108540Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8080 2024-10-22T14:08:21.108562Z TRACE hyper::client::conn: client handshake Http2 2024-10-22T14:08:21.108589Z DEBUG h2::client: binding client connection 2024-10-22T14:08:21.108627Z DEBUG h2::client: client connection bound 2024-10-22T14:08:21.108656Z