Colo: Internal routing issues when running testnets on public IPs

[t-nelson]

Problem

When passing the -P flag to net/colo.sh create, specifying use of public IPs, the following error occurs upon validator start up:

[2019-10-11T17:24:24.287149381Z WARN  solana_netutil] ip_echo_server request failed: Failed to deserialize: Io(Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" })
[2019-10-11T17:24:29.287378265Z ERROR solana_netutil] Received no response at tcp/8899, check your port configuration: timed out waiting on channel

Some experimenting with the solana-ip-address{,-server} progs from the netutil directory uncover the issue.

1) Connecting to a solana-ip-address-server instance, using its public IP, from a host external to the Colo LAN, reports the expected external host's public IP.

trent@trent-build:~/solana$ target/debug/solana-ip-address 216.24.140.152:8899
35.203.184.187

2) Connecting to the same server instance, this time using its private IP address from another host on the Colo LAN, again reports as expected, the client host's private IP.

solana@foosball:~$ ./solana-ip-address 10.1.1.23:8899
10.1.1.21

3) Now connecting to the server from the same LAN host as in [2], but this time using the server's public IP, results in the connecting coming from the LAN gateway private IP.

solana@foosball:~$ ./solana-ip-address 216.24.140.152:8899
10.1.1.1

Proposed Solution

I believe this is a network equipment configuration issue. I've opened a line of communication with the contractors who helped us with setup and am waiting for feedback.

[pgarg66]

@t-nelson , do we have a network topology for Colo captured in a diagram somewhere? Basically a wire diagram that has all network equipment (machines, switches, gateways) and the interconnectivity..

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale[bot]]

This stale issue has been automatically closed. Thank you for your contributions.