prefer private over public IPs for peer connectivty

[rade]

In weave deployments that span multiple data centres, peers inside the same data centre are often reachable via two IPs - a public and a private one. Using the latter is more efficient, so ideally we want weave to pick that, when there is a choice.

[bboreham]

Maybe create a way to specify tuples of addresses for the same endpoint, so only if the first address fails we try the second, and so on?

[rade]

That would work for addresses specified on the command line, but not addresses learned via gossip.

[rade]

Also, if you are going to specify the addresses on the command line anyway, then surely you'd just supply the private IPs for all 'local' hosts, and public IPs for the others.

[fermayo]

So what would happen if a peer can access another one with two different IPs? Which one will weave use today? The first it learns about, or the first it successfully connects to? What about choosing a private IP over a public IP when it finds out that the same peer can be reached by both? So even if it learns about the public IP of a peer via gossip, it still prefers using the private IP.

[rade]

So what would happen if a peer can access another one with two different IPs? Which one will weave use today? The first it learns about, or the first it successfully connects to?

The latter.

What about choosing a private IP over a public IP when it finds out that the same peer can be reached by both? So even if it learns about the public IP of a peer via gossip, it still prefers using the private IP.

That won't work. Weave peers don't know what peer is at a particular IP address. The only way they can find that out is by connecting to the address.

Also, what should happen when a peer learns about a public IP, connects to it, and later learns about a private IP and finds the same peer there? Should it drop the first connection?

[fermayo]

I meant, if after connecting to the IP it finds an already connected peer. If it was connected via public IP, and when connecting to a private IP it finds the same peer, I'd try to switch the connection to that peer via the private IP (is that possible without downtime?).

[rade]

is that possible without downtime?

Like all 'glitches' in peer connectivity, this can drop some packets. Whether that is observed as 'downtime' depends on the app; for example, TCP connections should remain intact.

[fermayo]

I'd say that's a reasonable compromise

[rade]

@fermayo how much do rely on weave's peer discovery mechanism? i.e. that once one peer has a connection to another, other peers find out about it and in turn establish connections.

I am asking because I wonder whether there should be a way to disable that feature, thus giving users complete control over the topology.

[fermayo]

We don't rely on that. If a new node is deployed, we weave connect all the others to it, and weave connect it to the others.

[rade]

@fermayo So if there was a way to disable discovery, then this issue here wouldn't arise for you. Correct?

[fermayo]

Correct

[rade]

I wonder whether there should be a way to disable that feature, thus giving users complete control over the topology.

847

[bboreham]

We could prefer trusted subnets over untrusted.

[bboreham]

Note the flag suggested at https://github.com/weaveworks/weave/issues/646#issuecomment-108844179 was implemented as --no-discovery