Segregate routable and non-routable networks

Stebalien commented 5 years ago

Currently, we announce all IP addresses we're listening, even private ones, and will dial any address we find for a peer, even private ones. We do this intentionally so we can connect to nodes on our local network. Unfortunately, this causes issues like https://github.com/ipfs/go-ipfs/issues/5511 and really pisses off users (for obvious reasons).

One reasonable solution (IMO) is to segregate routable and non-routable networks. That is:

If we learn about a non-routable IP address from a routable network, discard it.
If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses.

We'll still be able to discover and dial nodes on non-routable networks through local discovery (mDNS). Additionally, if we're running IPFS on a VPN and all peers have non-routable addresses, everything should "just work".

This will mostly require changes to the DHT and the Identify protocol.

Future work:

https://github.com/libp2p/notes/issues/3
Separate DHTs? We could have a private one and a public one.

rob-deutsch commented 5 years ago

Potentially a silly question, but what's the definition of "routable network" and "non-routable network" in this proposal?

vyzo commented 5 years ago

The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.

raulk commented 5 years ago

I've been thinking about this as well in the context of https://github.com/libp2p/libp2p/issues/47.

One possible design is to add an API in go-multiaddr-net that allows to query Routability() of multiaddrs, returning an "enum" with the following values:

PUBLIC.
PRIVATE => as per Private Network.
PROXIMITY => future: Bluetooth or proximity-based technologies.
LOOPBACK.
NA => not applicable.

raulk commented 5 years ago

^ Identify could use this API to filter addresses. And the peerstore could use this to prune existing entries.

upperwal commented 5 years ago

Maybe after segregation we can try connecting top to down like if WAN network is same reject it and move to LAN network if that is same dial using loopback on different port.

Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)

If A wants to connect with B: Because they are on the same LAN and WAN network they can connect using loopback:port

Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B: WAN (1.2.3.4/24) -> LAN (192.168.1.200) -> LOOPBACK (127.0.0.1:4000)

WAN is same, connect on LAN.

Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B: WAN (2.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)

Totally different network. Dial on WAN.

Can this work?

Stebalien commented 5 years ago

@upperwal I'm having trouble parsing your first sentence.

upperwal commented 5 years ago

@Stebalien So instead of trying all the observed addresses we can segregate the addresses into different categories like PRIVATE, PUBLIC or LOOPBACK (as @raulk suggested) and then we can try connecting in the following order:

PUBLIC: Connect using PUBLIC endpoint only if peer A and B are on different PUBLIC IP's. If they are on the same PUBLIC IP's a shorter path exist. Case 3 above shows this.
PRIVATE: If they are on the same PUBLIC IP check if they are on the same PRIVATE IP. If they are not, connect using PRIVATE endpoint. Case 2.
LOOPBACK: If their PUBLIC and PRIVATE IP's or endpoints are same they are on the same machine. Use loopback:port to dial. Case 1.

Stebalien commented 5 years ago

~I'm not sure that'll solve the issue. Many nodes simply aren't reachable so we'll end up spamming a bunch of non-routable addresses anyways. Additionally, IMO, we generally want to take the opposite approach when dialing. We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.~

edit: I need to read entire comments before responding.

rob-deutsch commented 5 years ago

The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.

Okay. Maybe it wasn't such a silly question, because I assumed a different definition.

I think we should limit the distribution of non-routable addresses even more. I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:

If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses in same same subnet as the non-routable IP.

Also, could the proposal be amended to say that we send only our non-routable IP to non-routable nodes in the same subnet. I.e. we don't also send the routable IP. This will avoid NAT and IP-level routing problems, and I would definitely prefer it. However, I don't know if there are edge cases, or advanced network setups, that this might be non-ideal for. I'm especially unsure about IPv6.

In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.

Stebalien commented 5 years ago

I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:

(that was actually my first thought as well)

Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.

In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.

Manual configuration and service discovery through mDNS (local multicast). We have the second but the first probably isn't going to work well for us (although we could provide some kind of configuration language...).

However, I'd would like to amend my original proposal:

Localhost addresses should only be advertised to other localhost nodes.
IPv6 link-local addresses (when we eventually add support for them) should only be advertised to nodes on the same link. This isn't an issue currently (we don't support these) but we should keep it in mind.

upperwal commented 5 years ago

We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.

@Stebalien The given approach does prioritise loopback/private addresses and it also gives you a way to check if these addresses work.

Let me explain it again.

Solution to this problem is, to somehow identify the location (network) of peer A and B.

if samePhysicalHost(A, B):
  dial(loopback)
else if sameLocalNetwork(A, B):
  dial(private_ip)
else
  dial(public_ip)

Now we start by comparing the public_ip (comparing is simple string matching which can happen on the node) of the destination node with the source node, If they match, we can safely assume they are on the same network hence destination should not be dialed using public_ip (we skip dial using this address so no overhead). If they don't match, they are on different network, safe to dial using this address and stop dialing using other addresses
We do the same with private _ip but only if public_ip matches because now the best candidate is private_ip. If private_ip for both matches we again skip using it (because they are on the same private network, loopback is the best option here). If they do not match we assume the private network for peer A and B are different hence safe to dail using private_ip. Do not dial with any other address.
If private_ip and public_ip is same go with the loopback because both the peers are on the same machine.

This way we are checking for the subnets in which peer A and B lies and dialing accordingly.

This can also work for complex networks like you described above given we can query the subnet router for it's interface address (and subnets will have different IP signatures so distinguishable). If not, a fallback strategy can be used where:

If peer A and B are on different subnet within an organisation connected to a VPN with observable addresses as follows:

Peer A: 1.2.3.4/24:1234 -> [some subnet X which isn't visible] -> 192.168.1.2 -> 127.0.0.1:3000 Peer B: 1.2.3.4/24:6789 -> [some subnet Y which isn't visible] -> 192.168.1.2 -> 127.0.0.1:4000

Now we will assume that they are on the same host but they are not. We can try connecting using loopback but if it fails we try private IP if that fails we goto the public one. This case is applicable only when we could not find the subnet IP.

Man, I need a diagram. 😛 I hope I am much more clear this time.

rob-deutsch commented 5 years ago

Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.

That's a very good point, but there's a few other sides to the coin:

1) Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.

2) Existent but unrouteable private addresses - Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN). In a way, this is an extension of the problem we're trying to solve here i.e. not publishing private IPs to public networks.

(To be clear, I am not married to any particular proposal solution here. I just think its an interesting topic, and I hope I'm contributing to the conversation.)

For what its worth, I think that your proposal is a strict improvement on the current setup, and I would be all-for it being implemented, but I'm not sure I'm convinced that its the end of the road in terms of solutions.

Stebalien commented 5 years ago

@rob-deutsch

Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.

Yeah, this bugs me too (related https://github.com/ipfs/go-ipfs/issues/1771). And you're right, that's bad form. Really, we should just make this configurable (to some extent). Personally, I'd start with my proposal (as you said, it's a strict improvement but is less restrictive) and we can then add configuration options for more restrictive setups.

Existent but unrouteable private addresses. Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN)

This I'm less worried about. Yes, it can happen but oh well, a few dials fail in rare cases. IMO, the bigger issue is dialing private IP addresses on every dial.

@upperwal

Ah. I saw the in-order part, read the headers, and completely misread your proposal. That's actually a really cool solution.

There are a few drawbacks:

We could both have different public IP addresses but be on the same VPN.
One or both peers may not know their public IP addresses. They may even have multiple public IP addresses on some of the nastier NATs. In this case, we'd have to just dial every address.
It still leaks private information (@rob-deutsch's issue).

However, it's a really cool heuristic and we can probably just plug it into the dialer with little work.

Stebalien commented 5 years ago

@upperwal want to try plugging this logic into go-libp2p-swarm/swarm_dial.go (if you have time)?

upperwal commented 5 years ago

@Stebalien Yeah, its a simple solution with little refactor. Sure, let me give it a try.

upperwal commented 5 years ago

@Stebalien

I am able to order the ip list as []public, []private, []loopback by implementing Routability()

unordered
[/ip4/192.168.2.229/tcp/28447 /ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/127.0.0.1/tcp/4001 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/14.114.26.227/tcp/10782]

ordered
[/ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/14.114.26.227/tcp/10782 /ip4/192.168.2.229/tcp/28447 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001]

Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address. InterfaceListenAddresses() can only give [/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]. Hence segregating subnet would be difficult. Any suggestion?

cannium commented 5 years ago

A quick and (somehow) dirty solution would be to pass *BasicHost when creating swarm instance(https://github.com/libp2p/go-libp2p/blob/master/config/config.go#L94), and get observed addresses by *BasicHost.IDService().OwnObservedAddrs() A more graceful solution might need some refactoring, store observed addresses together with peerstore. Just my personal opinion.

vyzo commented 5 years ago

Unfortunately that can't work -- it would create a circular dependency.

cannium commented 5 years ago

So some refactoring is unavoidable...

upperwal commented 5 years ago

I think identity protocol can help (with little refactoring) where destination peer can send us back our observed address during protocol negotiation. Observed address can be stored and made available by some API.

Edit: Keeping our observed address can help us take intelligent and optimised decisions when it comes to routing and connectivity.

Stebalien commented 5 years ago

Damn. That's annoying. I don't know of a simple fix.

Next step would be to find the correct subnet.

We don't really need to know anything about subnets to get this to work, we just need our observed external address.

upperwal commented 5 years ago

Damn. That's annoying.

I know.

By "Next step would be to find the correct subnet.", I meant to find the shortest path to dial on and yes that would only require our observed address.

@Stebalien Is #427 related to the other peer's observed addresses (the one I would be connected to)?

Stebalien commented 5 years ago

Yes but that won't help. The issue here is that the swarm doesn't have access to information like this.

One solution is to tell the peerstore about observed addresses. That's likely the best solution. Thoughts @raulk/@bigs?

raulk commented 5 years ago

@Stebalien I feel we need a component in the system that serves the role of a "topology manager", so that it can take decisions based on a number of factors like:

our observed address(es),
the node's primary role (e.g. specialised types: relay, DHT, rendezvous, etc. or just a general node),
our target network (e.g. public, private, etc.)

Personally I'd like to keep the peerstore on the thin side, ideally circumscribed to storing data about peers without taking decisions.

In the context of this issue, the topology manager would segment peers based on their routability, and it would decide which maddrs to keep for each peer, based on the host's interest and role.

Stebalien commented 5 years ago

Yeah, you're probably right. Basically, we'd have:

Peerstore: Storage for peer addresses.
Peer routing: Source for new peer addresses.
Dialing decision manger? This is the special sauce that needs to know which addresses should and shouldn't be dialed (pulling them from the peer routing and peerstore subsystems).

(at least that's what I'd do)

It would also be nice to push access to the router down into the swarm while we're at it.

h1z1 commented 5 years ago

What happens in the case of say client isolated networks? That is networks where clients cannot connect directly to each other but technically have the same networks. Very common example is a /24 broken up into /28 or /29's and most home networks being 192.168.0.0/24. Rather then trying to decern or evern assume local routing policies, can IPFS be told to listen when told a host is unreachable? Right now there's nothing to stop a host from advertising networks it's neither part of nor has any relation to it.

rob-deutsch commented 5 years ago

Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address. InterfaceListenAddresses() can only give [/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]. Hence segregating subnet would be difficult. Any suggestion?

Does the dialling peer need to know its observed address? I think that it doesn't.

Maybe the dialling peer can just look at net.Interfaces()?

This address-choice-mechanism was predicated on "after segregation". After the segregation it will be reasonable to assume that if a remote IP is in the peerstore then our node must be listening on an local IP that we can use to reach the remote IP.

upperwal commented 5 years ago

Different colour arrows show shortest path to connect.

untitled-1-01

Does the dialling peer need to know its observed address? I think that it doesn't.

Yes, it is needed to identify if peer A and E are on the same network or not. Look at the diagram above. You cannot distinguish between peer A and E because they have the same private (10.X.X.X) and local (192.168.X.X) addresses but still they are on entirely different networks.

net.Interfaces() can only give you local interfaces and not the one on the public side. That can only be obtained from a router gateway which has a public IP.

After the segregation it will be reasonable to assume that if a remote IP is in the peerstore then our node must be listening on an local IP that we can use to reach the remote IP.

Can you please elaborate on this.

upperwal commented 5 years ago

@raulk @Stebalien totally agree. Can we use identity protocol to get our observed addresses from other peers?

So whenever peers connect and agree on protocols they exchange routable addresses of the other peer. This way we get all observed addresses which can be used later.

rob-deutsch commented 5 years ago

Can you please elaborate on this.

In your first post in this topic you said "maybe after segregation we can try connecting top to down like if WAN network is same" so I was commenting as the Routability() change will be made AFTER the segregation of announcements.

I now realise that the intention is to integrate this Routability() logic before the segregation of announcements.

Open question: Would it be worthwhile doing the segregation of announcements prior to adding this Routability() logic? Especially since the next steps of the Routability() integration require non-trivial development effort.

raulk commented 5 years ago

I feel we have several discussions inflight, all of which are addressing facets of the same question:

Which addresses of mine should I announce to the world?

this discussion.
AutoNAT pull request: https://github.com/libp2p/go-libp2p-autonat/pull/1
Peer routing records to deal with "address abusers": https://github.com/libp2p/libp2p/issues/47

For the sake of my own clarity, I'm collecting the discussion themes here. Hopefully this will be useful to others!

Contextualised announcements => depending on who I'm speaking to [EDIT: and the venue, i.e. DHT, Identify, etc.], I should announce the proper addresses of mine.
Learning my own addresses => feeding them as an input to (1). Two methods so far:
- Identify protocol => observed address.
- Candidate AutoNAT implementation => new ambient protocol to ask peers to try dialling us back on our suspected public addresses.
Rejecting non-routable addresses => not all peers will upgrade at once, so more intelligent peers should filter the addresses returned by old peers.
Cleaning up peerstores => use the filtering logic to clean up existing peerstores (likely not a problem now, as everybody is using the in-memory peerstore, so a restart to upgrade the application, e.g. IPFS, would anyway start from a blank peerstore).
Certified addresses => self-signing my address list so it can be relayed without tampering via DHT or pubsub.
- This record should be stored separately in the peerstore from the "effective addresses" we're using for that peer.
Remote address selection => given a list of remote maddrs, how to choose the optimal one based on information inferred from the network topology?

@Stebalien – do I have a good handle on the problem domain? Anything missing from this list?

EDIT: added point 6 as per below.

rob-deutsch commented 5 years ago

Is it possibly an even more generalised question about node addresses and path discovery?

Also, could I suggest a modification to your first point?

Contextualised announcements => depending on who I'm speaking to, I should announce the proper addresses of ~~mine~~ myself and peers I know about.

And an additional point that @upperwal has been working on could be...

Remote address choice => how do a choose which of a remote node's addresses I should communicate with?

raulk commented 5 years ago

@rob-deutsch

Also, could I suggest a modification to your first point?

... I should announce the proper addresses of ~~mine~~ myself and peers I know about.

Dunno, I think this is too much magic. Peer A should not make a decision about how Peer B addresses Peer C. Also, peers certifying their own routing records would prohibit third parties from modifying them, as the signature would no longer match.

Remote address choice => how do a choose which of a remote node's addresses I should communicate with?

Reworded it and included it. Good catch.

vyzo commented 5 years ago

Instead of trying to do it in the peerstore and the advertisement logic, we might be able to do it with some more dialer smarts that are aware of our own network addresses.

The dialer will need to filter in the aggregate and do the segregation. If it sees a private network address, it can skip dialing it unless the ip address of its own non-private network address matches that in the address set.

So let's say we are we have addrs 127.0.0.1,10.0.1.10,123.123.123.123 and we want to dial 127.0.0.1,10.1.2.3,134.134.134. Here we can notice that the private addresses are behind a different public address, so we can deduce that they are in a different private network and remove them from the dial set.

raulk commented 5 years ago

@vyzo that's a fair place to start, indeed. That will be a quick win before making deeper (and more dangerous) changes.

It's been bugging me that the dialing logic in go-libp2p-swarm is hardcoded and not abstractable, so I'm turning it into a "dialing strategy" we can have multiple implementations of, to select dynamically.

bigs commented 5 years ago

catching up on this lengthy and insightful issue!

@Stebalien I feel we need a component in the system that serves the role of a "topology manager", so that it can take decisions based on a number of factors like:

our observed address(es),

the node's primary role (e.g. specialised types: relay, DHT, rendezvous, etc. or just a general node),

our target network (e.g. public, private, etc.)

Personally I'd like to keep the peerstore on the thin side, ideally circumscribed to storing data about peers without taking decisions.

In the context of this issue, the topology manager would segment peers based on their routability, and it would decide which maddrs to keep for each peer, based on the host's interest and role.

i like this approach quite a bit. i think we could follow the BasicHost -> RoutedHost convention and wrap the Peerstore in something like a SegmentedPeerstore

edit: or a PeerRouter

bigs commented 5 years ago

@vyzo that's a fair place to start, indeed. That will be a quick win before making deeper (and more dangerous) changes.

It's been bugging me that the dialing logic in go-libp2p-swarm is hardcoded and not abstractable, so I'm turning it into a "dialing strategy" we can have multiple implementations of, to select dynamically.

i think this is reasonable, just be careful when making choices about composability. i think this is long overdue, but i'd be sure to account for the multiple wrapping layers here and try your best to tease them apart! something like:

type Dialer interface {
  func Dial(...) ...
}

type DialerStrategy interface {
  Dialer

  func Wrap(Dialer) Dialer
}

then we could have

var MultiaddrDialer Dialer // manet stuff
var SyncDialer DialerWrapper // lets us parallelize and cancel upon success
var ResourceLimitedDialer DialerWrapper // lets us constrict file handle usage

bigs commented 5 years ago

one thing i'm immediately wondering about/concerned about is how "peer-to-peer friendly" an identity service is. feels like it has the potential to centralize things quite a bit if gone about wrong. perhaps a node doesn't announce until it's fully bootstrapped, which might include identifying itself by multiple peers in different subnets?

furthermore, i think raul's concerns about backwards compatibility point towards a dialer-focused solution as the best steps forward.

rob-deutsch commented 5 years ago

Also, could I suggest a modification to your first point?

... I should announce the proper addresses of mine myself and peers I know about.

Dunno, I think this is too much magic. Peer A should not make a decision about how Peer B addresses Peer C. Also, peers certifying their own routing records would prohibit third parties from modifying them, as the signature would no longer match.

Regardless of how much magic this is, isn't it required?

For example, lets say that Peer A and Peer B both have their own public addresses but are also on the same private network (because they're in a datacenter or VPN).

Peer A and Peer B might tell eachother both their public and private IP addresses. But what happens when Peer A wants to tell Peer C (on the other side of the internet) about Peer B? Will Peer A tell Peer C the private addresses too?

upperwal commented 5 years ago

@Stebalien

Damn. That's annoying. I don't know of a simple fix.

got a trivial solution. Calling dht.FindPeer(self) would return our observable Multiaddr from one of our immediate neighbour.

Stebalien commented 5 years ago

got a trivial solution. Calling dht.FindPeer(self) would return our observable Multiaddr from one of our immediate neighbour.

I'd rather not make any network requests to discover this... @raulk is now working on a dialing refactor to make things like this possible.

upperwal commented 5 years ago

cool

upperwal commented 5 years ago

@raulk Any update on this?

I noticed that identify already knows about observedAddrs which can be used for remote address selection in swarm before dialling. We can identify the topology based on our addresses (observed + local + private) and remote addresses. This way we dial to only one remote address.

This will require ObservedAddrSet to be available within swarm. As swarm is created before basicHost.

https://github.com/libp2p/go-libp2p/blob/3e2dc09212ad570c9173376ca9e3c54ae8a7d394/config/config.go#L105-L116

Maybe we can have something like swrm.SetObservableAddrs()

This is a quick fix. Let me know if you are working on something bigger and I would be happy to contribute.

Edit: This will also help close libp2p/go-libp2p-pubsub#40 and resolve error ERROR pubsub: already have connection to peer: PubSub error (except in the case where A and B simultaneously connects to each other)

raulk commented 5 years ago

@upperwal Here's a WIP PR for a change that modularises the swarm dialer in a way that allows applications to determine how they want to prioritise multiaddrs, plan the dials over time, and throttle each attempt based on resources: https://github.com/libp2p/go-libp2p-swarm/pull/88.

Maybe we can have something like swrm.SetObservableAddrs()

We'll definitely need to expose the observed addresses on the Network interface somehow. Haven't thought about the API, but Planners in dialerv2 will want this info.

bonedaddy commented 4 years ago

Any updates to this?

Stebalien commented 4 years ago

Progress:

Two DHTs: https://github.com/libp2p/go-libp2p/issues/803.
Scoped addresses: https://github.com/libp2p/go-libp2p/issues/793

libp2p / go-libp2p

Segregate routable and non-routable networks #436