Open Stebalien opened 5 years ago
Potentially a silly question, but what's the definition of "routable network" and "non-routable network" in this proposal?
The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.
I've been thinking about this as well in the context of https://github.com/libp2p/libp2p/issues/47.
One possible design is to add an API in go-multiaddr-net
that allows to query Routability()
of multiaddrs, returning an "enum" with the following values:
^ Identify could use this API to filter addresses. And the peerstore could use this to prune existing entries.
Maybe after segregation we can try connecting top to down like if WAN network is same reject it and move to LAN network if that is same dial using loopback on different port.
Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)
Peer B: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)
If A wants to connect with B: Because they are on the same LAN and WAN network they can connect using loopback:port
Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)
Peer B: WAN (1.2.3.4/24) -> LAN (192.168.1.200) -> LOOPBACK (127.0.0.1:4000)
WAN is same, connect on LAN.
Peer A: WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)
Peer B: WAN (2.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)
Totally different network. Dial on WAN.
Can this work?
@upperwal I'm having trouble parsing your first sentence.
@Stebalien So instead of trying all the observed addresses we can segregate the addresses into different categories like PRIVATE, PUBLIC or LOOPBACK (as @raulk suggested) and then we can try connecting in the following order:
~I'm not sure that'll solve the issue. Many nodes simply aren't reachable so we'll end up spamming a bunch of non-routable addresses anyways. Additionally, IMO, we generally want to take the opposite approach when dialing. We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.~
edit: I need to read entire comments before responding.
The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.
Okay. Maybe it wasn't such a silly question, because I assumed a different definition.
I think we should limit the distribution of non-routable addresses even more. I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:
- If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses in same same subnet as the non-routable IP.
Also, could the proposal be amended to say that we send only our non-routable IP to non-routable nodes in the same subnet. I.e. we don't also send the routable IP. This will avoid NAT and IP-level routing problems, and I would definitely prefer it. However, I don't know if there are edge cases, or advanced network setups, that this might be non-ideal for. I'm especially unsure about IPv6.
In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.
I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:
(that was actually my first thought as well)
Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.
In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.
Manual configuration and service discovery through mDNS (local multicast). We have the second but the first probably isn't going to work well for us (although we could provide some kind of configuration language...).
However, I'd would like to amend my original proposal:
We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.
@Stebalien The given approach does prioritise loopback/private addresses and it also gives you a way to check if these addresses work.
Let me explain it again.
Solution to this problem is, to somehow identify the location (network) of peer A and B.
if samePhysicalHost(A, B):
dial(loopback)
else if sameLocalNetwork(A, B):
dial(private_ip)
else
dial(public_ip)
Now we start by comparing the public_ip
(comparing is simple string matching which can happen on the node) of the destination node with the source node, If they match, we can safely assume they are on the same network hence destination should not be dialed using public_ip
(we skip dial using this address so no overhead). If they don't match, they are on different network, safe to dial using this address and stop dialing using other addresses
We do the same with private _ip
but only if public_ip
matches because now the best candidate is private_ip
. If private_ip
for both matches we again skip using it (because they are on the same private network, loopback is the best option here). If they do not match we assume the private network for peer A and B are different hence safe to dail using private_ip
. Do not dial with any other address.
If private_ip
and public_ip
is same go with the loopback because both the peers are on the same machine.
This way we are checking for the subnets in which peer A and B lies and dialing accordingly.
This can also work for complex networks like you described above given we can query the subnet router for it's interface address (and subnets will have different IP signatures so distinguishable). If not, a fallback strategy can be used where:
If peer A and B are on different subnet within an organisation connected to a VPN with observable addresses as follows:
Peer A: 1.2.3.4/24:1234 -> [some subnet X which isn't visible] -> 192.168.1.2 -> 127.0.0.1:3000 Peer B: 1.2.3.4/24:6789 -> [some subnet Y which isn't visible] -> 192.168.1.2 -> 127.0.0.1:4000
Now we will assume that they are on the same host but they are not. We can try connecting using loopback but if it fails we try private IP if that fails we goto the public one. This case is applicable only when we could not find the subnet IP.
Man, I need a diagram. 😛 I hope I am much more clear this time.
Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.
That's a very good point, but there's a few other sides to the coin:
1) Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.
2) Existent but unrouteable private addresses - Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN). In a way, this is an extension of the problem we're trying to solve here i.e. not publishing private IPs to public networks.
(To be clear, I am not married to any particular proposal solution here. I just think its an interesting topic, and I hope I'm contributing to the conversation.)
For what its worth, I think that your proposal is a strict improvement on the current setup, and I would be all-for it being implemented, but I'm not sure I'm convinced that its the end of the road in terms of solutions.
@rob-deutsch
Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.
Yeah, this bugs me too (related https://github.com/ipfs/go-ipfs/issues/1771). And you're right, that's bad form. Really, we should just make this configurable (to some extent). Personally, I'd start with my proposal (as you said, it's a strict improvement but is less restrictive) and we can then add configuration options for more restrictive setups.
Existent but unrouteable private addresses. Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN)
This I'm less worried about. Yes, it can happen but oh well, a few dials fail in rare cases. IMO, the bigger issue is dialing private IP addresses on every dial.
@upperwal
Ah. I saw the in-order part, read the headers, and completely misread your proposal. That's actually a really cool solution.
There are a few drawbacks:
However, it's a really cool heuristic and we can probably just plug it into the dialer with little work.
@upperwal want to try plugging this logic into go-libp2p-swarm/swarm_dial.go
(if you have time)?
@Stebalien Yeah, its a simple solution with little refactor. Sure, let me give it a try.
@Stebalien
I am able to order the ip list as []public, []private, []loopback by implementing Routability()
unordered
[/ip4/192.168.2.229/tcp/28447 /ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/127.0.0.1/tcp/4001 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/14.114.26.227/tcp/10782]
ordered
[/ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/14.114.26.227/tcp/10782 /ip4/192.168.2.229/tcp/28447 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001]
Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address. InterfaceListenAddresses()
can only give [/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]
. Hence segregating subnet would be difficult. Any suggestion?
A quick and (somehow) dirty solution would be to pass *BasicHost
when creating swarm instance(https://github.com/libp2p/go-libp2p/blob/master/config/config.go#L94), and get observed addresses by *BasicHost.IDService().OwnObservedAddrs()
A more graceful solution might need some refactoring, store observed addresses together with peerstore. Just my personal opinion.
Unfortunately that can't work -- it would create a circular dependency.
So some refactoring is unavoidable...
I think identity protocol can help (with little refactoring) where destination peer can send us back our observed address during protocol negotiation. Observed address can be stored and made available by some API.
Edit: Keeping our observed address can help us take intelligent and optimised decisions when it comes to routing and connectivity.
Damn. That's annoying. I don't know of a simple fix.
Next step would be to find the correct subnet.
We don't really need to know anything about subnets to get this to work, we just need our observed external address.
Damn. That's annoying.
I know.
By "Next step would be to find the correct subnet.", I meant to find the shortest path to dial on and yes that would only require our observed address.
@Stebalien Is #427 related to the other peer's observed addresses (the one I would be connected to)?
Yes but that won't help. The issue here is that the swarm doesn't have access to information like this.
One solution is to tell the peerstore about observed addresses. That's likely the best solution. Thoughts @raulk/@bigs?
@Stebalien I feel we need a component in the system that serves the role of a "topology manager", so that it can take decisions based on a number of factors like:
Personally I'd like to keep the peerstore on the thin side, ideally circumscribed to storing data about peers without taking decisions.
In the context of this issue, the topology manager would segment peers based on their routability, and it would decide which maddrs to keep for each peer, based on the host's interest and role.
Yeah, you're probably right. Basically, we'd have:
(at least that's what I'd do)
It would also be nice to push access to the router down into the swarm while we're at it.
What happens in the case of say client isolated networks? That is networks where clients cannot connect directly to each other but technically have the same networks. Very common example is a /24 broken up into /28 or /29's and most home networks being 192.168.0.0/24. Rather then trying to decern or evern assume local routing policies, can IPFS be told to listen when told a host is unreachable? Right now there's nothing to stop a host from advertising networks it's neither part of nor has any relation to it.
Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address.
InterfaceListenAddresses()
can only give[/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]
. Hence segregating subnet would be difficult. Any suggestion?
Does the dialling peer need to know its observed address? I think that it doesn't.
Maybe the dialling peer can just look at net.Interfaces()
?
This address-choice-mechanism was predicated on "after segregation". After the segregation it will be reasonable to assume that if a remote IP is in the peerstore then our node must be listening on an local IP that we can use to reach the remote IP.
Different colour arrows show shortest path to connect.
Does the dialling peer need to know its observed address? I think that it doesn't.
Yes, it is needed to identify if peer A and E are on the same network or not. Look at the diagram above. You cannot distinguish between peer A and E because they have the same private (10.X.X.X) and local (192.168.X.X) addresses but still they are on entirely different networks.
net.Interfaces()
can only give you local interfaces and not the one on the public side. That can only be obtained from a router gateway which has a public IP.
After the segregation it will be reasonable to assume that if a remote IP is in the peerstore then our node must be listening on an local IP that we can use to reach the remote IP.
Can you please elaborate on this.
@raulk @Stebalien totally agree. Can we use identity protocol to get our observed addresses from other peers?
So whenever peers connect and agree on protocols they exchange routable addresses of the other peer. This way we get all observed addresses which can be used later.
Can you please elaborate on this.
In your first post in this topic you said "maybe after segregation we can try connecting top to down like if WAN network is same" so I was commenting as the Routability() change will be made AFTER the segregation of announcements.
I now realise that the intention is to integrate this Routability() logic before the segregation of announcements.
Open question: Would it be worthwhile doing the segregation of announcements prior to adding this Routability() logic? Especially since the next steps of the Routability() integration require non-trivial development effort.
I feel we have several discussions inflight, all of which are addressing facets of the same question:
Which addresses of mine should I announce to the world?
For the sake of my own clarity, I'm collecting the discussion themes here. Hopefully this will be useful to others!
@Stebalien – do I have a good handle on the problem domain? Anything missing from this list?
EDIT: added point 6 as per below.
Is it possibly an even more generalised question about node addresses and path discovery?
Also, could I suggest a modification to your first point?
- Contextualised announcements => depending on who I'm speaking to, I should announce the proper addresses of
minemyself and peers I know about.
And an additional point that @upperwal has been working on could be...
@rob-deutsch
Also, could I suggest a modification to your first point?
... I should announce the proper addresses of
minemyself and peers I know about.
Dunno, I think this is too much magic. Peer A should not make a decision about how Peer B addresses Peer C. Also, peers certifying their own routing records would prohibit third parties from modifying them, as the signature would no longer match.
- Remote address choice => how do a choose which of a remote node's addresses I should communicate with?
Reworded it and included it. Good catch.
Instead of trying to do it in the peerstore and the advertisement logic, we might be able to do it with some more dialer smarts that are aware of our own network addresses.
The dialer will need to filter in the aggregate and do the segregation. If it sees a private network address, it can skip dialing it unless the ip address of its own non-private network address matches that in the address set.
So let's say we are we have addrs 127.0.0.1,10.0.1.10,123.123.123.123 and we want to dial 127.0.0.1,10.1.2.3,134.134.134. Here we can notice that the private addresses are behind a different public address, so we can deduce that they are in a different private network and remove them from the dial set.
@vyzo that's a fair place to start, indeed. That will be a quick win before making deeper (and more dangerous) changes.
It's been bugging me that the dialing logic in go-libp2p-swarm is hardcoded and not abstractable, so I'm turning it into a "dialing strategy" we can have multiple implementations of, to select dynamically.
catching up on this lengthy and insightful issue!
@Stebalien I feel we need a component in the system that serves the role of a "topology manager", so that it can take decisions based on a number of factors like:
- our observed address(es),
- the node's primary role (e.g. specialised types: relay, DHT, rendezvous, etc. or just a general node),
- our target network (e.g. public, private, etc.)
Personally I'd like to keep the peerstore on the thin side, ideally circumscribed to storing data about peers without taking decisions.
In the context of this issue, the topology manager would segment peers based on their routability, and it would decide which maddrs to keep for each peer, based on the host's interest and role.
i like this approach quite a bit. i think we could follow the BasicHost
-> RoutedHost
convention and wrap the Peerstore
in something like a SegmentedPeerstore
edit: or a PeerRouter
@vyzo that's a fair place to start, indeed. That will be a quick win before making deeper (and more dangerous) changes.
It's been bugging me that the dialing logic in go-libp2p-swarm is hardcoded and not abstractable, so I'm turning it into a "dialing strategy" we can have multiple implementations of, to select dynamically.
i think this is reasonable, just be careful when making choices about composability. i think this is long overdue, but i'd be sure to account for the multiple wrapping layers here and try your best to tease them apart! something like:
type Dialer interface {
func Dial(...) ...
}
type DialerStrategy interface {
Dialer
func Wrap(Dialer) Dialer
}
then we could have
var MultiaddrDialer Dialer // manet stuff
var SyncDialer DialerWrapper // lets us parallelize and cancel upon success
var ResourceLimitedDialer DialerWrapper // lets us constrict file handle usage
one thing i'm immediately wondering about/concerned about is how "peer-to-peer friendly" an identity service is. feels like it has the potential to centralize things quite a bit if gone about wrong. perhaps a node doesn't announce until it's fully bootstrapped, which might include identifying itself by multiple peers in different subnets?
furthermore, i think raul's concerns about backwards compatibility point towards a dialer-focused solution as the best steps forward.
Also, could I suggest a modification to your first point?
... I should announce the proper addresses of mine myself and peers I know about.
Dunno, I think this is too much magic. Peer A should not make a decision about how Peer B addresses Peer C. Also, peers certifying their own routing records would prohibit third parties from modifying them, as the signature would no longer match.
Regardless of how much magic this is, isn't it required?
For example, lets say that Peer A and Peer B both have their own public addresses but are also on the same private network (because they're in a datacenter or VPN).
Peer A and Peer B might tell eachother both their public and private IP addresses. But what happens when Peer A wants to tell Peer C (on the other side of the internet) about Peer B? Will Peer A tell Peer C the private addresses too?
@Stebalien
Damn. That's annoying. I don't know of a simple fix.
got a trivial solution. Calling dht.FindPeer(self)
would return our observable Multiaddr
from one of our immediate neighbour.
got a trivial solution. Calling dht.FindPeer(self) would return our observable Multiaddr from one of our immediate neighbour.
I'd rather not make any network requests to discover this... @raulk is now working on a dialing refactor to make things like this possible.
cool
@raulk Any update on this?
I noticed that identify already knows about observedAddrs
which can be used for remote address selection in swarm before dialling. We can identify the topology based on our addresses (observed + local + private) and remote addresses. This way we dial to only one remote address.
This will require ObservedAddrSet
to be available within swarm. As swarm is created before basicHost.
Maybe we can have something like swrm.SetObservableAddrs()
This is a quick fix. Let me know if you are working on something bigger and I would be happy to contribute.
Edit: This will also help close libp2p/go-libp2p-pubsub#40 and resolve error ERROR pubsub: already have connection to peer:
PubSub error (except in the case where A and B simultaneously connects to each other)
@upperwal Here's a WIP PR for a change that modularises the swarm dialer in a way that allows applications to determine how they want to prioritise multiaddrs, plan the dials over time, and throttle each attempt based on resources: https://github.com/libp2p/go-libp2p-swarm/pull/88.
Maybe we can have something like
swrm.SetObservableAddrs()
We'll definitely need to expose the observed addresses on the Network
interface somehow. Haven't thought about the API, but Planners
in dialerv2 will want this info.
Any updates to this?
Progress:
Currently, we announce all IP addresses we're listening, even private ones, and will dial any address we find for a peer, even private ones. We do this intentionally so we can connect to nodes on our local network. Unfortunately, this causes issues like https://github.com/ipfs/go-ipfs/issues/5511 and really pisses off users (for obvious reasons).
One reasonable solution (IMO) is to segregate routable and non-routable networks. That is:
We'll still be able to discover and dial nodes on non-routable networks through local discovery (mDNS). Additionally, if we're running IPFS on a VPN and all peers have non-routable addresses, everything should "just work".
This will mostly require changes to the DHT and the Identify protocol.
Future work: