Open voidpointertonull opened 1 year ago
calmh
commented
1 year ago I see no real downsides here. Mostly it seems to depend on host configuration whether multiple paths are eventually used or not.
Notably, this does not, today, provide multiple connections between single-homed hosts (as is also requested somewhat frequently).
voidpointertonull
commented
1 year ago Correct, the advantage of this compared to the planned QUIC multipath solution is that just like any other networking configuration, setting up multipath is also up to the sysadmin on the host level.
Today it may not provide multiple connections in that described situation yet, but IPv6 and IPv4 mixing is supported, so given the right setup, a lower quality IPv4 subflow going over possibly multiple NATs as it's not uncommon nowadays can be supplemented and essentially gracefully replaced with a likely faster IPv6 support which may enjoy faster routing. Support for that is surely somewhat recent: https://github.com/multipath-tcp/mptcp_net-next/issues/269 But it's already in the kernel, so adoption should be just a matter of time: https://github.com/torvalds/linux/commit/b9d69db87fb77fc80997993d40f091b323b3651e
Not sure what kind of other multi-connection setup would be desired, but if it's a matter of some stateful router in the middle melting down from one high bandwidth connection, then just assigning multiple addresses to one of the hosts would already do the trick with MPTCP as depending on the scheduler, it may establish one subflow per address. On the other hand if the issue is with Syncthing getting compute bound by possibly one peer only being processed on one core, then that's a whole another matter which surely won't be solved by any of the mentioned multipath solutions.
imsodin
commented
1 year ago I don't think we need to do anything to get it, it will become the default behaviour eventually. In that case the only question is if we want to opt-in to using it in it's first iteration in go (1.21) or only once deemed working good enough for everyone (sounds like that's expected for 1.22).
Just for context as it came up: I think the typical request for parallel/inverse-multiplexed tcp conns is for low-quality (high roundtrip times and/or packet loss) connections. Typical examples given are all kind of sync/torrent applications that do that.
On first glance at the "multi-path" concept I got super exited, as I thought it would actully try to discover/make use of multiple paths all along the transmission (e.g. different peering combinations from the ISP at one and and the other). That would likely have fully solved the above use-cases. I guess I got a bit overexited there, probably already multiple addresses at the endpoints as in this proposal is complicated enough :)
bt90
commented
1 year ago I really need to look more deeply into MPTCP. So far, it's been the graphene of networking. It's great on paper, but it never seemed to make it out of the lab.
How does the user avoid unfavorable routing like e.g via his laptops 5G connection?
voidpointertonull
commented
1 year ago While "Go 1.22 or later" does sound like it might be just a matter of time, I'm not that optimistic. As mentioned earlier, Syncthing doesn't care much about IP addresses due its P2P nature even if the current peer address shown is really handy, but on the other hand I can think of quite a few mostly server programs where the administrators would get upset about opt-out MPTCP causing only the initiator IP addresses to be logged. Whether the IP address "fetishization" done by many services makes sense is a whole another topic, but what matters is that mostly enterprise users would be upset by that change without them adapting to supporting multiple addresses first, so I really don't see opt-out becoming the default in golang any soon, but maybe years down the road. I'd expect the following steps:
Torrent wouldn't benefit as much as it tends to have many parallel connections, but in that case it would be still possible to eliminate a fancy multi-WAN router utilizing multiple links in favor of the increasingly more common approach of hosts taking care of more needs by themselves and just expecting "dumb" links to the internet. Syncthing would benefit more as it's (typically) not distributed P2P, so only one connection is used to get data for a specific directory which can't take advantage of multiple paths.
How multiple subflows get established will depend on the host setup. A single interface with 2 addresses setup should be already enough to establish 2 subflows which may exit through 2 different ISP connections, essentially aggregating bandwidth, so increasing reliability is not the only option. Also, as mentioned earlier, IPv4 and IPv6 mixing is possible, and either IPv6 could perform better due to multiple NAT routers in the path, or IPv4 may be better due to messed up IPv6 deployments (where it exists at all) are still not rare. Going beyond that is surely getting a bit too excited, it would be a mini miracle if ISPs could be convinced to have routing which is good for the customers not just their wallets, and it's definitely not possible to alter their routing decisions. A multi-homed setup could end up with wildly different routes though due to simply having different ISPs.
MPTCP surely has an odd history because v0 has been around for a while, but aside from Apple using it, it hasn't really seen large scale adoption. However this time it's the new v1 specification which ended up getting an implementation merged into the Linux kernel, and there's continuous improvement and quite decent support. https://github.com/multipath-tcp/mptcp_net-next is where the new goods are made with Matthieu Baerts doing incredible work seemingly non-stop. https://github.com/multipath-tcp/mptcpd is Intel's contribution which may be already ready for prime time. https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1315 shows that NetworkManager also has preparations for handling MPTCP. https://www.redhat.com/en/blog/using-multipath-tcp-better-survive-outages-and-increase-bandwidth and other writings by RedHat are already getting sysadmins ready. So generally it seems like that this is the iteration that will "make it out of the lab", it just needs enough awareness as developers tend to have setups which are less likely to suffer from all the issues solved by MPTCP, so they are less likely to notice a solution without even looking for one.
One significant upside of MPTCP compared to other implementations on different layers is that the sysadmin is in control of the behavior. It isn't just possible to exclude interfaces from being used, but they can be also marked as backup to be only used when the other links fail, although that's mostly for silly issues like data caps. Aside from that, link usage should be up to the packet scheduling logic which is always going to be implementation dependent. A high latency and low bandwidth link is likely to see only occasional spillover if there's a significantly better link available, but if your 5G connection is almost as good as your wired one (or WiFi making them even more comparable), then if you don't set the 5G link as backup, you could end up seeing the two links sharing the load evenly. A low latency and high bandwidth link will be always preferred, so adding extra links shouldn't degrade performance.
matttbe
commented
1 year ago @voidpointertonull: thank you for this good idea!
I really need to look more deeply into MPTCP. So far, it's been the graphene of networking. It's great on paper, but it never seemed to make it out of the lab.
@bt90 it depends on what you consider a lab but MPTCP is used by Apple with some of their apps (Siri, Music, Maps, etc.) on quite a few millions of devices. It is also used by quite a large number of people with "hybrid access networks".
OK in both cases, the majority of the customers doesn't even know about that but I think we are no longer in a lab case.
How does the user avoid unfavorable routing like e.g via his laptops 5G connection?
@bt90 It is a question of configuration. If nothing is configured, likely only one path will be used (the same as what would be used without MPTCP). If you configure it properly, you can mark this costly path as "backup", ask to use it in priority, or attach to existing connections when the primary connection fail, etc.
While "Go 1.22 or later" does sound like it might be just a matter of time, I'm not that optimistic.
@voidpointertonull: indeed: to change the default behaviour, we will need feedback. It is then a good idea to have such option to get this feedback.
administrators would get upset about opt-out MPTCP causing only the initiator IP addresses to be logged.
@voidpointertonull: once the (MPTCP) connection is established, you have the same user behind even if other subflows are created. I don't know if you need to track that but technically, the userspace application can get the list of subflows with IP addresses, etc. if needed.
Note that to me, it looks interesting to at least have MPTCP support enabled for the server side. If a "plain TCP" (without MPTCP) connection request is sent to the server, the "accepted" socket will be in "plain TCP": MPTCP stack is bypassed. For the client side, MPTCP can also be used. If it is not configured, only one path will be used but that's OK, very small overhead but still prepared to support handovers and network aggregation.
where the new goods are made with Matthieu Baerts doing incredible work seemingly non-stop.
:laughing: I don't think that's correct :stuck_out_tongue:
(If you say that by looking at my Github stats, that's because our CI is doing a daily sync with upstream and for technical reasons, the last commit has to be set to my name as committer to allow other CIs to do their validation :) )
But nice summary for the rest!
adding extra links shouldn't degrade performance.
(we are still working on corner cases but that's the idea)
calmh
commented
1 year ago I don't think we need to do anything to get it, it will become the default behaviour eventually. In that case the only question is if we want to opt-in to using it in it's first iteration in go (1.21) or only once deemed working good enough for everyone (sounds like that's expected for 1.22).
I think we do want a knob for this, but the default value should probably match Go's. That means something that can be enabled by early adopters, and later turned off by ... whoever needs to do that.
voidpointertonull
commented
1 year ago once the (MPTCP) connection is established, you have the same user behind even if other subflows are created. I don't know if you need to track that but technically, the userspace application can get the list of subflows with IP addresses, etc. if needed.
The program can get all addresses, but by just opting into MPTCP usage it would miss out on everything else but the initiating address, and that's definitely undesired in some cases which is why I don't see opt-out becoming the default for libraries at least for a long time. Surely in some cases I can actually see the need, but also keep enterprise logic in mind where measures are often taken for compliance even if they degrade user experience and in some cases even security (password expiry, SMS 2FA, you get the idea). For example brushing logging needs aside, a platform tying an IP address to a session token would end up regularly invalidating sessions due to the initiator IP address regularly changing if MPTCP would be just kind of forced on it without further support in the program.
:laughing: I don't think that's correct :stuck_out_tongue:
Hey, it definitely looks that way. I know that there are others from Tessares, but wherever I see MPTCP goods, I mostly see your name, and that includes a ton of support, not just commit formalities. My mind is set, you are making MPTCP adoption possible, and the company is better off keeping your name plastered as a selling point at this point. :stuck_out_tongue:
(we are still working on corner cases but that's the idea)
Surely another reason why it's going to stay opt-in for a while, and it does seem like it's mostly suitable for increasing reliability first and foremost with bandwidth aggregation coming second, still being a bit rough at the edges. Syncthing would mostly benefit from the bandwidth aggregation, but then generally it's also an excellent collection of networking best practices so I'm quite sure that it will help to explore the other potential benefits like the earlier example of figuring whether IPv4 or IPv6 performs better in a given situation.
Off topic, but I wish there was a VPN solution at least remotely comparable to this project. With performance and user experience varying so heavily based on what's picked from the currently quite fragmented landscape, I occasionally seriously really contemplated dishing out VPN configs through Syncthing in some cases because so far that has been the only really reliable connection maker between hosts, especially when mobility is involved.
I think we do want a knob for this, but the default value should probably match Go's. That means something that can be enabled by early adopters, and later turned off by ... whoever needs to do that.
Sounds reasonable, there's really nothing else required as a starter, just letting users easily opt-in. It's not just a significantly better user experience than forcing MPTCP usage with external tools, but it's also more important to have here as the usual LD_PRELOAD approach of forcing doesn't really work with golang, so the bar is higher than elsewhere.
Eventually it would be great to have the extras like all subflow addresses being shown instead of just the initiator which may be even an address which disappears over the lifetime of the MPTCP connection, but that's really for later.
matttbe
commented
1 year ago once the (MPTCP) connection is established, you have the same user behind even if other subflows are created. I don't know if you need to track that but technically, the userspace application can get the list of subflows with IP addresses, etc. if needed.
The program can get all addresses, but by just opting into MPTCP usage it would miss out on everything else but the initiating address, and that's definitely undesired in some cases which is why I don't see opt-out becoming the default for libraries at least for a long time. Surely in some cases I can actually see the need, but also keep enterprise logic in mind where measures are often taken for compliance even if they degrade user experience and in some cases even security (password expiry, SMS 2FA, you get the idea). For example brushing logging needs aside, a platform tying an IP address to a session token would end up regularly invalidating sessions due to the initiator IP address regularly changing if MPTCP would be just kind of forced on it without further support in the program.
It makes sense to have an option to opt-out just in case.
Except if there is an explicit monitoring of the MPTCP "attributes", the IP of the connection doesn't change. Maybe a firewall could block new subflows but I'm not sure to see why MPTCP subflows would be blocked if "plain TCP" connections are allowed.
But yes, there are probably corner cases :)
laughing I don't think that's correct stuck_out_tongue
Hey, it definitely looks that way. I know that there are others from Tessares, but wherever I see MPTCP goods, I mostly see your name, and that includes a ton of support, not just commit formalities. My mind is set, you are making MPTCP adoption possible, and the company is better off keeping your name plastered as a selling point at this point. stuck_out_tongue
Thank you for these kind words! I should forward that to my employer :-D
(we are still working on corner cases but that's the idea)
Surely another reason why it's going to stay opt-in for a while, and it does seem like it's mostly suitable for increasing reliability first and foremost with bandwidth aggregation coming second, still being a bit rough at the edges. Syncthing would mostly benefit from the bandwidth aggregation, but then generally it's also an excellent collection of networking best practices so I'm quite sure that it will help to explore the other potential benefits like the earlier example of figuring whether IPv4 or IPv6 performs better in a given situation.
Indeed. Aggregation works well but for the moment, a very poor primary link can impact the overall performances. We know what kind of optimisation is needed but we need to dedicate time to that.
Off topic, but I wish there was a VPN solution at least remotely comparable to this project. With performance and user experience varying so heavily based on what's picked from the currently quite fragmented landscape, I occasionally seriously really contemplated dishing out VPN configs through Syncthing in some cases because so far that has been the only really reliable connection maker between hosts, especially when mobility is involved.
(An MPTCP (transparent) proxy might help maybe here)
I think we do want a knob for this, but the default value should probably match Go's. That means something that can be enabled by early adopters, and later turned off by ... whoever needs to do that.
Sounds reasonable, there's really nothing else required as a starter, just letting users easily opt-in. It's not just a significantly better user experience than forcing MPTCP usage with external tools, but it's also more important to have here as the usual LD_PRELOAD approach of forcing doesn't really work with golang, so the bar is higher than elsewhere.
Yes, it would be nice to have an option to force to opt-in (and opt-out) but by default, not doing anything special.
Eventually it would be great to have the extras like all subflow addresses being shown instead of just the initiator which may be even an address which disappears over the lifetime of the MPTCP connection, but that's really for later.
I guess the Golang "net" package should provide such info at some points.
TalalMash
commented
1 year ago I wish to see TCP Options (header) not filtered by many ISPs or sometimes mangled in my case, during travels I noticed that mobile data was mostly unfiltered, while landline did not pass the options field in the majority of time (CGNAT), haven't check IPv6 though.
voidpointertonull
commented
1 year ago I wish to see TCP Options (header) not filtered by many ISPs or sometimes mangled in my case, during travels I noticed that mobile data was mostly unfiltered, while landline did not pass the options field in the majority of time (CGNAT), haven't check IPv6 though.
While I have no direct insight, I believe that filtering that can cause such issues is just becoming more and more common both with CGNAT getting more common, and censorship getting more aggressive as DNS based blocking isn't considered good enough anymore in many places.
Aside from crappy configurations, even net neutrality likely never truly existed in most countries to begin with. Managed hosts in various countries, and aside from obvious issues with filtering and/or bad routing, I've looked into apt-get not working a bit too many times just to find the received package list actually being a silly localized Vodafone page warning about the data cap just because there's one silly side still considering bare HTTP to be good enough, and another side considering the lack of encryption an open invitation to injection. Anything susceptible to MitM is highly likely to have issues somewhere, there's really no good solution for that without preventing feature downgrade.
Initial multipath TCP support was merged into golang recently which should make its usage quite convenient: https://github.com/golang/go/issues/56539
Given that MPTCP is essentially an extension of TCP, basic support should only need opting into the extension at the time of socket creation. The kernel definitely supports graceful fallback to TCP in case the other side doesn't signal MPTCP support or a bogus middlebox breaks support, and apparently the golang implementation also adds fallback, even though the test case also explicitly checks for success: https://github.com/golang/go/commit/383a4e78861218bd861632d6cdcac6c8af6a5470
Support should be straight-forward (once the time is right with the appropriate golang release) with the mentioned fallbacks. A likely good practice for adoption would be adding a config option making the feature opt-in first, then given no issues and the community supporting the idea, opt-out could become the default in the future.
Security considerations: Eventually it might be a good idea to support the concept of one host connecting with multiple addresses at the same time. Given the typical usage of Syncthing and its security model, it already seems to be largely independent of relying on hosts using only specific IP addresses, and by starting with opt-in support, security considerations of likely mostly enterprise users would be revealed eventually. Also consider that having multiple addresses is not unique to MPTCP as Syncthing itself is already aware of the concept to some degree, and there are also plans for multipath support in QUIC just on a different layer: https://datatracker.ietf.org/doc/draft-ietf-quic-multipath/ https://github.com/quic-go/quic-go/issues/3343