transport: add UDP support?

[grrtrr]

For networks with high latencies and for certain messaging scenarios where a reliable underlying transport protocol is not required, UDP transport support is desirable.

In our REQ/REP scenario, with a worst-case RTT of 0.3 seconds, the initial TCP handshake alone would take 1 second, which is a lot of overhead for short exchanges.

Question: whether UDP support is planned for go-mangos?

For nanomessage, this was originally specified in sp-udp-mapping-01.txt.

During a proof-of-concept, there were 2 challenges I ran into:

1. asynchronous errors - in TCP, socket errors are instantaneously visible; UDP errors (via ICMP) are reported on the next socket call (which leads to e.g. 'connection refused' on send). Mitigations are
   • poll socket error queue (IP_RECVERR, Linux only);
   • retry operations multiple times (i.e. multiple send calls)
2. different socket semantics: go-mangos PipeListener requires Accept. This can be mimicked in the REQ/REP case using a single listener socket, which de-multiplexes incoming connections. In the PUB/SUB case, this would not work, since the SUB sockets require a passive connection setup.

As an alternative, TCP Fast Open might be an option. Go does not yet support it, but they are working on it.

[gdamore]

Its certainly something I've thought of. There are semantic differences for certain, but I think we can overcome them.

I think with UDP you just give up on getting errors -- its only best effort, and very loosely at that.

To handle Accept() I think you wind up faking out a single socket, or changing mangos to not require this, since UDP doesn't have these semantics. That might help address the semantic problems around Windows named pipes, which are also a bit different from traditional UNIX connected pipes.

[grrtrr]

I have some code used for req/rep communication via UDP. Would you be willing to review it?

[gdamore]

Yes.

[joeblew99]

if you use QUIC you get error correction.

https://github.com/lucas-clemente/quic-go

[gdamore]

Oh, this looks promising. Will have to look at it further.

[joeblew99]

i have been using it (not in production), and its easy to use.

I use it for peer to peer communications from mobiles to servers. Being golang and not being dependent on the linux kernel it runs fine.

On Wed, Mar 29, 2017 at 1:49 AM gdamore notifications@github.com wrote:

Oh, this looks promising. Will have to look at it further.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/go-mangos/mangos/issues/215#issuecomment-289939468, or mute the thread https://github.com/notifications/unsubscribe-auth/ALcac5D77DAfKK9fNOF13VwxXjzP3k2uks5rqZyXgaJpZM4I6U7_ .

[linkerlin]

Using kcptun as UDP tunnel will get a high-speed when you need to run mangos on Internet.

[joeblew99]

@linkerlin does it work on mobiles too ? U se golang on mobiles as well as desktops and servers.

QUIC works on mobiles, and so why i suggested it. But it does not have all the goodies on kcptun from what i can see, like the routing through different servers etc ?

@gdamore Did you get a chance to play with QUIC ?

this is a good write up too: https://www.callstats.io/2017/06/02/quic/

[gdamore]

I have looked at this, and it would appear that QUIC has matured somewhat. However, it seems that mostly it's just for browsers (chromium) at present; there doesn't seem to be a good implementation of it for C programs for example. (Well, the there is the chromium thing.)

kcp looks interesting too. I don't think kcptun is as interesting from mangos standpoint though.

I may write a kcp transport -- its probably superior to whatever I'd come up with for pure UDP.

[joeblew99]

QUIC is used for peer to peer as a replacement for webrtc. Its definitely NOT just for browsers :) It is backed into Chrome browser though and al google server use it instead of SPDY now

Its being used on android for many google products for things like hangout.

Its also being integrated into the IPFS and the multiformats repo. https://protocol.ai/ run it.

On Sat, Jul 22, 2017 at 3:19 PM gdamore notifications@github.com wrote:

I have looked at this, and it would appear that QUIC has matured somewhat. However, it seems that mostly it's just for browsers (chromium) at present; there doesn't seem to be a good implementation of it for C programs for example. (Well, the there is the chromium thing.)

kcp looks interesting too. I don't think kcptun is as interesting from mangos standpoint though.

I may write a kcp transport -- its probably superior to whatever I'd come up with for pure UDP.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/go-mangos/mangos/issues/215#issuecomment-317183321, or mute the thread https://github.com/notifications/unsubscribe-auth/ALcac4mNx4Gf9IziP56_34X6Bq55QtENks5sQfbdgaJpZM4I6U7_ .

[gdamore]

Most extant quic consumers are JavaScript and use the chrome in browser implementation. From what I can see the protocol is still quite experimental. The only implementation I know of other than the chromium port or the extraction thereof is the pure Go port. The lack of native ports for other languages like Java makes me doubt that it is widely used or mature enough to depend upon outside of the browser/javascript ecosystem. On Sun, Jul 23, 2017 at 4:12 PM jow blew notifications@github.com wrote:

QUIC is used for peer to peer as a replacement for webrtc. Its definitely NOT just for browsers :) It is backed into Chrome browser though and al google server use it instead of SPDY now

Its being used on android for many google products for things like hangout.

Its also being integrated into the IPFS and the multiformats repo. https://protocol.ai/ run it.

On Sat, Jul 22, 2017 at 3:19 PM gdamore notifications@github.com wrote:

I have looked at this, and it would appear that QUIC has matured somewhat. However, it seems that mostly it's just for browsers (chromium) at present; there doesn't seem to be a good implementation of it for C programs for example. (Well, the there is the chromium thing.)

kcp looks interesting too. I don't think kcptun is as interesting from mangos standpoint though.

I may write a kcp transport -- its probably superior to whatever I'd come up with for pure UDP.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/go-mangos/mangos/issues/215#issuecomment-317183321, or mute the thread < https://github.com/notifications/unsubscribe-auth/ALcac4mNx4Gf9IziP56_34X6Bq55QtENks5sQfbdgaJpZM4I6U7_

.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/go-mangos/mangos/issues/215#issuecomment-317289196, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPDfVNySSvTmIjRdiEnQutphMA-F5bXks5sQ9NmgaJpZM4I6U7_ .

[linkerlin]

@joeblew99 In China, a lot of people using kcptun as a tunnel for speed-up shadowsocks which is a socks5-like proxy on Android, iOS, macos, windows. The kcptun is very handy tool.

[linkerlin]

@gdamore kcp-go is a Production-Grade Reliable-UDP Library for golang.

It provides fast, ordered and error-checked delivery of streams over UDP packets, has been well tested with opensource project kcptun. Millions of devices(from low-end MIPS routers to high-end servers) are running with kcp-go at present, including applications like online games, live broadcasting, file synchronization and network acceleration.

NOTICE: kcp-go is not kcptun.

[joeblew99]

@linkerlin thanks for the extra info. thats really useful because i need to run this on mobiles. Will try out kcp-go

[gdamore]

Btw, without a connection oriented view, there are real problems with a pure UDP transport. Basically, you can't use these things in "device" configurations, where you can forward across the transport, because you can't stuff a correct value in the backtrace header. (There is no correct value, since you don't have a pipe ID at all. You'd have to fabricate one and add state -- which is something I've been working on -- or you have to abandon one of SP's nicest semantics.

I have looked at KCP, and it looks pretty nice (more looking at the C implementation than the GO one btw.) Yeah, not kcptun at all.

I'm not sure if KCP answers the need for UDP in itself -- it still requires a handshake of a sort, which can hurt. Connection set up time optimizations in QUIC are also possible.

I guess this is part of the question; how critical is it to optimize the initial connection set up time. For browsers at Google scale, its a serious problem. Are folks using nanomsg in scenarios where this is also true? (Most of my use cases of nanomsg involve sockets that are generally connected longer.)

Where I see KCP winning is in a better tolerance for lossy/marginal networks; it degrades more gracefully (at the expense of bandwidth), so applications that fare poorly with stalled streams (real-time chat/video/gaming especially) tend to prefer to KCP -- it allows them to still use a connected stream semantic while faring better than TCP.

[joeblew99]

Thank you for the analysis. That really addresses me as i am planning to use this for both application data but also real time video / chat.

There is also a good P2P lib that also now has QUIC here: https://github.com/go-mangos/mangos/issues/278

[joeblew99]

You asked about how important connection setup time is. I do alot of IOT. The devices are always network starved and the connections very shaky. The devices are very often sending data metrics in my case.

100% pure golang with no choice is critical also because the devices have no c runtime for security reasons. See the go-krazy github project for example.

https://github.com/gokrazy/firmware

It provides a way to build embedded images that support transactional updates. They is no c code.

[glycerine]

For what its worth, I played with kcp-go and while its latency was good, its throughput was horrible. I got 1/10th of what TCP was getting even with crypto off, using the default FEC parameters.

QUIC is widely used in Chrome these days, and for lots of mobile video from Youtube. I expect it to be mature than kcp-go. C/C++ library is https://github.com/devsisters/libquic

[gdamore]

That is really terrible. Thanks for the info. On Thu, Aug 3, 2017 at 4:18 AM Jason E. Aten, Ph.D. < notifications@github.com> wrote:

For what its worth, I played with kcp-go and while its latency was good, its throughput was horrible. I got 1/10th of what TCP was getting even with crypto off, using the default FEC parameters.

QUIC is widely used in Chrome these days, and for lots of mobile video from Youtube. I expect it to be mature than kcp-go. C/C++ library is https://github.com/devsisters/libquic

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/go-mangos/mangos/issues/215#issuecomment-319941040, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPDfZmYF2AE9VePbniGn7sO1x3sr1Hnks5sUax4gaJpZM4I6U7_ .

[glycerine]

The github.com/lucas-clemente/quic-go implementation of QUIC is actually quite nice. In my quick and dirty California-to-NewJersey over the internet benchmarks, I'm getting the same throughput using QUIC (which is TLS encrypted) as I am with unencrypted TCP. Here's the code if anyone wants to replicate and/or do more scientific benchmarking in a controlled environment.

https://github.com/glycerine/pingpong/tree/master/cmd/quic

vs

https://github.com/glycerine/pingpong/tree/master/cmd/tcp

[glycerine]

Comment: For high bandwidth*delay networks, and in particular lossy networks, even though QUIC provides zero-round-trip connection setup, I doubt it is the right choice. Something[0] like Tsunami-UDP[1] would be needed. There is the original C and several branches that patch it[2][3], and two half-finished implementations in Go[4][5].

[0] https://www.globus.org/sites/default/files/Survey.pdf
[1] http://tsunami-udp.sourceforge.net/
[2] https://github.com/sebsto/tsunami-udp
[3] https://github.com/rriley/tsunami-udp
[4] https://github.com/longbai/tsunami-udp-go
[5] https://github.com/mcqueenorama/tsunami

[gdamore]

The thing to remember is that there are very different decisions depending on whether you want to

a) reduce round trip latency of regular messages b) reduce connection setup latency c) increase single stream bandwidth/throughput d) increase aggregate bandwidth/throughput e) eliminate/reduce network congestion

Historically (within the last decade or so), most systems have focused hard on “d”. Prior to that folks were most concerned about “e”. (Note that d & e are generally complementary.) These are areas where TCP shines.

However, for a-c single stream TCP is horrible And the optimizations you make for one of these generally penalize one of the others. Sometimes devastatingly so. (Examples of this are Nagle+DelayedACK causing 100s of msec of latency — not good for real time applications! Conversely, reducing latency for good networks may cause severe degradation and increased congestion by spamming replay requests when the network is degraded.

The upshot here is, you need to understand what you’re trying to achieve. It’s kind of silly to worry about long haul lossy networks when optimizing for intra-DC comms, and its also probably bad to worry about throughput on a low bandwidth & low latency protocol (e.g. FIX trading protocol. Dirty secret: high frequency traders use tiny amounts of bandwidth, but are happy to pay $$$$ for 100GbE or whatever — all in the drive to get their 200 byte frames from point A to B in the shortest time possible. The fact that might have only a few hundred of these to send per minute is irrelevant.)

Anyway, this RFE isn’t the place to do protocol analysis & comparisons. If someone wants to make new streaming protocol experiments in mangos, I welcome that work to arrive as PRs. I can’t promise I will merge them, but that effort goes far more towards convincing than just discussion. Such PRs should clearly stipulate what goals the project is trying to achieve (why is TCP not good enough?) and demonstrate some kind of meaningful improvements on those metrics.

• Garrett

On Mon, Aug 7, 2017 at 11:21 PM Jason E. Aten, Ph.D. < notifications@github.com> wrote:

Comment: For high bandwidth*delay networks, and in particular lossy networks, even though quick provides zero-round-trip connection setup, I doubt it is the right choice. Something[0] like Tsunami-UDP[1] would be needed. There is the original C and several branches that patch it[2][3], and two half-finished implementations in Go[4][5].

[0] https://www.globus.org/sites/default/files/Survey.pdf [1] http://tsunami-udp.sourceforge.net/ [2] https://github.com/sebsto/tsunami-udp [3] https://github.com/rriley/tsunami-udp [4] https://github.com/longbai/tsunami-udp-go [5] https://github.com/mcqueenorama/tsunami

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/go-mangos/mangos/issues/215#issuecomment-320861878, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPDfVvO8MacxvHNk1uBTDbqP0m2urMXks5sV_54gaJpZM4I6U7_ .