Transport Protocol

[liamzebedee]

What transport protocol shall XDen communicate over? Requirements:

• Reliable to some large extent
• Message-based (message boundaries are preserved)
• In-order not enforced

[liamzebedee]

Structured Stream Transport

• Ideal protocol however it appears to be dead, and the source cannot be accessed.
• Allows clients to create long-lived connections that allows for subsequent structured streams to be created. These streams, unlike TCP, do not require ACK-SYN etc. crap when sending data, because this was already sent when the connection was initiated. This is very efficient, as in Kademlia we realise the fact that long-lived nodes are likely to stay in the network - long-standing connections are efficient.
• Fufills requirements

[liamzebedee]

SCTP:

• Fufills all the requirements
• Congestion control
• Go library in development

Overall the easiest protocol to use.

[eternaleye]

Wanted to note a few things: First of all, SCTP doesn't permit adding channels after setup. This could be worked around by multiplexing messages, but then you lose the benefit of not forcing an ordering between messages on different channels. This may or may not be an issue depending on traffic characteristics.

Second, SST spawned a group called Tng (Transport Next Generation) as a collaboration between the DeDiS group at Yale and another college. As I understand it, part of the reason was that flow middleboxes made it very hard to deploy SST (which should be far less an issue on Hyperborea), but the author decided to work on addressing some of those issues by splitting the transport layer into sublayers. THAT is something that may be very interesting for Hyperborea, including how it looks to allow arbitrary semantics by composing components - for instance, it has the 'minion' system that allows out-of-order delivery over TCP or TLS, without the format on-the-wire being distinguishable by middleboxes from a vanilla stream.

[liamzebedee]

Interesting points, I was not aware of Tng (although it doesn't look like much is happening).

I haven't fully investigated SCTP [yet], could you explain this concept of channels?

[eternaleye]

Tng doesn't publish all that often, but they do publish every once in a while. They just aren't hugely public about 'in-progress' stuff. See: That gap in the publications between October '09 and November '10

In addition, the Tng architecture does make a good guideline for protocol design.

Also, in the case of "there's research, but no code": Don't be afraid to ask. If you want to see it, send them an email. The worst that can happen is they say "no" - but they might say yes. However, note that the SST prototype was done in C++ with Qt.

Regarding SCTP, I may be using the wrong terminology (wrt channels), but the basic idea is that on connection setup you declare how many independent streams of messages you want. This can't be changed after connection setup, which IIRC was one of the point brought up in the presentation the author of SST gave (there used to be a video floating around). They share congestion control, but have independent flow control. So you could have a stream for bulk data and a stream for low-latency data, but you couldn't do the SST thing of making a transient stream for each image in a webpage on the fly.

[liamzebedee]

Yeah just read this sufficient guide on SCTP which explained to me the concept of channels (similar to structured streams in SST). It's an implementation specific issue so I'll read up on it after Issue #1 is fixed.

Regarding SST and co., I've tried contacting Bryan Ford (author of SST) multiple times regarding the status of the library (I've hosted a mirror) but no response.

I'm hoping that with the aid of SWIG I'll be able to get the original SST implementation working with Go. SCTP is my second preference, only because it is more widely supported and maintained.

[liamzebedee]

This weekend I'll start work trying to get Structured Stream Transport working with Go.

[liamzebedee]

Work is being done on GoSST.

[liamzebedee]

I'm slowly understanding the fundamentals of SST. Today I discovered Channel-based Unidirectional Stream Protocol (CUSP) created by these researchers, which appears to be a combination of SCTP and SST (the two candidaites for the transport protocol).

I've downloaded the source and will try get it working with SWIG this afternoon. It doesn't appear to have any external dependencies, which works in my favour when building on Windows.

[eternaleye]

CUSP looks promising, especially with how it abandons the 'forking streams' model SST used. I suspect that if they hadn't used Qt's signals/slots mechanism, dealing with that paradigm would have been exceedingly messy. In pure C, I suspect they would have had to either do some ugly things to have a callback system or would have run into potential deadlocks.

EDIT: Ah, here it is.

Instead, an action expecting one or more answers supplies
in its stream the ID of one reply service for each response
type. A complex conversation with many possible forks and
outcomes is then modelled using one stream per step in the
logical flow. In the case of indirect multi-hop scenarios, reply
service IDs can be simply forwarded to the eventual responder.

So instead of forking streams from either end, the sender just says "If you want to respond, here's where I'm listening," which makes a lot more sense in several ways.

EDIT2: Mm, and I really like the explicit CHALLENGE/RESPONSE fallback

EDIT3: Although the way they roll their own crypto does bug me, and make me wonder if DTLS could be used in that role due to the channel layer being a packet transport. ISTR there being provisions for transport-specific ways of reducing roundtrips on setup. Nope, I was thinking of TLS False Start, which not only was for vanilla TLS but also never made it out of draft.

EDIT4: On further examination, it might still be feasible - DCCP permits application data payloads in its handshake, and the official spec for DTLS/DCCP [rfc5238] explicitly permits taking advantage of that in section 3.2.

[liamzebedee]

Ok so I've been working on building the library with SWIG (it isn't as hard as I thought). Currently I have an interface file defined as:

// cusp.i
%module cusp
%{
#include "cusp/cbindings/include/cusp.cpp"
%}

#include "cusp/cbindings/include/cusp.cpp"

Then I compile this using swig.exe -go cusp.i which generates the files cusp_wrap.c, cusp_gc.c and cusp.go.

As per SWIG Go Guide I try to build the wrapper first using gcc -x c++ -c -fpic cusp_wrap.c however I fail because libcusp.h cannot be found. Upon further investigation this library is in fact built with C, C++ and ML, and I require to prebuild another library - libcusp.a - which depends on ML.

Anyways, I'm going to sleep, I'll look into it tomorrow.

[liamzebedee]

Now that I'm running on Ubuntu Linux, it's easier to use all these unix-specific tools. None the less, it has still been troublesome to get cusp to compile. Currently, my compilations are a success, but I'm not producing the shared library which I needed.

I'm also thinking of developing my own transport protocol as an a project to unite all the existing ones I liked.

[liamzebedee]

Ok I've got libcusp compiling to a static library (.a) and the SWIG bindings compiling to a dynamic library (.so). Now I have to mess with 6g and co.

[liamzebedee]

I've sent an email to the developers of CUSP. A lot of work has been going on, trying to get the SWIG bindings working.