idea: tighter protocol

[dominictarr]

okay, so @owenb needs a tighter protocol for socket-stream.

ideas:

• use numbers to indicate message types or, the first character.
• drop ids if the message is from the same id as the previous one.
• use a length delimited protocol, not JSON.

I think that should get the protocol down pretty small, without changing the semantics.

[owenb]

Yes yes yes :) Sounds like a really good way of doing this. Two important things for me are:

1. Being able to send non-JSON strings with the minimal amount of bytes (I always use the example of a high-speed action game just ending move coordinates)
2. Being able to easily identify streams on the client-side, both in terms of sending and receiving data. Not tried to find an approach to this yet, but I'm hoping to get this working next week.

And on my long term wish-list:

1. Being able to send binary data through streams. This is not going to happen overnight but in theory, if you're using a pure websocket transport and a very modern browser, it should be possible to just mark certain frames as binary and send the rest as utf8 strings. As I say, this is not high priority. Reducing the amount of bandwidth has always been my main concern.

Thanks again for all your work on this.

[max-mapper]

+10000 to a binary version of mux-demux. it would be a way better solution than msgpack

[dominictarr]

with the leveldb stuff this is a rising priority.

[dominictarr]

I have been looking at this today.

A lot has changed since I first wrote mux-deumx. the next version will be a complete rewrite.

here are the aims

• lightweight framing - if a message is short, it should only be at most a few bytes longer because of mux-demux.
• has framing that supports binary, text, objects (i.e. json), and objects with binary.
• will work effectively over transports that support text, or binary.
• some reasonable QoS system (this may be pull based or push based, and or limits bandwidth of the entire stream, and maybe adaptive?)
• a reasonable story for nested mux-demuxes, although, lightweight framing should solve most of the problem here.

I'm not sure what the QoS system should look like. pause/resume messages are probably not suitable because of latency. If the roundtrip takes too long, then the pipe can get congested before a pause gets through.

Instead, it will probably need to be more like tcp, which sends a certain amount, and then waits for an ACK. If it gets a NAK, it slows down. of course, since it's already over tcp, there won't be any dropped packets, but could measure latency and throughput, and then slow things down to allow that much through! this is also complicated because the multiplexed streams may want to use their own backpressure...

Probably, I'll just make a basic mux-demux and then QoS will plug into it. This will be useful for other things that need QoS, like post-message streams to webworkers and iframes.