Raw and Object frame correlation

[joshperry]

Hey Jan, figured I would start doing things in a little bit less of a vacuum so we can decide what's best for the project.

I've had a few projects where I've needed the raw frame data as well as the decoded frame. There are actually a few use-cases like this where having a few different composable streams of data would be really helpful. I'd like to address this in streams but I'm not sure what would be best.

Here are a few streams I could see being useful:

FramingStream - pipe a binary stream into it and it de-frames and unescapes frames then the readable side of the stream, in object mode, sends them out as a Buffer object.

DecodingStream - read and write side are object mode, takes a stream of framed Buffer objects and streams out decoded object frames.

EncodingStream - basically the opposite of decoding stream except the write side is object mode and the read side is not. This could be written to allow piping from either of the other two streams, or allow directly writing frame objects to it, and stream out encoded frames as bytes. This stream could then be piped to a serial port stream or socket stream, or whatever stream the XBee is attached to.

I'm not sure exactly what would be the best way these should be implemented in relation to xbee-api. For code re-use, should xbee-api be reimplemented in terms of these streams while keeping the same API; or should these streams be implemented in terms of xbee-api?

What are your thoughts?

[jankolkmeier]

Hey Joshua,

I thought about streaming and xbee-api before. I didn't have a concrete use case, and I also never did much with node streams at my end. A little experiment was this library on top of xbee-api here https://github.com/jankolkmeier/xbee-stream/blob/master/lib/xbee-stream.js. I used it with muxdemux to stream objects (encoded) from a client through tcp to the xbee-connected server and receive the response as decoded objects. (see the example in the previous lib).

From this, there never came the idea for the "perfect" streamed implementation. What I saw that could maybe be useful would be a stream that takes strings/bytes rather than objects and packages it as transmit requests, with some intelligence/protocol layer in between for responses. But how exactly one would use this might depend heavily on the use case.

Similarly, your approach seems to be inspired by your use case. It sounds reasonable, but might not be useful to many others. (At least, so far I haven't gotten any requests for any streaming-like implementation.)

I don't like spending time on something that isn't broken. Given this - and given that it would also be node/npm best practices - my preference would very strongly go to have one streaming-capable abstraction(s) of the xbee-api as a separate modules - maintained by somebody else (you? :) ).

You mention re-implementation... there is also no problem in making some adjustments (non-api-breaking) to xbee-api in order to serve such modules better! But I don't know what changes would benefit streaming. As far as I understand, streaming has greatest benefits when you can send through/manipulate big objects without having to buffer them completely (images, videos, big xml/json objects where information can be parsed incrementally).

While there is some buffering going on in the raw parser to validate a complete packet and then to parse them into objects, these objects are not really huge in size. We are talking about less then 100 bytes typically for a frame. This is microscopic compared to typical streaming applications. A "parallel" implementation where the xbee objects are parsed with every byte that comes in might improve latency and (maybe) memory use, but I wonder if it is really by much (since typically, the bottleneck is the serialport, by far).

This is my intuition, I might be talking out of my ass - let me know what you think ;)

On Fri, Jan 16, 2015 at 10:02 PM, Joshua Perry notifications@github.com wrote:

Hey Jan, figured I would start doing things in a little bit less of a vacuum so we can decide what's best for the project.

I've had a few projects where I've needed the raw frame data as well as the decoded frame. There are actually a few use-cases like this where having a few different composable streams of data would be really helpful. I'd like to address this in streams but I'm not sure what would be best.

Here are a few streams I could see being useful:

FramingStream - pipe a binary stream into it and it de-frames and unescapes frames then the readable side of the stream, in object mode, sends them out as a Buffer object.

DecodingStream - read and write side are object mode, takes a stream of framed Buffer objects and streams out decoded object frames.

EncodingStream - basically the opposite of decoding stream except the write side is object mode and the read side is not. This could be written to allow piping from either of the other two streams, or allow directly writing frame objects to it, and stream out encoded frames as bytes. This stream could then be piped to a serial port stream or socket stream, or whatever stream the XBee is attached to.

I'm not sure exactly what would be the best way these should be implemented in relation to xbee-api. For code re-use, should xbee-api be reimplemented in terms of these streams while keeping the same API; or should these streams be implemented in terms of xbee-api?

What are your thoughts?

— Reply to this email directly or view it on GitHub https://github.com/jankolkmeier/xbee-api/issues/23.

[joshperry]

I guess the streaming style is only one way to do things, but I enjoy it. I think one less-known feature of Streams in node is objectMode where they can be used to stream objects instead of a raw bytes.

Here's an example of code at the end of a tool I'm currently working on. It's only a small part of the code where all the streams are composed, but I think it will make sense:

// Intercept frames from the app to the network
var apiTxStream = new ApiStream();
var throttlingStream = new ApiThrottlingStream();
var txBuildingStream = new ApiBuildingStream();
nullCom
  .pipe(apiTxStream)
  .pipe(throttlingStream)
  .pipe(routingStream)
  .pipe(debee.txTapStream)
  .pipe(buildingStream)
  .pipe(xbeeCom);

// Intercept frames from the network to the app
var apiRxStream = new ApiStream();
var rxBuildingStream = new ApiBuildingStream();
var txQueuePumpTap = new TapStream(function(frame) { if(frame.id) throttlingStream.pumpTxQueue(frame.id); });
xbeeCom
  .pipe(apiRxStream)
  .pipe(txQueuePumpTap)
  .pipe(debee.rxTapStream)
  .pipe(routeCacheTap)
  .pipe(rxBuildingStream)
  .pipe(nullCom);